How to Download a File from an API and Upload It to Another API in a Pipedream Workflow Without Using 'tmp' Directory or File Store?

Help
1

This topic was automatically generated from Slack. You can find the original thread here.

Hello, I’d like some input about how I can do the following in a Pipedream workflow, please:

  1. Download a file from some API
  2. Upload / send that same file to another API
    I’ve tried storing the downloaded file in the tmp directory, but confused about how to access it later on (and, according to the docs, it doesn’t have a big capacity, so I hesitate to use it at all - I need much more storage space).
    Is there a way I can access the downloaded file (bytes) as a return value in another step, where I’m uploading it to another API, without storing it in tmp or using a file store?

My current workflow is something like:
• Download the file from API1
• It seems to return as a string, so I have a Python code step that encodes it into a bytes array and returns it
• I try accessing this file in another step, but the return value is “Unserializable object of type bytes”
Alternatively - can I send a file to Pipedream in a request and then do something with it in the workflow? (And can Pipedream workflows return files?) I’m thinking I could potentially break this process up into two workflows, if these would work.

I appreciate any help. Thanks in advance!

2

There should be more than enough space in the /tmp directory.

3

Much more than passing your files around as strings or bytes, in fact.

4

To write a file to /tmp using a stream in NodeJS:

const res = await fetch(url);
const fileStream = fs.createWriteStream('/tmp/file.ext', { flags: 'wx' });
await finished(Readable.fromWeb(res.body).pipe(fileStream));
5

You could also use curl in a bash step.

6

We do that often to download GitHub repos:

curl -L \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: $GITHUB_TOKEN" \
  -H "X-GitHub-Api-Version: 2022-11-28" \
  --url 'https://api.github.com/repos/org/repo/tarball/master' \
  --fail | tar -xz -C /tmp/my_repo --strip-components=1
7

Disclaimer: the /tmp directory sometimes acts weird in the builder (it gets wiped frequently), but it works fine once deployed.

8

And using/reading a file in the /tmp directory works the same way as reading it from your local filesystem.

9

How would this work if I want to fetch the file from an app using end user credentials? I am fetching the file using a request step for one of the apps Pipedream supports

10

Use a code step instead, and pass in the user credentials as a prop

11

I may be downloading rather large files. The docs say it holds 2GB; Some of the files I’ll be downloading will be mp3s

12

I’ve never seen mp3s that large TBH.

13

And so - to read the file from tmp and to upload it to another API, I have to use a node code step?

14

Yes and no.

15

It depend what the Pipedream native steps allow.

16

For example, AWS S3 has a couple of options:

image.png
image.png347×397 21.2 KB

17

One of them is to upload from /tmp :point_up_2:

18

I’m confused; pipedream doesn’t seem like it supports returning a file from a step. So is the only option then to use code steps and perform all the file downloading/uploading in a node code step

19

Oh, I see

20

So if the action doesn’t have that option, the only other way is using code?