Issues with workflow to download file from Google Drive and upload to ImgBB

Help
,
1

This topic was automatically generated from Slack. You can find the original thread here.

Csaba Seregy : Hi team, i’m asking for a little help :slightly_smiling_face: unfortunately i’m new to nodejs but with pipedream’s implementation it’s not even difficult to get used to it :thumbsup: . I’m trying to migrate all of my google drive pictures from a specified folder to imgbb.com and I have 2 predefined steps that works perfectly for small sized pictures (1mb, every image beyond that limit exceeds the memory, because the base64 data is being stored in a variable, which is i know not the best practice), i’m using the steps.googledrive_download_file and the steps.upload_picture (for imgbb) . How can i combine these two with data streaming, can i reference to a file that got created with steps.googledrive_download_file inside of the steps.upload_picture? Any help would be highly appreciated. Thanks in advance

2

Dylan Sather (Pipedream) : thanks for reaching out. Typically I would save the file to the /tmp directory in the first step and read it from /tmp in the next step.

Does that work?

3

Csaba Seregy : Unfortunately no, “Out of Memoryyou’ve exceeded the default memory for this workflow, see https://docs.pipedream.com/errors/#out-of-memory”

4

Dylan Sather (Pipedream) : Would you mind clicking on the Share button in the top-right of your workflow and sharing it with dylan@pipedream.com? I’m away from my computer at the moment but will take a look ASAP

5

Csaba Seregy : The thing is , it gets the image first from google (binary) and saves it under a variable, if takes too much space it raises this error message, data streaming would be ideal, but i’m not entirely how can that be done within nodejs async

6

Dylan Sather (Pipedream) : Take a look at the section of this code where I download the file: https://pipedream.com/@dylburger/download-file-then-upload-file-via-multipart-form-data-request-p_QPCx7p/edit

7

Dylan Sather (Pipedream) : That will stream the file to disk, saving it at filePath

8

Csaba Seregy : Sure, i’ll try that, looks promising

9

Dylan Sather (Pipedream) : I worked on another workflow specific to the Google Drive case. The first step asks for a file ID from Google Drive and a location (like /tmp/file.png) where you’d like to save the file to disk. I published that as an action you can use (see GIF).

I’m not super familiar with the ImgBB upload API, but it looks like they accept multipart/form-data requests. You can stream a file upload in that manner with axios using some of the suggestions here and here.

2021-02-11 10.41.33.gif
2021-02-11 10.41.33.gif2416×1074 482 KB

10

Csaba Seregy : Cool stuff, i didn’t see it before when i selected the google drive and typed download… Now i just have to figure out the imgbb part and to find out how a photo can be streamed there from a file

11

Dylan Sather (Pipedream) : I haven’t tested it, but I believe you’ll essentially want something like this:

const fs = require("fs")
const FormData = require("form-data");

const formData = new FormData()
formData.append('image', fs.createReadStream(filePathFromPreviousStep))
const headers = formData.getHeaders()

const config = {
  method: "post",
  url: `<https://api.imgbb.com/1/upload?key=${auths.imgbb.api_key}>`,
  headers,
  data: formData,
}
return await require("@pipedreamhq/platform").axios(this, config)
12

Csaba Seregy : i’ll test it, Dylan you did already an amazing job with this

13

Csaba Seregy : :thumbsup:

14

Csaba Seregy : It worked, thank you for being so helpful :slightly_smiling_face: works like a charm

15

Dylan Sather (Pipedream) : that’s great!