Upload multiple files to Google Drive

Help
, , , ,
1

Hello,

New to pipedream, I just started creating a Workflow that go over my liked tweets, and if they contain media, those medias are uploaded to Google Drive.
I have three steps in my workflow :

  • A trigger (every 24 hours) on my liked tweets
  • A custom node.js step to generate, for each tweet, an array of objects containing the URL of the image (I add some parameters to the URL returned by the twitter API to get the large image size) and the name of the file to create on Google Drive (so that they reference the screen name and id of original tweet, instead of a random name)
  • An “upload file to Google Drive” step that is already available on pipedream. I can make it work, but unfortunately this only process ONE element from my array (one url / filename pair). But my array from the previous steps can contain from 0 and up to 4 elements to save to Google Drive.

I fail to understand what would be the clean way to do this on Pipedream. I’m almost hesitating just making a script from scratch in Python and running it myself on a server. But I’d love to try Pipedream.

Is there a way to create “events” from my custom node.js step, that could serve to run other workflows ? (I’d launch an event per media to serve, and another workflow would catch those event and just run a “upload file to Google Drive” step)
Or is there a way to somehow run a step for each elements on my array ?
Orrr, should I write my own “upload to Google Drive” custom step by taking inspiration on https://github.com/PipedreamHQ/pipedream/blob/master/components/google_drive/actions/upload-file/upload-file.mjs ? And if yes, how would I import the utilities like import googleDrive from “…/…/google_drive.app.mjs”; ?

2

Hi @Hexalyse

First off, welcome to the Pipedream community. Happy to have you!

You can solve this in a few ways. For minimal code writing (and maintaining) I like to use 2 separate workflows for this.

The first workflow is responsible for fetching a list of data, which you already have set up.

Then a second workflow is reponsible for performing work on each record in your data.

Then you can use the $.send.http destination feature to send background job powered HTTP requests to trigger your other workflow from the first:

tweets.forEach(tweet => {
  $.http.send({
     url: // your second workflow webhook URL here,
     body: {
        media_url: tweet.path.to.media.url
     }
  });
});

But you can also create your own Google Drive step that accepts an array of strings rather than one single URL string.

The advantage of that is you’d keep your process to one workflow instead of two.

1 Like
3

Hi @pierce

Thank you for the thorough answer.

Being new to Pipedream, and despite looking at some examples, I’m not sure I understand very well how I would easily access the Drive content. I guess there are some helper functions to upload, delete files, etc. but the example of custom Drive script directly calls an API endpoint with axios (idk if it’s the correct route to also upload files). I’d probably even go faster with starting from scratch without pipedream, haha.

So I think I’ll choose the option 1, with 2 workflows. I wouldn’t have thought about sending an HTTP request to… another workflow webhook on pipedream! That’s actually pretty smart.
Is there any downside to doing this instead of having one workflow ? Does it count as double the executions on the quota ? I didn’t check in detail how those executions are counted : is it each step that counts as one, or one entire workflow, etc. ?

4

Hi @Hexalyse,

All pre-built actions on Pipedream are wrappers around axios to interact with API’s directly.

The pre-built actions are just a no-code way of doing so.

When you need to integrate with an API endpoint, or apply custom logic like looping over an array in your case, then you can dip into the low-code step instead.

Yes, two workflows will scale infinitely because you’re essentially calling a serverless lambda per workflow but it will cost two invocations instead of one.

Invocation definition and other pricing details

I will say the nice thing about using Pipedream’s “Use with any App Name API” action (shown in the screenshot below) is that you don’t have to think about OAuth, handling refresh tokens, or even remembering how to pass tokens in the API request.

CleanShot 2022-06-02 at 09.36.32@2x
CleanShot 2022-06-02 at 09.36.32@2x2148Ă—966 182 KB

Instead you just need to concern yourself with passing the data from step exports from other steps and shaping the payload correctly to the other API.

5

Thanks again for the details @pierce

Yes, I have no problem with diving in the code (I’m a developer). That’s even the reason I wanted to try pipedream : instead of just exposing pre-built actions and being confined to what is exposed and available, you can just write your own code, but pipedream facilitate all the annoying “auth” part… AND also gives you some practical pre-build steps that you’ll generally need.

Okay so yeah, the fact it doubles the amount of invocations counted isn’t really great. I’d prefer keeping it all in one workflow, then.

My concern was more that… I then need to dive into the Google Drive API and how to implement file upload by calling their endpoint with axios. And pipedream just facilitate the auth process by giving me direct access to the authorization token to pass in my HTTP requests.
I guess if I can see the code of the pre-built action for file upload (I guess it’s accessible somewhere on Github ?) it could give me an idea of how to do it without having to go full dive into Google’s API - which is something I’m not eager to. I’ve already dealt with Google services and it’s always a nightmare of complexity compared to all the other APIs I’ve used in my life

6

Sure thing @Hexalyse happy to help.

Correct all pre-built actions & triggers are open source, you can find them on our public Github repo. Here’s the path to the Google Drive Upload File action specifically: pipedream/components/google_drive/actions/upload-file at master · PipedreamHQ/pipedream · GitHub

Now there are 2 different ways of going about customizations.

1. Just create Node.js code steps

Pros

  • No CLI or local code to manage
  • Add an app prop to connect the given app, and this.google_drive.$auth will contain your refresh tokens

Cons

  • No access to the methods defined in the google_drive.app.mjs file in the public repo
  • Not reusable across many workflows

2. Deploy your own actions

Pros

  1. You can deploy your own private actions to your account for reuse
  2. You can utilize existing actions and app.mjs files in our public repo for your actions

Cons

  1. You’ll need to install and setup the Pipedream CLI tool
  2. You’ll have to develop locally and use pd publish your-action-name.mjs and also up the version every time you make a change

But you may prefer to develop locally. But for quick step development I tend to just create Node.js steps.

I hope this helps! Let me know if I can clarify anything else.

1 Like
7

Hi @pierce,

I’m trying to figure out how to upload multiple files to Google Drive, but I’m struggling with implementing the custom logic. Would you mind sharing any resources on how to structure the /upload requests?

8

Hi @irinagabe ,

Sure, the best place for a reference is Google Drive’s API docs themselves. Here’s a link to the /upload endpoint specifically:

Could you share the code you have written so far? Might be able to help from there.

9

Thanks, here’s the code snippet:

import { axios } from "@pipedream/platform"
export default defineComponent({
  props: {
    google_drive: {
      type: "app",
      app: "google_drive",
    }
  },
  async run({steps, $}) {
    const files = steps.trigger.event.fields[14].value
    const folderId = steps.Create_Project_Folder.$return_value.id

    for(let i = 0; i < files.length; i++){
      
      await axios($, {
        url: `https://www.googleapis.com/upload/drive/v3/files?uploadType=media`,
        headers: {
          Authorization: `Bearer ${this.google_drive.$auth.oauth_access_token}`,
          'Content-Type': files[i].mimeType,
          'Content-Length': files[i].size,
          'Host':  `www.googleapis.com`
        },
        method: 'POST',
        data: {
          "name": files[i].name,
          "mimeType": files[i].mimeType,
          "url": files[i].url,
          "parents": [
            folderId
          ]
        }
      })
    }
  },
})

I’m struggling with passing the file’s data from the URL

10

Hi @irinagabe

Sorry, I’m not sure of the structure of the step export referenced in the code you shared (steps.trigger.event.fields or steps.Create_Project_Folder.$return_value)

But if you’re struggling with uploading files and want leverage the pre-built actions instead, you can try following this guide to split your workflow into two:

That way you can use the easy pre-built Google - Upload Files action that takes care of the code problem.

1 Like
11

Thanks @pierce, I’ll try splitting it into 2 workflows. Maybe the option to upload multiple files can be added to the pre-built action in the future?

12

Great, hope that helps @irinagabe !

Even better, there’s a design in the works to allow you to iterate in the same workflow on any pre-built action. Not just Google Drive - Upload file.

We’ll be making a big announcement when that arrives, but you can also follow that Github issue for instant updates.

13

Awesome! I was wondering when that was coming to Pipedream because I’ve used that functionality on n8n. I’ll definitely be on the lookout for that :slight_smile: