This topic was automatically generated from Slack. You can find the original thread here.
I have 10 public s3 link of the file which I need to download into gdrive, Suggest me the workflow steps to achieve please.
To create a workflow that downloads files from S3 links and uploads them to Google Drive, you can follow these steps: 1. Use an HTTP trigger for your workflow. 2. Add a Code step to define the S3 links as an array. 3. Add a Loop step to iterate over the S3 links. 4. Within the loop, add a Code step to download the file from the S3 link using axios. 5. Add a Google Drive - Upload File action to upload the downloaded file to Google Drive. Here’s an example of the code for step 2: export default defineComponent({ async run({ steps, $ }) { const s3Links = [ "https://s3.example.com/file1", "https://s3.example.com/file2", // Add more links here ]; return s3Links; }, }); And here’s an example of the code for step 4: import { axios } from "@pipedream/platform"; export default defineComponent({ async run({ steps, $ }) { const s3Link = steps.loop.item; const response = await axios($, { method: "GET", url: s3Link, responseType: "arraybuffer", }); const fileData = response.data; const fileName = s3Link.split("/").pop(); return { fileData, fileName }; }, }); Remember to connect your Google Drive account to the Google Drive - Upload File action and configure the action to use the fileData and fileName from the previous code step. This workflow will download each file from the provided S3 links and upload them to your Google Drive.
I have used this code
**import** axios **from** 'axios';
**import** { google } **from** 'googleapis';
**import** { JWT } **from** 'google-auth-library';
**import** { Readable } **from** 'stream';
export **default** defineComponent({
**async** run({ steps, $ }) {
**try**{
`**const** profilePicLinks=steps.get_profile_pic_links_of_Employees_1.$return_value`
`**const** credentials=JSON.parse(process.env.PROFILE_PIC_GDRIVE_SERVICEACCKEY)`
`**const** jwtClient = **new** JWT({`
`email: credentials.client_email,`
`key: credentials.private_key,`
`scopes: ['https://www.googleapis.com/auth/drive'],`
`});`
`**const** drive = google.drive({`
`version: 'v3',`
`auth: jwtClient,`
`});`
`**const** requests = [];`
**`async`** `**function** uploadImageToGoogleDrive(imageBuffer, fileName) {`
`**const** imageStream = **new** Readable();`
`imageStream.push(imageBuffer);`
`imageStream.push(**null**);`
`**const** media = {`
`mimeType: 'image/jpeg',`
`body: imageStream,`
`};`
`**const** request = {`
`resource: {`
`name: fileName,`
`parents: ['folder-id'],`
`},`
`media: media,`
`};`
**`return`** `**new** Promise((resolve, reject) => {`
`drive.files.create(request, (err, file) => {`
`**if** (err) {`
`reject(err);`
`} **else** {`
`resolve(file.data);`
`}`
`});`
`});`
`}`
`profilePicLinks.forEach(**async** (picLink) => {`
`**const** employeeNumber = picLink.employeeId;`
`**const** fileName = `${employeeNumber}.jpg`;`
`**const** s3Response = **await** axios.**get**(picLink.s3Link, { responseType: 'arraybuffer' });`
`**const** imageBuffer = Buffer.**from**(s3Response.data, 'binary');`
`**const** uploadPromise = uploadImageToGoogleDrive(imageBuffer, fileName);`
`requests.push(uploadPromise);`
`});`
`**await** Promise.all(requests);`
`console.log('All images uploaded successfully.');`
`} **catch** (error) {`
`console.error("Error:", error.message);`
`}`
},
});
it says all images uploaded successfully but giving a warning how to resolve this
This step was still trying to run code when the step ended. Make sure you promisify callback functions and await all Promises. (Reason: GetAddrInfoReqWrap, Learn more: Running asynchronous code in Node.js)
It seems like there might be some asynchronous code that’s not being awaited properly in your component. To resolve the warning, you should ensure that all Promises are awaited and callback functions are promisified. In your code, the issue might be related to the profilePicLinks.forEach loop. Since you’re using an async function within the loop, you should use Promise.all with map instead of forEach. Try updating your loop like this: ```
await Promise.all(profilePicLinks.map(async (picLink) => { const employeeNumber = picLink.employeeId; const fileName = ${employeeNumber}.jpg; const s3Response = await axios.get(picLink.s3Link, { responseType: ‘arraybuffer’ }); const imageBuffer = Buffer.from(s3Response.data, ‘binary’); const uploadPromise = uploadImageToGoogleDrive(imageBuffer, fileName); requests.push(uploadPromise); }));
It is giving
We got an unhandledRejection in this step. Please check that you are awaiting all your promises. The error was: Error: User rate limit exceeded.
How to resolve this?
It seems like you’re hitting the user rate limit for the Google Drive API. To handle this issue, you can implement a retry mechanism with exponential backoff in your uploadImageToGoogleDrive function. This will help you manage the rate limits by retrying the request after a certain delay if the rate limit is exceeded. First, you’ll need to update the uploadImageToGoogleDrive function to handle retries: ```
async function uploadImageToGoogleDrive(imageBuffer, fileName, retries = 3) { const imageStream = new Readable(); imageStream.push(imageBuffer); imageStream.push(null); const media = { mimeType: ‘image/jpeg’, body: imageStream, }; const request = { resource: { name: fileName, parents: [‘folder-id’], }, media: media, }; return new Promise(async (resolve, reject) => { try { const file = await drive.files.create(request); resolve(file.data); } catch (err) { if (err.message === ‘User rate limit exceeded’ && retries > 0) { setTimeout(() => { uploadImageToGoogleDrive(imageBuffer, fileName, retries - 1) .then(resolve) .catch(reject); }, 2 **** (3 - retries) * 1000); } else { reject(err); } } }); }
still the same error exists.
I apologize for the inconvenience. It seems the rate limit issue persists despite implementing the retry mechanism with exponential backoff. In this case, I recommend reaching out to Pipedream support for further assistance. They might be able to help you with raising the rate limit or provide alternative solutions. You can contact them at https://pipedream.com/support.