This topic was automatically generated from Slack. You can find the original thread here.
I’m trying to add an event of creating a new file in Google Drive and it doesn’t work. I’m trying to upload a csv as a spreadsheet. Tried from a local file and from a URL, both didn’t work. This is the error message I’m getting:
Bad Request
DETAILS
at Gaxios._request (/tmp/pdg/dist/code/62625b4914a6b775586a93a92193f6c4af3b0d7ee4e5cdb7690a8b7c76a3950e/node_modules/.pnpm/gaxios@6.1.1/node_modules/gaxios/build/src/gaxios.js:141:23)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at OAuth2Client.requestAsync (/tmp/pdg/dist/code/62625b4914a6b775586a93a92193f6c4af3b0d7ee4e5cdb7690a8b7c76a3950e/node_modules/.pnpm/google-auth-library@9.2.0/node_modules/google-auth-library/build/src/auth/oauth2client.js:379:18)
at Object.createFile (file:///tmp/pdg/dist/code/62625b4914a6b775586a93a92193f6c4af3b0d7ee4e5cdb7690a8b7c76a3950e/code/google_drive.app.mjs:1014:9)
at Object.run (file:///tmp/pdg/dist/code/62625b4914a6b775586a93a92193f6c4af3b0d7ee4e5cdb7690a8b7c76a3950e/code/actions/create-file/create-file.mjs:202:18)
at null.executeComponent (/var/task/launch_worker.js:267:22)
at MessagePort.messageHandler (/var/task/launch_worker.js:764:28)
I’m here to help, but I need a bit more context to understand Itamar’s suggestion. Could you please provide more details or clarify what specific suggestion from Itamar you’re referring to? This will help me give you the most accurate and helpful response.
I’m trying to add an event of creating a new file in Google Drive and it doesn’t work. I’m trying to upload a csv as a spreadsheet. Tried from a local file and from a URL, both didn’t work. This is the error message I’m getting:
Bad Request
DETAILS
at Gaxios._request (/tmp/pdg/dist/code/62625b4914a6b775586a93a92193f6c4af3b0d7ee4e5cdb7690a8b7c76a3950e/node_modules/.pnpm/gaxios@6.1.1/node_modules/gaxios/build/src/gaxios.js:141:23)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at OAuth2Client.requestAsync (/tmp/pdg/dist/code/62625b4914a6b775586a93a92193f6c4af3b0d7ee4e5cdb7690a8b7c76a3950e/node_modules/.pnpm/google-auth-library@9.2.0/node_modules/google-auth-library/build/src/auth/oauth2client.js:379:18)
at Object.createFile (file:///tmp/pdg/dist/code/62625b4914a6b775586a93a92193f6c4af3b0d7ee4e5cdb7690a8b7c76a3950e/code/google_drive.app.mjs:1014:9)
at Object.run (file:///tmp/pdg/dist/code/62625b4914a6b775586a93a92193f6c4af3b0d7ee4e5cdb7690a8b7c76a3950e/code/actions/create-file/create-file.mjs:202:18)
at null.executeComponent (/var/task/launch_worker.js:267:22)
at MessagePort.messageHandler (/var/task/launch_worker.js:764:28)
I’m sorry to hear that you’re experiencing issues with creating a new file in Google Drive. The error message “Bad Request” typically indicates that the input parameters or the structure of the request is not as expected by the Google Drive API. To assist you better, I would need to see the specific code you’re using to create the file in Google Drive. However, I can provide you with a general example of how to upload a CSV file as a Google Sheets spreadsheet using Pipedream. Please ensure that you have the correct OAuth scopes and permissions set for your Google Drive integration. Here’s an example Pipedream component that uploads a CSV file as a Google Sheets spreadsheet: import { axios } from "@pipedream/platform"; export default defineComponent({ props: { google_drive: { type: "app", app: "google_drive", }, }, async run({ steps, $ }) { const fileMetadata = { name: 'my-new-spreadsheet', mimeType: 'application/vnd.google-apps.spreadsheet', }; const media = { mimeType: 'text/csv', body: fs.createReadStream('/path/to/file.csv'), // Replace with the path to your CSV file }; return await axios($, { method: "post", url: "https://www.googleapis.com/upload/drive/v3/files", params: { uploadType: "multipart", }, headers: { Authorization: `Bearer ${this.google_drive.$auth.oauth_access_token}`, "Content-Type": "multipart/related", }, data: { // Construct multipart body with metadata and media }, }); }, }); Please replace '/path/to/file.csv' with the actual path to your CSV file or the content of the CSV if you’re uploading it from a URL. Also, ensure that the google_drive prop is correctly configured with your Google Drive app. If you continue to face issues, could you please provide the code snippet you’re using to create the file, or any additional details about the steps you’re taking? This will help me provide a more accurate solution. If you need further assistance, you can also visit Pipedream Support for more help.
To use a value from a previous step in your Pipedream component code, you can access the step’s exported data using the steps object provided in the run method. Here’s an example of how you might do this: export default defineComponent({ async run({ steps, $ }) { // Accessing data from a previous step named `step1` const previousStepData = steps.step1.exportedData; // Now you can use `previousStepData` in your current step // For example, if you want to use it in an API request: return await axios($, { url: 'https://api.example.com/data', method: 'POST', data: { value: previousStepData, }, }); }, }); In this example, step1 is the name of the previous step, and exportedData is the name of the data that was exported from that step. Make sure to replace these with the actual step name and export name from your workflow. Remember that the data returned from steps must be JSON-serializable, and you can use $.export to export specific data at any point in a step or return the data at the end of the step.
I get this error: ReferenceError
fs is not defined DETAILS
at Object.run (file:///tmp/__pdg__/dist/code/16b92a677cceca75bed0bc1a40b0f5f34ae61995ee81d0569f80753a2f295e8a/component.mjs:18:13)
at null.executeComponent (/var/task/launch_worker.js:267:53)
at MessagePort.messageHandler (/var/task/launch_worker.js:764:28)
The error you’re encountering, “ReferenceError: fs is not defined,” suggests that you’re trying to use the fs module, which is a part of Node.js core modules, but it has not been imported into your Pipedream component. In a Pipedream Node.js code step, you don’t have access to the fs module directly because the environment is serverless and doesn’t have a persistent filesystem. However, you can use the /tmp directory for temporary file storage within a single execution of a step. If you need to read or write files within a Pipedream workflow, you can do so using the @pipedream/platform package, which provides a way to work with the filesystem in the /tmp directory. Here’s an example of how you might write to a file: import { axios, $ } from "@pipedream/platform"; export default defineComponent({ async run({ steps }) { // Writing to a file in the /tmp directory await $.writeFile('/tmp/myfile.txt', 'Hello, world!'); // Reading from a file in the /tmp directory const content = await $.readFile('/tmp/myfile.txt', 'utf8'); // Use the content as needed console.log(content); }, }); In this example, $.writeFile and $.readFile are methods provided by the @pipedream/platform package that allow you to interact with the filesystem in the /tmp directory. Please adjust your code to remove the fs module and use the methods provided by @pipedream/platform for file operations. If you need further assistance or have more details to share about what you’re trying to achieve, please let me know!
I’m sorry for the confusion earlier. However, in the Pipedream serverless environment, you cannot import the fs module as you would in a traditional Node.js environment because the filesystem is not available in the same way. Pipedream workflows are designed to be stateless, and the only writable filesystem path in the Pipedream execution environment is /tmp. If you need to work with files within a Pipedream workflow, you can use the built-in methods provided by the @pipedream/platform package, such as $.writeFile and $.readFile, to interact with the /tmp directory. If you are trying to use fs for a different purpose, could you please provide more details on what you are trying to achieve? This will help me assist you better.
To construct a multipart body with metadata and media in a Pipedream workflow, you would typically use a Node.js code step with the axios library to make a multipart/form-data POST request. Here’s an example of how you might do that: import { axios } from "@pipedream/platform"; export default defineComponent({ async run({ steps, $ }) { // Construct the form data const formData = new $$.FormData(); // Append metadata part formData.append('metadata', JSON.stringify({ name: "file_name", description: "file_description", })); // Append file part // Assuming `fileContent` is a variable holding the binary content of the file // and `fileName` is the name of the file formData.append('file', $$.file(fileContent, fileName)); // Make the POST request const response = await axios($, { method: "post", url: "https://your.api/endpoint", headers: { "Content-Type": "multipart/form-data", }, data: formData, }); return response; }, }); In this example, replace https://your.api/endpoint with the actual URL you’re posting to. Also, replace fileContent and fileName with the actual content and name of the file you’re uploading. The metadata is a JSON object that you can customize based on the API’s requirements. Please note that you’ll need to adjust this example to fit the specific API you’re working with, as different APIs may require different fields or formats for the metadata and media parts of the request. If you need further assistance with a specific API, please provide more details, and I’ll be happy to help you construct the appropriate request.
at Function.DelayedStream.create (/tmp/__pdg__/dist/code/02782f3e2d7a8bf0b9cd66f36c7eb7ec32fda1072666c8dd3142166ed9602fda/node_modules/.pnpm/delayed-stream@1.0.0/node_modules/delayed-stream/lib/delayed_stream.js:33:10)
at FormData.CombinedStream.append (/tmp/__pdg__/dist/code/02782f3e2d7a8bf0b9cd66f36c7eb7ec32fda1072666c8dd3142166ed9602fda/node_modules/.pnpm/combined-stream@1.0.8/node_modules/combined-stream/lib/combined_stream.js:45:37)
at FormData.FormData.append (/tmp/__pdg__/dist/code/02782f3e2d7a8bf0b9cd66f36c7eb7ec32fda1072666c8dd3142166ed9602fda/node_modules/.pnpm/form-data@4.0.0/node_modules/form-data/lib/form_data.js:75:3)
at Object.run (file:///tmp/__pdg__/dist/code/02782f3e2d7a8bf0b9cd66f36c7eb7ec32fda1072666c8dd3142166ed9602fda/component.mjs:27:14)
at null.executeComponent (/var/task/launch_worker.js:267:22)
at MessagePort.messageHandler (/var/task/launch_worker.js:764:28)
at Function.DelayedStream.create (/tmp/__pdg__/dist/code/02782f3e2d7a8bf0b9cd66f36c7eb7ec32fda1072666c8dd3142166ed9602fda/node_modules/.pnpm/delayed-stream@1.0.0/node_modules/delayed-stream/lib/delayed_stream.js:33:10)
at FormData.CombinedStream.append (/tmp/__pdg__/dist/code/02782f3e2d7a8bf0b9cd66f36c7eb7ec32fda1072666c8dd3142166ed9602fda/node_modules/.pnpm/combined-stream@1.0.8/node_modules/combined-stream/lib/combined_stream.js:45:37)
at FormData.FormData.append (/tmp/__pdg__/dist/code/02782f3e2d7a8bf0b9cd66f36c7eb7ec32fda1072666c8dd3142166ed9602fda/node_modules/.pnpm/form-data@4.0.0/node_modules/form-data/lib/form_data.js:75:3)
at Object.run (file:///tmp/__pdg__/dist/code/02782f3e2d7a8bf0b9cd66f36c7eb7ec32fda1072666c8dd3142166ed9602fda/component.mjs:27:14)
at null.executeComponent (/var/task/launch_worker.js:267:22)
at MessagePort.messageHandler (/var/task/launch_worker.js:764:28)
The error you’re encountering, source.on is not a function, suggests that there’s an issue with the stream you’re trying to append to the FormData object. When using form-data with axios in a Pipedream component, you need to ensure that the data you’re appending is in the correct format and that you’re handling streams properly. In your code, you’re converting the file read from the filesystem to a string before appending it to the FormData. However, form-data expects a stream or a buffer for file uploads. You should append the file as a stream directly, without converting it to a string. Here’s how you can modify your code to append the file as a stream: import { axios } from "@pipedream/platform"; import FormData from "form-data"; import fs from "fs"; export default defineComponent({ props: { google_drive: { type: "app", app: "google_drive", }, }, async run({ steps, $ }) { const fileMetadata = { name: steps.python.$return_value.file_name, mimeType: 'application/vnd.google-apps.spreadsheet', }; const media = { mimeType: 'text/csv', body: fs.createReadStream('/tmp/items.csv'), }; const formData = new FormData(); formData.append('metadata', JSON.stringify(fileMetadata), { contentType: 'application/json', }); formData.append('file', media.body, { filename: 'items.csv', contentType: media.mimeType, }); const headers = formData.getHeaders(); headers.Authorization = `Bearer ${this.google_drive.$auth.oauth_access_token}`; return await axios($, { method: "post", url: "https://www.googleapis.com/upload/drive/v3/files", params: { uploadType: "multipart", }, headers: headers, data: formData, }); }, }); Please note the following changes: 1. The body property of the media object now uses fs.createReadStream to create a readable stream from the file. 2. The formData.append for the file now passes media.body directly. 3. The filename in the formData.append for the file does not need the full path, just the file name. 4. The headers for the axios request are obtained using