This topic was automatically generated from Slack. You can find the original thread here.
Hello
I’m a new user to PipeDream (have been a programmer for a very long time). I’m trying to do something which seems simple, which is to upload a csv file into my database, which is Supabase. I have tried a great many approaches to this and have found it way mor challenging than I would have expected. First up I could not get the file that I’m uploading to work from Dropbox despite trying several different URL formats. I abandoned that and went to Google Drive, which works although isn’t my preferred choice. However I do not see a straightforward mechanism for this pretty basic need. I have written a node.js program to extract rows from the csv and put them into a REST/POST command, but this seems extremely clunk and is naturally non-performative. Is there a simple way to do this that does not involved looping over all the csv rows and doing a REST call on each? This seems like something that a lot of people would use PipeDream for, i.e. automating through triggers the upload of a csv file. Perhaps the issue is Supabase as opposed to another DB, or perhaps I’m missing something obvious. Any guidance or help is appreciated. m
I’m a new user to PipeDream (have been a programmer for a very long time). I’m trying to do something which seems simple, which is to upload a csv file into my database, which is Supabase. I have tried a great many approaches to this and have found it way mor challenging than I would have expected. First up I could not get the file that I’m uploading to work from Dropbox despite trying several different URL formats. I abandoned that and went to Google Drive, which works although isn’t my preferred choice. However I do not see a straightforward mechanism for this pretty basic need. I have written a node.js program to extract rows from the csv and put them into a REST/POST command, but this seems extremely clunk and is naturally non-performative. Is there a simple way to do this that does not involved looping over all the csv rows and doing a REST call on each? This seems like something that a lot of people would use PipeDream for, i.e. automating through triggers the upload of a csv file. Perhaps the issue is Supabase as opposed to another DB, or perhaps I’m missing something obvious. Any guidance or help is appreciated. m
To upload a CSV file into your Supabase database using Pipedream, you can streamline the process by using batch operations instead of making individual REST calls for each row. Here’s a step-by-step guide to help you achieve this: 1. Upload the CSV File: - If you’re using Google Drive, you can use the “Google Drive - New File” trigger to detect when a new CSV file is uploaded. - If you prefer Dropbox, you can use the “Dropbox - New File” trigger. 2. Read the CSV File: - Use a code step to read the CSV file. You can use the csv-parser package to parse the CSV data. 3. Batch Insert into Supabase: - Instead of inserting each row individually, you can batch the rows and insert them into Supabase in one go. Supabase supports bulk inserts, which can be more efficient. Here’s an example of how you can set up a Pipedream workflow to achieve this: import { axios } from "@pipedream/platform"; import csv from "csv-parser"; import { Readable } from "stream"; export default defineComponent({ props: { supabase: { type: "app", app: "supabase", }, fileUrl: { type: "string", label: "CSV File URL", }, }, async run({ steps, $ }) { // Fetch the CSV file const response = await axios($, { url: this.fileUrl, responseType: "stream", }); // Parse the CSV file const rows = []; const stream = response.data.pipe(csv()); for await (const row of stream) { rows.push(row); } // Batch insert into Supabase const url = `https://${this.supabase.$auth.subdomain}.[supabase.co/rest/v1/your_table_name`;](http://supabase.co/rest/v1/your_table_name`;) const headers = { Authorization: `Bearer ${this.supabase.$auth.service_key}`, apikey: `${this.supabase.$auth.service_key}`, "Content-Type": "application/json", }; // Adjust batch size as needed const batchSize = 100; for (let i = 0; i < rows.length; i += batchSize) { const batch = rows.slice(i, i + batchSize); await axios($, { method: "post", url, headers, data: batch, }); } return { success: true }; }, }); ### Key Points: - Batch Size: Adjust the batchSize variable as needed to optimize performance. - CSV Parsing: The csv-parser package is used to parse the CSV data. Ensure your CSV structure matches the expected format. - Supabase Table: Replace your_table_name with the actual table name in your Supabase database. This approach should help you efficiently upload CSV data to Supabase without making individual REST calls for each row. If you encounter any issues or need further assistance, feel free to reach out to Pipedream Support.
Yes, although something like “we are about to release an integration that will do this in one step” would have been preferred. In any event thanks for getting back to me and I will see how this works. m
I’m working with Airtable also, but I haven’t explored what Pipedream can do with it yet. If I have any feedback I will let you know. I have a question about the above code, which is that I’m not sure where the “${this.supabase.$auth.subdomain}” comes from. Is this supposed to be a variable defined in Pipedream? I can certainly insert the appropriate string into the code, but deriving it from the environment would probably be more elegant. I’m not precisely sure how the “props” gets used but I’m guessing that is reaching outside the code block to somewhere. Probably should read the documentation more closely, but if you are able to steer me that would be great.
Unfortunately this code (even after I stripped out the Batch insert code) errors with: "This step was still trying to run code when the step ended. Make sure you promisify callback functions and await all Promises. (Reason: TLSSocket, "
It seems like you’re encountering an error related to asynchronous code execution in a Node.js code step on Pipedream. This error typically occurs when asynchronous operations are not properly awaited, causing the step to end before the operations are complete. Here are some steps you can take to resolve this issue: 1. Ensure All Promises Are Awaited: Make sure that every asynchronous operation in your code is awaited. This includes any function that returns a Promise. Use the await keyword before calling these functions. 2. Promisify Callback Functions: If you’re using functions that rely on callbacks (common in older Node.js libraries), you need to wrap these in Promises. You can use the util.promisify function from Node.js to convert callback-based functions to return Promises. Here is an example of how you might promisify a callback function: const { promisify } = require('util'); const someCallbackFunction = promisify(originalCallbackFunction); async function run() { try { const result = await someCallbackFunction(args); console.log(result); } catch (error) { console.error(error); } } 3. Check for Unresolved Promises: Review your code to ensure there are no unresolved Promises. Every Promise should be handled with either await or .then() and .catch(). For more detailed guidance, you can refer to the Asynchronous section of the Node.js documentation. If you continue to experience issues, you might want to visit Pipedream support for further assistance.
In any event thanks for getting back to me and I will see how this works. m