This topic was automatically generated from Slack. You can find the original thread here.
I am trying to iterate over an array of files that I need to upload, and I keep running out of memory and/or Pipedream doesn’t respond or timeout, and I’m wondering if it could be due to too many streams being opened and not closed or disposed of. Is there a recommended structure to looping over opening a remote file from URL, then posting it via axios with FormData?
To handle the iteration over an array of files for upload without running out of memory or encountering timeouts, it’s crucial to manage streams properly and ensure they are closed after use. Here’s a recommended structure using axios and FormData for uploading files, which includes proper handling of streams to avoid memory issues: import { axios } from "@pipedream/platform"; import got from "got"; import FormData from "form-data"; import { pipeline } from "stream/promises"; export default defineComponent({ async run({ steps, $ }) { // Example array of file URLs to upload const fileUrls = [ "https://example.com/file1.pdf", "https://example.com/file2.pdf", // Add more file URLs as needed ]; for (const fileUrl of fileUrls) { // Create a stream for downloading the file const downloadStream = got.stream(fileUrl); // Create a FormData instance to hold the file data const formData = new FormData(); formData.append("file", downloadStream); // Get the headers for the multipart/form-data request const headers = formData.getHeaders(); // Configure the request for uploading the file const config = { method: "POST", url: "https://example.com/upload", // Replace with your target upload URL headers, data: formData, }; // Perform the upload const response = await axios($, config); // Optionally, handle the response console.log(`Uploaded file from ${fileUrl}:`, response); // Ensure the stream is properly closed after each iteration await pipeline(downloadStream, formData); } }, }); This code snippet demonstrates how to iterate over an array of file URLs, download each file using a stream, and upload it using axios with FormData. The use of pipeline from the stream/promises module ensures that streams are properly managed and closed after each iteration, which helps prevent memory leaks and other issues related to open streams. Remember to replace "https://example.com/file1.pdf", "https://example.com/file2.pdf", and "https://example.com/upload" with your actual file URLs and target upload URL.