Can I Download and Pass Large Volumes of Data in One Step in Node.js Using Pipedream?

Help
1

This topic was automatically generated from Slack. You can find the original thread here.

I just ran into this: GitHub

I’m using a Node.js step to download many pages of data. Each record is about 500 bytes. I currently need to get 85 pages @ 1000 records, which is 44MB of data. It sounds as if I won’t be able to download all of that in one step and pass it on to another step to be written to a file, right? So that means I have to change the step that’s downloading the data so that it tracks the data volume and writes it to file from time to time, yeah?

2

You should be able to gradually stream the data into a file as you retrieve it.