Hi @pillcollin! The good thing about this task, is that the data is in CSV format, which means it can be streamed, line by line. Here’s an approach I’d explore:
- Upload file to workflow as an attachment.
- Verify rate limits per API key and per account, consider any other automations/apps using the same API key
Trigger: runs a timer
- Read 1 line from the csv file
- Send a POST request to the external API
$checkpoint with latest line number
Repeat up to a maximum number of lines that won’t hit rate limits or throttling. Also each time the workflow runs, it should refer to
$checkpoint to get the current line number.
If you’re really concerned about processing too many lines, you can process them one at a time. This would make the workflow timer the singular controller of when submissions are sent to the external API.
Once the all lines in the CSV have been processed, it could be helpful to send yourself an email. I’m guessing you’ll only want to read the CSV file one time. If so, you can include a reminder to disable the workflow in the email.