What is the most efficient way to make up to N concurrent API requests, without exceeding my API's limit?

This topic was automatically generated from Slack. You can find the original thread here.

Top Flight Assistance : I want to articulate this thought out loud and see if I’m going about this in the most efficient way. I need to trigger a function inside Zoho CRM that runs about 100 lines of code. I need to trigger this function for a thousand individual records.

I can not run the function within Zoho because they have an execution limit of 5000 lines.

Additionally, they have an API limit of 20 concurrent calls.

The function I am triggering takes about 3 seconds to run. My thought is to trigger a web hook and pipe dream with the needed payload to then invoke the function in CRM with said payload. But in order to avoid more than 20 concurrent calls. I will increment each call by a 4 second delay using pipe dreams built in scheduler.

Is there a more efficient way to accomplish this?

Dylan Sather (Pipedream) : thanks for the detail. Have you seen our concurrency / throttling controls? This was designed for use cases like yours. Using those settings should allow you to control the number of invocations your workflow runs within a given time period, allowing you to manage your execution given the API rate limits.

Let us know if that works for you!

Top Flight Assistance : That is exactly what I needed. Thank you.

Dylan Sather (Pipedream) : Great