This topic was automatically generated from Slack. You can find the original thread here.
@pi I have a use case where I need to send approximately 1000 http trigger events at a time and make a api call to send the response through custom response action. How to reduce the time taken and get success for all events ?Is there any configuration?
I have a use case where I need to send approximately 1000 http trigger events at a time and make a api call to send the response through custom response action. How to reduce the time taken and get success for all events ?Is there any configuration?
To handle 1000 HTTP trigger events at a time and send a custom response, you can use the “Return HTTP Response” action in your Pipedream workflow. However, you should be aware of the rate limits imposed by Pipedream. By default, you can send an average of 10 requests per second to your HTTP trigger. Exceeding this limit may result in rate limiting and a 429 Too Many Requests response. To optimize the time taken and ensure success for all events, consider the following: 1. If you control the application sending requests, implement a retry mechanism with exponential backoff or a similar technique to handle rate-limited requests. 2. If you need a higher rate limit, you can request an increase by reaching out to the Pipedream Support team. This option is available for Professional, Teams, and Enterprise customers. 3. Optimize your workflow by minimizing the number of steps and API calls, and ensure that your workflow processes the events efficiently. Remember to use the “Return HTTP Response” action to send custom responses. You can find this action by searching for the HTTP / Webhook app from the step menu and selecting the “Return HTTP Response” action. For more information on Pipedream’s rate limits, visit Limits.
What if I change the execution control like throttling and concurrency control. Will it affect the response time and success rate of events?
Yes, adjusting the execution controls like throttling and concurrency control can affect the response time and success rate of events in your Pipedream workflow. 1. Concurrency control: By managing the number of workers (concurrent events), you can control how many events are executed in parallel. Increasing the number of workers can improve the response time, but it may also increase the chances of hitting rate limits or overloading your target API. On the other hand, setting the concurrency to 1 worker will ensure serialized, in-order execution, which may increase the overall processing time but guarantees that each event is processed only after the previous one is complete. 2. Throttling: By enabling throttling and configuring the limit and interval, you can control the rate at which events are processed. This can help you avoid hitting rate limits or overloading your target API, but it may also increase the overall processing time. You can manage these settings in the “Execution Controls” section of your Workflow Settings. Keep in mind that finding the optimal balance between concurrency, throttling, and response time may require some experimentation and fine-tuning based on your specific use case. For more information on concurrency and throttling, visit Concurrency and Throttling.