This topic was automatically generated from Slack. You can find the original thread here.
how do I use bottleneck to restrict a max of 10 http requests per second?
In Pipedream, you can control the rate of workflow execution using execution controls. To throttle your workflow execution, follow these steps: 1. Go to your workflow settings. 2. Enable throttling. 3. Configure the limit and interval. For your specific case, set the limit to 10 and the interval to 1 second. This will allow a maximum of 10 events to be processed every second. Please note that Pipedream’s execution controls are designed to manage the rate of workflow execution and not specifically the number of HTTP requests. However, by controlling the rate of workflow execution, you can indirectly control the number of HTTP requests made within the workflow. For more information on throttling and concurrency, you can refer to the Pipedream documentation: Concurrency and Throttling
cool, but how do I do it in a custom node.js step that uses the bottleneck npm package?
To use the Bottleneck npm package in a custom Node.js step within a Pipedream workflow, you can follow these steps: 1. Add a new Node.js code step in your workflow. 2. Import the Bottleneck package at the top of your code step: import Bottleneck from "bottleneck"; 3. Create a new Bottleneck instance with the desired rate limit configuration. In your case, you want to limit the rate to 10 requests per second: const limiter = new Bottleneck({ minTime: 1000 / 10, // 1000 ms (1 second) divided by 10 requests }); 4. Use the limiter.schedule() function to schedule your HTTP requests. This function accepts a callback that contains the code for your HTTP request. For example, if you’re using the axios package to make HTTP requests, your code would look like this: ```
import Bottleneck from “bottleneck”; import { axios } from “@pipedream/platform”; const limiter = new Bottleneck({ minTime: 1000 / 10, }); export default defineComponent({ async run() { const makeRequest = async () => { const response = await axios(this, { method: “GET”, url: “https://example.com/api/endpoint”, }); return response; }; const throttledRequest = await limiter.schedule(makeRequest); return throttledRequest; }, });