This topic was automatically generated from Slack. You can find the original thread here.
Hi is there any recommendation for running a workflow a set number of times and passing in that number to use as the page number in the api call?
Hi thanks for reaching out. I’m going to ask our new Q&A not to see how it does, and will supplement that answer myself too
Hi is there any recommendation for running a workflow a set number of times and passing in that number to use as the page number in the api call?
To run a workflow a set number of times and pass the iteration number as the page number in an API call, you can use a loop within your workflow. Here’s an example of how you can achieve this using a Pipedream component: export default defineComponent({ props: { totalPages: { type: "integer", label: "Total Pages", }, apiUrl: { type: "string", label: "API URL", }, }, async run({ $ }) { for (let pageNumber = 1; pageNumber <= this.totalPages; pageNumber++) { const response = await axios($, { method: "GET", url: this.apiUrl, params: { page: pageNumber, }, }); // Process the response data as needed console.log(response); } }, }); In this example, you can set the totalPages prop to the number of times you want to run the workflow, and the apiUrl prop to the API endpoint you want to call. The loop will iterate from 1 to totalPages, making an API call with the current page number as a parameter.
Let me know if that helps or if you’re trying to do something different.
i think that would work! can you run a workflow without having a trigger as I would only need to run this once to get historical data and then ongoing i will need another workflow with event triggers
Yes, in fact triggers are optional. Try adding just a code step to a workflow and deploy it without a trigger. You’ll see a “Run now” option in place of the trigger that you can click to run the workflow manually.
You can technically do all of the historical data pulls when you’re developing the workflow. Since you can test any code / pre-built actions in development mode, I frequently run ad hoc code in a development workflow to initialize anything / pull historical data, then just write the workflow for the future case (with the standard trigger for new events) and deploy that.
Let me know if that helps
ok i can give that a go. I need to run it 2500 times though to get all the historical data!
Do you simply need to iterate through 2500 pages (i.e. 2500 distinct API requests) and store those somewhere? Or are you doing something more complex to transform / aggregate the data?
I need to loop through the endpoint 2500 times as thats how many pages of data we have for 1 year. Then i need to get the response and map it differently to then send data new formatted data to a different endpoint
Got it. So another approach is to write a workflow with an HTTP trigger that accepts a page number. Then the workflow fetches data for that page and does whatever processing you need to do.
Another workflow can call that workflow 2500 times: one for each page. So you’d just write a simple for loop that iterated over all page numbers and send an HTTP request to the first workflow with that page number.
Would that work?
I think that makes sense!