This topic was automatically generated from Slack. You can find the original thread here.
Hey guys,
I am coming from Make.com and have limited coding experience. Is there a way to create a get request and have it process through all of the pages that come through paginated? I did this in a couple different steps in Make.com but I have been struggling trying to make this work for the last few hours.
Try asking for help.
Give him as much details as possible to help him come up with the right solution.
First of all, it would be helpful to know which API you’re working with.
I’m sorry, but your message seems to be incomplete. Could you please provide more details about the issue you’re facing? This will help me understand your problem better and provide a more accurate solution.
Thanks .
@U05A7KLNALC I am using Jetbuilt’s API and I am trying to GET all projects in the contract, install, and review stage. I am able to get the first 25 which is on the first page but I can’t figure out how to get all of the projects to come through as objects within one workflow request.
In Pipedream, you can create a workflow that makes a GET request to the Jetbuilt API and processes through all the paginated pages. Here’s a simple example of how you could do this: import { axios } from "@pipedream/platform"; export default defineComponent({ async run({ $ }) { let page = 1; let projects = []; while (true) { const response = await axios($, { url: `https://api.jetbuilt.com/projects?stage=contract,install,review&page=${page}`, headers: { Authorization: `Bearer YOUR_JETBUILT_API_KEY`, }, }); if (response.length === 0) { break; } projects = projects.concat(response); page++; } return projects; }, }); Please replace YOUR_JETBUILT_API_KEY with your actual API key. This code will make a GET request to the Jetbuilt API for each page until it gets an empty response, indicating there are no more pages. It concatenates all the projects into a single array which is returned from the workflow. Please note that you might need to adjust the URL and headers according to the Jetbuilt API documentation. This is just a basic example and might not work out of the box.
That looks like a pretty good start! 
If you run into any issues, you can tell Pi about them, and he’ll try to come up with a solution.
I am trying this but I am getting a timeout error. What am I missing?
import { axios } from "@pipedream/platform";
export default defineComponent({
async run({ $ }) {
let page = 1;
let projects = [];
while (true) {
const response = await axios($, {
url: `https://app.jetbuilt.com/api/projects?page=${page}`,
headers: {
Authorization: `Token token=************`,
},
});
if (response.length === 0) {
break;
}
projects = projects.concat(response);
page++;
}
return projects;
},
});
It might be taking more than 30 seconds to get all the pages.
Or maybe the code is stuck in an infinite loop.
The timeout error might be due to the API taking too long to respond. Pipedream has a default timeout of 30000ms (30 seconds) for each step in a workflow. If a step takes longer than this, it will be stopped and a timeout error will be thrown. However, there’s another potential issue in your code. When you’re using the axios function from @pipedream/platform, the response is the data itself, not an object with a data property. So when you’re checking response.length === 0, it might not be working as expected if the response is not an array. Here’s a revised version of your code: import { axios } from "@pipedream/platform"; export default defineComponent({ async run({ $ }) { let page = 1; let projects = []; while (true) { const response = await axios($, { url: `https://app.jetbuilt.com/api/projects?page=${page}`, headers: { Authorization: `Token token=************`, }, }); // If response is not an array or is an empty array, break the loop if (!Array.isArray(response) || response.length === 0) { break; } projects = projects.concat(response); page++; } return projects; }, }); If you’re still experiencing the timeout issue, you might need to increase the timeout for the axios request. You can do this by adding a timeout property to the axios config: const response = await axios($, { url: `https://app.jetbuilt.com/api/projects?page=${page}`, headers: { Authorization: `Token token=************`, }, timeout: 60000, // Increase timeout to 60 seconds }); Please replace ************ with your actual token. If you’re still having issues, please let me know!
The terminating condition if (response.length === 0) { might never be met.
: According to the JetBuilt API documentation, they use a Link header to provide URLs for the next page.
Please rewrite the code to use that header for the pagination.
Sure, I can help with that. Here’s how you can modify your code to use the Link header for pagination: import { axios } from "@pipedream/platform"; export default defineComponent({ async run({ $ }) { let projects = []; let url = "https://app.jetbuilt.com/api/projects?page=1"; while (url) { const response = await axios($, { url, headers: { Authorization: `Token token=************`, }, }); projects = projects.concat(response); // Check if there is a next page const linkHeader = response.headers.link; const nextLink = linkHeader.split(',').find(link => link.includes('rel="next"')); if (nextLink) { // Extract the URL of the next page url = nextLink.split(';')[0].slice(1, -1); } else { url = null; // No more pages } } return projects; }, }); This code will extract the URL of the next page from the Link header and use it for the next request. When there is no next link, it means there are no more pages and the loop will stop. Please replace ************ with your actual token. If you’re still having issues, please let me know!
: Try that 
TypeError
Cannot read properties of undefined (reading ‘link’)
Try with a capital L
response.headers.Link;
Same error