How to Create a Get Request to Process Through All Paginated Pages with Limited Coding Experience?

: How to extract response headers when using import { axios } from "@pipedream/platform";?

Looks like response.headers returns undefined, which is weird.

When using axios from @pipedream/platform, the response headers can be accessed by setting returnFullResponse to true in the axios request configuration. This will return the full axios response object, which includes the headers property. Here’s an example: import { axios } from "@pipedream/platform"; export default defineComponent({ async run({ $ }) { const response = await axios($, { url: "https://api.example.com", returnFullResponse: true, // This will return the full axios response }); const headers = response.headers; // Access the headers from the response return headers; }, }); In this example, response.headers is an object where each key is a header name and each value is the header value. Please replace "https://api.example.com" with your actual URL.

Ah, that explains it! :point_up_2:

Need to add returnFullResponse: true,

Now we got this

TypeError
Cannot read properties of undefined (reading ‘split’)

Sorry for being a total newbie

Try doing console.log(response.headers), and paste the result here.

{
  date: 'Tue, 21 Nov 2023 02:59:09 GMT',
  'content-type': 'application/json; charset=utf-8',
  'transfer-encoding': 'chunked',
  connection: 'close',
  'cf-ray': '8295accf997e39b0-IAD',
  'cf-cache-status': 'DYNAMIC',
  'cache-control': 'max-age=0, private, must-revalidate',
  etag: 'W/"1f44af8fd71ce548d7e6f9fa1105e790"',
  link: '<https://app.jetbuilt.com/api/projects/442172/purchasing?page=12>; rel="last", <https://app.jetbuilt.com/api/projects/442172/purchasing?page=2>; rel="next"',
  'set-cookie': [
    '_jetbuilt_session=97OED6YcMrddLJKB9HvaC3v3i402Wy86I5AEw708RKaHRgszuiTugqWBXc%2FIj7QO8aTZBo3O82SNfmqDTM5UYVQn3JItGudAzGZ7Ge0ySzYj9S75hLVxKOaFBoA7j22Ovsp7Jifkkww%2BrmcSrvVZrAO7eyJRofB2c8kzgVjEEx%2FG5I1ZguxPL0ycyHMOy0daK4zkd1CZNRiqB%2FIgbtFZypDMHQ%2FKRhNsKA%3D%3D--qil0f%2FYu2HGInSN9--IOA8gNp3JqNwWUFQuYmAxg%3D%3D; path=/; secure; HttpOnly; SameSite=Lax'
  ],
  'strict-transport-security': 'max-age=31536000; includeSubDomains; preload',
  vary: 'Accept-Encoding, Origin',
  via: '1.1 vegur',
  'cf-apo-via': 'origin,host',
  nel: '{"report_to":"heroku-nel","max_age":3600,"success_fraction":0.005,"failure_fraction":0.05,"response_headers":["Via"]}',
  'referrer-policy': 'strict-origin-when-cross-origin',
  'report-to': '{"group":"heroku-nel","max_age":3600,"endpoints":[{"url":"https://nel.heroku.com/reports?ts=1700535549&sid=e11707d5-02a7-43ef-b45e-2cf4d2036f7d&s=b3tTy0G%2FqBBGlkCSpnefr329RqonC1B8TqpDrxdriYw%3D"}]}',
  'reporting-endpoints': 'heroku-nel=https://nel.heroku.com/reports?ts=1700535549&sid=e11707d5-02a7-43ef-b45e-2cf4d2036f7d&s=b3tTy0G%2FqBBGlkCSpnefr329RqonC1B8TqpDrxdriYw%3D',
  'x-content-type-options': 'nosniff',
  'x-download-options': 'noopen',
  'x-frame-options': 'SAMEORIGIN',
  'x-per-page': '25',
  'x-permitted-cross-domain-policies': 'none',
  'x-request-id': '72612a15-51ba-40b4-a2e2-e744d76b0857',
  'x-runtime': '0.086309',
  'x-total-count': '290',
  'x-xss-protection': '0',
  server: 'cloudflare'
}

Ok, so link is there! :+1:

But back to a lowercase l :sweat_smile:

More progress!

TypeError
Invalid URL

weird :thinking_face:

console.log the url!

https://app.jetbuilt.com/api/projects/442172/purchasing?page=1

That is all it gave me

Try putting the console.log at the top of the code, before the API call (const response = ...).

https://app.jetbuilt.com/api/projects/442172/purchasing?page=1

<https://app.jetbuilt.com/api/projects/442172/purchasing?page=2>; rel="next

ah, weird… :thinking_face:

So the important bits are:

link: '<https://app.jetbuilt.com/api/projects/442172/purchasing?page=12>; rel="last", <https://app.jetbuilt.com/api/projects/442172/purchasing?page=2>; rel="next"'

      // Check if there is a next page
      const linkHeader = response.headers.link;
      const nextLink = linkHeader.split(',').find(link => link.includes('rel="next"'));

      if (nextLink) {
        // Extract the URL of the next page
        url = nextLink.split(';')[0].slice(1, -1);
      }

Try nextLink.split(';')[0].trim().slice(1, -1);

It works in my console: