Hi, I need to get some results from a 3rd party API. The nature of the API requires me to implement a while-loop construct that pulls pages of results till I get to a certain result that flags that I don’t need to get any more results.
I ended up implementing it using recursion, which is quite cool and where I think PD really shines, since it allows you to run your own code.
This post is mainly to show the outcome of this work and to confirm whether this is the right approach or maybe there are better ways?
In short:
- I defined a recursive function (
fetchTicketsPage()
) - I make the initial call to the function with initial parameters and from there it’ll behave like a while loop
- When the exist condition is met (reached an old enough ticket) and the recursion ends, the results are returned and processed for generating the report this step is in charge of
Thoughts?
Thanks!
import { axios } from "@pipedream/platform"
export default defineComponent({
props: {
freshdesk: {
type: "app",
app: "freshdesk",
},
days_limit: {
type: "integer",
label: "Fresh Desk History Limit (in days)",
default: 120
}
},
async run({steps, $}) {
const fetchTicketsPage = async (tickets, page, date_limit, $) => {
const results = await axios($, {
url: `https://${this.freshdesk.$auth.domain}.freshdesk.com/api/v2/tickets?order_type=desc&per_page=100&page=${page}`,
auth: {
username: `${this.freshdesk.$auth.api_key}:X`,
password: ``,
},
})
//if no results then default to date_limit
const last_ticket_date = results.length > 0 ? new Date(results[results.length - 1].created_at) : date_limit
tickets.report = tickets.report.concat(results)
//recursive into next page of results until we hit date limit
if (last_ticket_date > date_limit) {
await new Promise(resolve => setTimeout(resolve, 500)); //sleep 500 millis to avoid FD error 429
tickets = await fetchTicketsPage(tickets, ++page, date_limit, $)
}
return tickets
}
const date_limit = new Date()
date_limit.setDate(date_limit.getDate() - this.days_limit)
const tickets = await fetchTicketsPage({ report: [] }, 1, date_limit, $)
$.export("reports", {
from_date: date_limit,
report: {
count: tickets.report.length,
oldest_date: tickets.report.length > 0 ? tickets.report[tickets.report.length - 1].created_at : "N/A",
},
})
}
})