Recursion in a step

Hi, I need to get some results from a 3rd party API. The nature of the API requires me to implement a while-loop construct that pulls pages of results till I get to a certain result that flags that I don’t need to get any more results.

I ended up implementing it using recursion, which is quite cool and where I think PD really shines, since it allows you to run your own code.

This post is mainly to show the outcome of this work and to confirm whether this is the right approach or maybe there are better ways?

In short:

  • I defined a recursive function (fetchTicketsPage())
  • I make the initial call to the function with initial parameters and from there it’ll behave like a while loop
  • When the exist condition is met (reached an old enough ticket) and the recursion ends, the results are returned and processed for generating the report this step is in charge of

Thoughts? :slight_smile:

Thanks!

import { axios } from "@pipedream/platform"
export default defineComponent({
  props: {
    freshdesk: {
      type: "app",
      app: "freshdesk",
    },
    days_limit: {
      type: "integer",
      label: "Fresh Desk History Limit (in days)",
      default: 120
    }
  },
  async run({steps, $}) {
    const fetchTicketsPage = async (tickets, page, date_limit, $) => {
      const results = await axios($, {
        url: `https://${this.freshdesk.$auth.domain}.freshdesk.com/api/v2/tickets?order_type=desc&per_page=100&page=${page}`,
        auth: {
          username: `${this.freshdesk.$auth.api_key}:X`,
          password: ``,
        },
      })

      //if no results then default to date_limit
      const last_ticket_date = results.length > 0 ? new Date(results[results.length - 1].created_at) : date_limit

      tickets.report = tickets.report.concat(results)
      
      //recursive into next page of results until we hit date limit
      if (last_ticket_date > date_limit) {
        await new Promise(resolve => setTimeout(resolve, 500)); //sleep 500 millis to avoid FD error 429
        tickets = await fetchTicketsPage(tickets, ++page, date_limit, $)
      }

      return tickets
    }

    const date_limit = new Date()
    date_limit.setDate(date_limit.getDate() - this.days_limit)

    const tickets = await fetchTicketsPage({ report: [] }, 1, date_limit, $)

    $.export("reports", {
      from_date: date_limit,
      report: {
        count: tickets.report.length,
        oldest_date: tickets.report.length > 0 ? tickets.report[tickets.report.length - 1].created_at : "N/A",
      },
    })
  }
})

Hi @davidt

First off, welcome to the Pipedream community. Happy to have you!

Thanks for sharing! That is super cool! Impressive use of props too :wink:

I think this is a great use of Node.js code steps with recusion, and to limit your invocations too.

The only issue I might see is if your ticket volume grows or FreshDesk starts to rate limit your queries, then you might hit the timeout limit on your workflow, but you can extend it in your Execution Controls section in your workflow’s settings:

It’s a bit more advanced, but you can create a custom Source component that triggers a workflow on unique Freshdesk tickets, and still pass your days_limit prop to limit the initial historical tracking.

Then your source will automatically retry to pull tickets on a polling schedule, and only trigger workflows when new unique tickets are detected.

Our Source Quickstart shows you the ropes on how to build one from scratch.

1 Like

Hi @pierce and thanks for your reply!

Props - yup, learned it from the master - love your videos and watched them all!

Thanks for the tip about sources. I didn’t yet create a custom source, though I understand the concept. It won’t work for this specific use case, but it’s definitely another tool I know exists for when the right use case arrives.

Cheers and looking forward to your next vids on Components!
D

1 Like

@davidt thank you! :smiley: So glad you’re enjoying them.

Yes new components videos are coming soon, and they’ll show you how to build reusable components to build workflows even faster.

Right, you start to realize what is a source vs an action naturally. Sometimes you don’t need the deduping or polling that a source provides, but other times it’s very handy.

Please let us know if we can answer any of your questions! Or if you need other concepts explained we’re happy to make additional videos/docs.

1 Like