This topic was automatically generated from Slack. You can find the original thread here.
Hello! I need advice on strategy to process heavy workflow. For context, I’m trying to build a workflow to get a summary data based on a set of item id’s passed into the workflow. The workflow then proceed to get the data from a source (ClickUp) and process it before returning the output.
Few challenges:
• The item id passed as input need to fetch additional info from the sources (eg. linked by Property ID)
• There are 2 sources I need to fetch eg. Item & Property
• The sources API don’t have the ability to filter the ID, hence I need to fetch the entire source database which can be huge (>10,000 records)
• From there the workflow will find and fill in the data from sources to the items.
• Return the items as output take quite long for first run (77s).
Some workaround I tried and using now:
• Use Data Store to cache the output if there are recent request in short time
Workflow:
• Trigger (HTTP) - to get item id input
• Data Store - check if there is valid cache and return and end the workflow
• Get ClickUp List Data (1) - From given item id input
• Get ClickUp List Data (2) - All of Property data source
• Combine & Respond - Fill (1) with additional info from (2), then respond back (at the same time caching it to data store also)
I’m a bit worry on how this can scale well, the source records are going to scale from here onwards probably to >20,000 records soon. As it is even the workflow editor when debugging slows to a crawl and crash the browser tab (probably run out of memory).
I appreciate any advice on how to approach this 
