This topic was automatically generated from Slack. You can find the original thread here.
I have a workflow that pulls a series of records from a database and then processes each of them separately. The number can vary from a few to a few hundred or even thousand. Once all records have processed, I want to trigger another bulk process. What’s the recommended way to know that all records from the first batch have run?
One option I thought of is to iterate through the list within the workflow but I’m afraid I will hit a timeout if volume gets higher. The other option is to build the workflow to process the records separately which would be more scalable but I would lose the context of when the total volume has run.
I’m here to help! However, it seems like your question might be incomplete or missing some context. Could you please provide more details or clarify what specific information or assistance you need? This will help me give you a more accurate and helpful response.
I have a workflow that pulls a series of records from a database and then processes each of them separately. The number can vary from a few to a few hundred or even thousand. Once all records have processed, I want to trigger another bulk process. What’s the recommended way to know that all records from the first batch have run?
One option I thought of is to iterate through the list within the workflow but I’m afraid I will hit a timeout if volume gets higher. The other option is to build the workflow to process the records separately which would be more scalable but I would lose the context of when the total volume has run.
To handle the processing of a large number of records and ensure that a subsequent bulk process is triggered only after all records have been processed, you can consider the following approach: ### Recommended Approach 1. Separate Workflows for Processing and Monitoring: - Processing Workflow: Create a workflow that processes each record individually. This workflow will be triggered for each record, allowing it to scale with the number of records without hitting a timeout. - Monitoring Workflow: Use a separate workflow to monitor the completion of all processing tasks. This workflow can be triggered once all records have been processed. 2. Using a Data Store to Track Progress: - Use Pipedream’s Data Store to keep track of the number of records processed. Each time a record is processed, update the Data Store. - Store the total number of records to be processed in the Data Store at the start. 3. Trigger the Bulk Process: - Once the number of processed records matches the total number of records, trigger the bulk process. ### Example Implementation 1. Processing Workflow: - This workflow processes individual records and updates the Data Store. export default defineComponent({ props: { data: { type: "data_store" }, }, async run({ steps, $ }) { // Process the record here // Update the processed count const processedCount = await this.data.get('processedCount') ?? 0; await this.data.set('processedCount', processedCount + 1); }, }); 2. Monitoring Workflow: - This workflow checks if all records are processed and triggers the bulk process. ```
export default defineComponent({ props: { data: { type: “data_store” }, }, async run({ steps, $ }) { const totalRecords = await this.data.get(‘totalRecords’); const processedCount = await this.data.get(‘processedCount’) ?? 0; if (processedCount === totalRecords) { // Trigger the bulk process console.log(“All records processed. Triggering bulk process.”); // Add code to trigger the bulk process here } }, });