This topic was automatically generated from Slack. You can find the original thread here.
Hi ! I have a workflow called many times at the same time, but I need to make sure the full workflow has been executed before processing the next one (So I need sequential workflows execution instead of parallel workflow execution). Is it possible ?
Based on the information provided in the Pipedream documentation on concurrency and throttling, you can control the rate at which your workflow executes to ensure that it runs sequentially instead of in parallel. Here’s what you can do: 1. Visit the Pipedream documentation on concurrency and throttling at Concurrency and Throttling. 2. Implement execution controls to limit the maximum number of times your workflow may be invoked over a specific period of time. For example, you can set it to allow up to 1 event every second, which would ensure that the full workflow has been executed before processing the next one. This approach will help you manage the execution rate of your workflows and avoid parallel execution.
Hi ! I need sequential workflows execution instead of parallel workflow execution but I am with a Basic Plan, so I can’t use the concurrency and throttling feature of Pipedream. Any suggestion using queues ?
Based on your requirements, it seems you’re looking to process events sequentially in your workflow. While the concurrency and throttling features are indeed a way to manage sequential execution, they are not available on the Basic Plan as you’ve mentioned. However, you can still implement a form of sequential processing using a queue mechanism within your workflow. Here’s a general approach you can take: 1. Use a data store to act as a queue. You can push events to this data store and process them one by one. 2. Create a workflow where the trigger adds the event to the data store. 3. Have another step in the workflow that checks if there’s an event being processed. If not, it takes an event from the data store and marks it as being processed. 4. Process the event. 5. Once processing is complete, remove the event from the data store and mark that there’s no event being processed. 6. Trigger the workflow again to check for the next event in the queue. This is a manual way to create a queue system within your workflow. It’s important to handle race conditions and ensure that only one event is processed at a time. For more advanced queueing mechanisms or to increase the queue size for your workflows, you might need to consider upgrading your plan. If you need further assistance or have more specific requirements, please let me know, and I can provide more detailed guidance or code examples.