This topic was automatically generated from Slack. You can find the original thread here.
My workflow generates 1 event per day at 2am. I would like to consume this event in third-party code running on cron at 4am while avoiding any state or receiving incoming requests on the cron side. What are my options?
My workflow generates 1 event per day at 2am. I would like to consume this event in third-party code running on cron at 4am while avoiding any state or receiving incoming requests on the cron side. What are my options?
To achieve your goal of consuming a Pipedream event in third-party code running on a cron schedule without maintaining state or receiving incoming requests, you can use Pipedream’s event sources and REST API. Here’s a step-by-step approach: 1. Emit Events from Pipedream Workflow: Ensure your Pipedream workflow emits events at 2am as you described. 2. Use Pipedream REST API: You can retrieve the emitted events using Pipedream’s REST API. This allows your third-party code to pull the events when it runs at 4am. 3. Setup Your Cron Job: In your third-party environment, set up a cron job to run at 4am. This job will make an HTTP request to Pipedream’s REST API to fetch the events. 4. Process the Events: Once your cron job retrieves the events, you can process them as needed in your third-party code. Here’s a simple example of how you might structure the HTTP request in your cron job using a tool like curl or a programming language of your choice: bash curl -X GET "https://api.pipedream.com/v1/events" \ -H "Authorization: Bearer YOUR_API_KEY" Replace YOUR_API_KEY with your actual Pipedream API key. This request will fetch the events emitted by your workflow. For more details on using the Pipedream REST API, you can refer to the Pipedream API documentation. If you have any further questions or need clarification, feel free to ask!
Hey ! Honestly I’m trying to understand why you want to make it stateless when one event triggers at 2AM and the workflow consumes it at 4AM
How would you do that outside of Pipedream?
The cron code is AWS Lambda so the less state the better
The /events endpoint approach would work. Just “collect the last event” while the state is kept on the Pipedream side.
But the endpoint proposed by Pi doesn’t exist.
Does this work?
Oh wait got it you want to consume the emitted event
What’s the issue with using state?
This can be done easily using a Data Store, for example
I can use state on the pipedream side, just not on the Lambda side. So you suggest putting events to a Data Store and then adding an HTTP-triggered workflow? How can I return data over HTTP from the workflow?
I’m thinking of adding the events in a Data Store — you could use it as queue
And then your second workflow that triggers at 4AM would consult the data in the data store
And operate on the data and clear it after the operation
You wouldn’t need an HTTP trigger, just the CRON trigger and use Data Stores actions to fetch the data and delete it later
There is no second workflow. The consumer is another system unrelated to Pipedream
Ah gotcha
Yeah than using the HTTP trigger makes sense
The external system calls the HTTP trigger, where the workflow fetches the data from the data store and returns it in the HTTP response
You can configure the trigger to return a custom response from the workflow. And then at some point in the workflow, you need to add a HTTP response action