Source: Reset emitted events or manually trigger a workflow that is triggered by a source

lharney · February 22, 2022, 12:04pm

Hey,

I’ve created a source that triggers a workflow and it works well.

Is it possible to reset the emitted events from the source or to manually trigger the workflows?
The reason I ask is that when I upload the source, the source makes a request to the external API before the workflow is created. This means that any events emitted by the source, before the workflow is created, does not trigger the workflow.

Thanks,
Lar

pierce · February 22, 2022, 3:06pm

Hi @lharney ,

First off, welcome to the Pipedream community. Happy to have you!

I was having this same issue myself with a custom component, I ended up creating a prop that would allow the source to paginate backwards into the history up to a certain time so I could cover missed events.

I made an issue on our public tracker, does this describe the issue well enough or would you like to add more?

github.com/PipedreamHQ/pipedream

[FEATURE] Allow re-emitting of source events

opened 03:06PM - 22 Feb 22 UTC

ctrlaltdylan

enhancement

**Is your feature request related to a problem? Please describe.** As a compo…nent developer, specifically for polling APIs, it's not possible to deploy a custom source component to a specific workflow. https://pipedream.com/community/t/source-reset-emitted-events-or-manually-trigger-a-workflow-that-is-triggered-by-a-source/1964 If your source is polling an API, it will being to emit events before you can create a workflow. This means that your workflow will miss events. **Describe the solution you'd like** We'd like a workaround to either re-send emitted events in the Source dashboard, or some mechanism to allow a workflow to manually run missed events that occurred before the workflow was added to the source. **Do you have a workaround?** At this time you have to develop a polling component to programmatically go backwards in history so you can have a workflow for past events, and another workflow for future events.

lharney · February 23, 2022, 3:17pm

Hi @pierce,

Thanks for the response.

I understand creating a prop in the source with a record type id and then paginate backwards.

Would I not be left with the same issue that the emitted records still have the same ID. Hence, they will not trigger a workflow because the events with that id have already been emitted?

Thanks,
Lar

pierce · February 23, 2022, 4:31pm

Hi @lharney ,

I’m sorry I’m not quite following that last sentence, could you perhaps share a step by step account of the problem?

The other solution I can think of is to add an additional trigger to the workflow to accept an HTTP webhook, and still dedupe events from within the $.service.db at the workflow level.

The workflow would receive events from both the forward and backward facing sources, but would dedupe based on the event ID.