Server-Sent Events (SSE), clear the stream content

Dev
1

Hello to everyone.
In one of mine workflow I’m using the Server-Sent Events (SSE) feature succesfully.

There is a way to clear the whole stream with some command or procedure?
I have a lot of test messages that i would like to remove.

2

Hi @cotestatnt , thanks for reaching out. Just to confirm, are you using the $send.sse() function in a workflow, or the SSE interface exposed by an event source?

Currently, the events emitted by $send.sse() cannot be cleared. Are you able to filter out these events in some manner in the client you’re using to connect to the stream?

The SSE interface exposed by event sources implements some improvements to the workflow interface, if it helps:

  • When a client connects to the stream, you’ll only receive new messages from the point of connection moving forward
  • We protect the stream with auth (your Pipedream API key), which can be useful if you’re connecting to the stream from a server.

Let me know if that helps or if you have any other questions!

3

Thanks @dylburger .
I’m able to filter events from client side but of course at the very beginning, client have to read the whole stream. For now it’s not a big problem, but what happen when the stream will increase during time?

Furthermore, very old events are not relevant for the client so this is just a waste of time and resources.
For this reason i would like to clear stream (from pipedream side or only if client is authorized of course).

4

@cotestatnt in the $send.sse() model, we truncate older events from the stream. e.g. if I emit 1,000 events via $send.sse(), I’ll only see the last ~200 events when I load the stream.

If you use the SSE interface for event sources, that will also give you a clear stream every time. We’re definitely open to implementing this behavior for the $send.sse() model, as well.

5

This is good to know and it’s enougth to dissipate my doubts.

Thank you for the information and your support!

1 Like
6

No problem, let us know if you have any other questions or feedback.

7

@cotestatnt For the workflow SSE stream you will notice that there is an ID assigned to ever event. You can use that Id to specify where to start the stream from. Example:

# curl http://sdk.m.pipedream.net/pipelines/p_95CKLyK/sse
event: yo
id: 1620316713777-0
data: {"hi":true}

:keepalive

event: yo
id: 1620316784450-0
data: {"hi":true}

event: yo
id: 1620316784626-0
data: {"hi":true}

event: yo
id: 1620316785572-0
data: {"hi":true}

event: yo
id: 1620316785629-0
data: {"hi":true}

:keepalive

^C
# curl -H 'last-event-id: 1620316785572-0'  http://sdk.m.pipedream.net/pipelines/p_95CKLyK/sse                   
event: yo
id: 1620316785629-0
data: {"hi":true}

The first curl just follows the stream, and the in the second curl I resume the stream with the second to last event id, so it returns the last event, and then it will stream from there. (Pro-tip, you can also send in a epoch millisecond timestamp to resume from a specific time)

1 Like
8

When a client connects to the stream, you’ll only receive new messages from the point of connection moving forward

just to be clear–every time a client connects to the EventSource, they’ll get the entire cache of all events ever sent (that’s what I’m seeing now) and from then on only new events, OR when a client connects should they only get new events from that point on (the behavior I expect, but which I’m not seeing)

Or do we need to clear the event cache in our code? (BTW this is in node on pipedream)

thanks! (new to pipedream, amazing stuff!)