This topic was automatically generated from Slack. You can find the original thread here.
Vish K : Is there a way to see further back in sources logs? I have a source with a 5 minute timer running and can only see the past 10 executions (~ 1 hour) on the logs page. Is this something the paid tier provides?
Dylan Sather (Pipedream) : not currently in the Pipedream UI, but we’re tracking that here if you’d like to follow.
It is possible to listen for new logs via the Pipedream API via a subscription, which allows you to trigger a workflow on every new log entry, writing that somewhere like Amazon S3 or a logging service. When creating a subscription for a source, you can specify an event_name of $logs, which tells the subscription to specifically route the logs of that source to your intended listener_id (the workflow, webhook, or other resource receiving the events).
Would that work in your case?
Vish K : Thanks very much - I can definitely work with that…but will keep an eye on this FR, would be great to keep it all within pipedream
So I’ve come back to this issue a little later and am now diving into to building observer workflows that subscribe to other workflows’ $logs emitter, as you described.
What I’m noticing, out of the gate, is that $logs isn’t populated by each workflow’s invocation – the actual execution of the workflow as it appears in Pipedream’s inspector left-hand panel. I’m assuming this means it is only populated by the workflow’s actions that console.log anything along the way (but please correct me if not!)
What I would truly like is to subscribe to the workflow’s invocation, (perhaps an invocation channel?) thereby capturing metadata related to that particular execution (as displayed in Pipedream’s inspector panel), so:
Before I start building this all out by hand, is there any way to leverage the API above to subscribe to/listen for the invocations themselves?
…and is my understanding of $logs correct? That it is only populated by data sent to console.log in each wf?