This topic was automatically generated from Slack. You can find the original thread here.
Nicholas Reilingh : Is there a cap on the number of historical executions that can be viewed in the workflow editor? I was hoping to inspect one that happened earlier yesterday, but I’m not seeing it anymore.
Dylan Sather (Pipedream) : There is. Currently up to the last 100 events are stored (the cap is implemented a little differently, so you may see fewer). We’re tracking a feature to increase this here.
Dylan Sather (Pipedream) : Do you need to see the full Pipedream workflow observability in this case, or are you looking to inspect a particular value? Sometimes in the latter case I’ll often store specific data to a table in the the SQL Service or another external log.
Nicholas Reilingh : I was hoping to see the full workflow output at each step to try to understand what may have been a race condition with calls to Airtable that resulted in an extra duplicate reference value being created. I take it that Pipedream processes incoming events in a “multithreaded” async style instead of serially?
Dylan Sather (Pipedream) : By default, yes, but if you think there could be a race, you can serialize execution (among other controls): Managing Concurrency and Throttling
Dylan Sather (Pipedream) : Definitely could be an issue - very common with Airtable / Google Sheets
Nicholas Reilingh : Oh! Wow that’s fantastic – really really great how flexible this is
Nicholas Reilingh : I now have Pipedream receiving events created from a SQL Server stored procedure behind a firewall – the procedure kicks off a PowerShell script on the server that queries for the new event data and then sends it up to Pipedream via webhook. Totally blows the pants off of my former limitations with this environment.
Nicholas Reilingh : One feature request, perhaps, would be for the webhook source to be configurable with a bearer token – I’m currently checking for this in a step immediately after the event step for safety.
Dylan Sather (Pipedream) : That’s a cool workflow! Sounds like a good blog post
Dylan Sather (Pipedream) : Does this address that use case? [FEATURE] I'd like a simple way to enable Basic Auth validation on an endpoint · Issue #905 · PipedreamHQ/pipedream · GitHub
Nicholas Reilingh : Perhaps–actually the “New Scheduled Tasks (Alpha)” action’s optional “secret” configuration is exactly what I’d be hoping to see added to the generic webhook. Looks like it uses an X-PD-Secret header instead of Authorization.
Nicholas Reilingh : I think I would shy away from basic auth since username/password is overly complex for the use case compared to a single token value (either Bearer or X-PD-Secret)
Dylan Sather (Pipedream) : I hear you, I think we would make the auth configuration generic to cover these use cases (the ticket was raised by someone who needed Basic auth, since that’s how their particular service communicated credentials)
Dylan Sather (Pipedream) : Sharing all this feedback with the team so we can think about this for future iterations of the HTTP trigger
Nicholas Reilingh : sure, makes sense. I think the main thing is just a way to have the absence of a specific valid header cause the workflow not to be counted as an execution
Dylan Sather (Pipedream) : Yes I agree, not charging for invocations in that case is key. Ideally you’d be able to run other basic checks on incoming data to apply a generic filter: [FEATURE] Filter event data prior to triggering a source / workflow invocation · Issue #556 · PipedreamHQ/pipedream · GitHub
Dylan Sather (Pipedream) : would that be useful in other workflows?
Nicholas Reilingh : Hm, interesting… I feel like recently I haven’t had to deal with webhooks that just give you a firehose of data that you have to filter. It seems like as the webhook “ecosystem” has matured, it’s more common now to be able to subscribe to a webhook for a single stream of relevant data. So I don’t think it would affect me very much, but I can see how it could be useful. Of course it also does solve for the security problem in an extremely generic way so I could see that as being your direction if you’re trying to significantly increase the robustness of the webhook source.