Are Pipedream Webhooks and Email Triggers Not Working?

This topic was automatically generated from Slack. You can find the original thread here.

Are Pipedream webhooks down?

I cannot receive any emails using the Email trigger, and I have also noticed I am getting unsuccessful requests when generally using webhooks

Example attached

image.png

image.png

Email-wise, sent multiple requests to emt9plfyqse4gqf@upload.pipedream.net

Yet nothing was received

Will change the webhook after this ticket - but here it is for the sake of debugging

Retried many times, nothing coming through

Seems to have stopped 3 hours ago based on my logs

There was a very brief outage, but it should be back up — are you still seeing an issue now?

Yes it’s still out for me

Email or HTTP webhook? Or both?

Both

Just received the email webook

HTTP webhook still seems to be out, I’ll test again

I just tested both on my end, both are working

By the way, the incident last about 10 minutes, so if you were seeing an issue for 3 hours, that sounds unrelated

We were affected by this as well.

Seemed like it lasted much longer than ten minutes as we missed multiple callbacks over a longer period of time.

But at least it’s working now. :slightly_smiling_face: :raised_hands:

What was the rough time span you saw, Marco? We traced the deploy that caused the outage and had rolled back within 10 minutes, but let me know if you’re seeing something else.

This affected multiple workflows waiting for a callback, each one with different timeout values… so tricky to calculate.

And it looks like we also deployed something just a bit earlier, and that is the cause of some missing callbacks as well (we actually received the callback, but the workflow failed to resume due to the deployment while it was suspended).

So it looks like the window is roughly 15h30 to 15h50 (PDT).

That’s when we have definitely missing callbacks.