user-1
(User 1)
October 11, 2024, 12:11am
1
This topic was automatically generated from Slack. You can find the original thread here .
Are Pipedream webhooks down?
I cannot receive any emails using the Email trigger, and I have also noticed I am getting unsuccessful requests when generally using webhooks
Example attached
user-1
(User 1)
October 11, 2024, 12:11am
2
Email-wise, sent multiple requests to emt9plfyqse4gqf@upload.pipedream.net
Yet nothing was received
Will change the webhook after this ticket - but here it is for the sake of debugging
user-1
(User 1)
October 11, 2024, 12:11am
3
Retried many times, nothing coming through
user-1
(User 1)
October 11, 2024, 12:11am
4
Seems to have stopped 3 hours ago based on my logs
user-1
(User 1)
October 11, 2024, 12:11am
5
There was a very brief outage, but it should be back up — are you still seeing an issue now?
user-1
(User 1)
October 11, 2024, 12:11am
6
Yes it’s still out for me
user-1
(User 1)
October 11, 2024, 12:11am
7
Email or HTTP webhook? Or both?
user-1
(User 1)
October 11, 2024, 12:11am
9
Just received the email webook
user-1
(User 1)
October 11, 2024, 12:11am
10
HTTP webhook still seems to be out, I’ll test again
user-1
(User 1)
October 11, 2024, 12:11am
11
I just tested both on my end, both are working
user-1
(User 1)
October 11, 2024, 12:11am
12
By the way, the incident last about 10 minutes, so if you were seeing an issue for 3 hours, that sounds unrelated
user-1
(User 1)
October 11, 2024, 12:11am
13
We were affected by this as well.
user-1
(User 1)
October 11, 2024, 12:11am
14
Seemed like it lasted much longer than ten minutes as we missed multiple callbacks over a longer period of time.
user-1
(User 1)
October 11, 2024, 12:11am
15
But at least it’s working now.
user-1
(User 1)
October 11, 2024, 12:11am
16
What was the rough time span you saw, Marco? We traced the deploy that caused the outage and had rolled back within 10 minutes, but let me know if you’re seeing something else.
user-1
(User 1)
October 11, 2024, 12:11am
17
This affected multiple workflows waiting for a callback, each one with different timeout values… so tricky to calculate.
user-1
(User 1)
October 11, 2024, 12:11am
18
And it looks like we also deployed something just a bit earlier, and that is the cause of some missing callbacks as well (we actually received the callback, but the workflow failed to resume due to the deployment while it was suspended).
user-1
(User 1)
October 11, 2024, 12:11am
19
So it looks like the window is roughly 15h30 to 15h50 (PDT).
user-1
(User 1)
October 11, 2024, 12:11am
20
That’s when we have definitely missing callbacks.