How to Resolve the "Event Queue Full" Error in a Consistently Run Workflow?

Help
1

This topic was automatically generated from Slack. You can find the original thread here.

I have a particular workflow that is has been giving me the “Event Queue Full” error once a day for the past few days. No new events are showing up in the list of live events, even when I trigger the workflow manually to try to troubleshoot. This workflow has been running reliably for a long time before this and the current deployment has been running since February, so I don’t think the issue would be connected to anything I would have changed on my end. What can I do to get this workflow back up and running?

2

Increase the size of the queue!

3

Pipedream doesn’t handle small queue sizes very well.

4

And events can remain in the queue for much longer than their execution.

5

We had a use case where we wanted to use a queue size of 1 single event, with one execution every few hours.

But we were running into this “Event Queue Full” error all the time.

6

So basically, Pipedream just doesn’t support small queue sizes right now. :man-shrugging:

7

The queue size is the default of 100. How large of a queue do you recommend?

8

I changed the queue size to 1000 and tested the workflow (I have a scheduled polling source that searches for records to sync). I can see in the source log that 5 records were found, but none of them are making it through into the workflow. Is there a way to reset the event queue or anything like that? Maybe I just have to duplicate this workflow and start fresh . . . . Sure seems like a weird glitch on the backend.

9

what’s the workflow URL? Can you share it with us so we can take a look?

10

If you’re blocked and need it to work ASAP, duplicating it will definitely help

11

Yes, I made a copy this morning as a workaround.

Here’s the url for the original: Sign Up - Pipedream

12

Thanks, I’ll forward this to the team

13

I looked into your workflow copy and some events are getting rate limited by Airtable (thus get paused), but are set to retry automatically. You probably are aware of this but just FYI

14

Yes, the automatic retry is an intentional workaround to avoid having to manually re-run workflows when there’s an Airtable error (rate limit or expired auth token). Are you saying there’s a better way to handle that? Or that the issue with the event queue might be related in some way?

15

Got it, yeah the Airtable OAuth expiring especially is frustrating. Regarding rate limiting, I think 1 execution per second is pretty “limited”, so I wonder what is the cause for these errors. But I looked again now and they seemed to have executed successfully after trying again

16

If I recall, the Airtable rate limit is 5 per second. I think if a workflow has enough Airtable steps, it’s possible that they will happen quickly enough to trigger the rate limit. That or two different workflows just happen to go off at the same time and together they hit it. I have a lot of workflows that touch Airtable and get a handful of rate limit errors per day. Fortunately I’ve been able to migrate most of them to v2 to get the auto-retry in place :slight_smile:

17

Got it, makes sense if there are many Airtable steps and/or workflows! According to this doc the rate limit is 5 requests per second and 50 requests per second for a PAT