This topic was automatically generated from Slack. You can find the original thread here.
Hi all! When I set up pipelines that use Google Cloud as the trigger to run a Big Query query I seem to have issues with them running on their schedule. So if I set it up to run every 15 minutes it doesn’t seem to work. Even when I press “Try now” it doesn’t pull in the new events. Has anyone else experienced this? Am I doing something wrong?
Hi , could you share the specific configuration of your source?
Are you using BigQuery - New Row or BigQuery - Query Results?
I would like to see the source log as well, could you access Pipedream source page > Select your BigQuery source > click LOGS tab?
sorry for the delay. Working on too many things concurrently. I really appreciate you diving in.
I’ve tried both new row and query results with an event size of 1, which should achieve similar results.
Just checked the logs and I see “Out of Memory: you’ve exceeded the memory for this source”
Worth noting that I’ve got round this problem historically by using the Custom Interval trigger and then just doing the query in a step.
Got it , we’ll check and get back to you soon
Hey , I’ve created an issue here for our component dev to check: [BUG] BigQuery - New Query Result throws out of memory error · Issue #18352 · PipedreamHQ/pipedream · GitHub
I’ve added to Pipedream backlog as well
Thanks , I see this is already triaged. Thanks for jumping in and helping.
Posted thread to Discourse: Why Aren't My Google Cloud Pipelines Running Scheduled Big Query Jobs Properly?
Hello , we have just released a new version of this source. Would you mind checking again?
Sure, I will test. Thanks for this!
I deployed a new trigger to poll GCP every 15 minutes. It ran the first time and generated rows. I’ll keep an eye on it when I see new events should be adding rows to the table.
Thanks for your information , I’ve reopened the issue
