Why Aren't My Google Cloud Pipelines Running Scheduled Big Query Jobs Properly?

Help
1

This topic was automatically generated from Slack. You can find the original thread here.

Hi all! When I set up pipelines that use Google Cloud as the trigger to run a Big Query query I seem to have issues with them running on their schedule. So if I set it up to run every 15 minutes it doesn’t seem to work. Even when I press “Try now” it doesn’t pull in the new events. Has anyone else experienced this? Am I doing something wrong?

2

Hi , could you share the specific configuration of your source?

Are you using BigQuery - New Row or BigQuery - Query Results?

I would like to see the source log as well, could you access Pipedream source page > Select your BigQuery source > click LOGS tab?

3

sorry for the delay. Working on too many things concurrently. I really appreciate you diving in.

4

I’ve tried both new row and query results with an event size of 1, which should achieve similar results.

5

I’m currently trying with query results. This is my config

image.png
image.png675×960 62.4 KB

6

Just checked the logs and I see “Out of Memory: you’ve exceeded the memory for this source”

7

Worth noting that I’ve got round this problem historically by using the Custom Interval trigger and then just doing the query in a step.

8

Got it , we’ll check and get back to you soon

9

Hey , I’ve created an issue here for our component dev to check: [BUG] BigQuery - New Query Result throws out of memory error · Issue #18352 · PipedreamHQ/pipedream · GitHub

I’ve added to Pipedream backlog as well

10

Thanks , I see this is already triaged. Thanks for jumping in and helping.