Create Scheduled Query with Google Cloud API

Pipedream makes it easy to connect APIs for Google Cloud and 2,400+ other apps remarkably fast.

Trigger workflow on

HTTP requests, schedules and app events

Next, do this

Create Scheduled Query with the Google Cloud API

No credit card required

▶

Watch us build a workflow

8 min

Watch now ➜

Trusted by 1,000,000+ developers from startups to Fortune 500 companies

Developers ♥ Pipedream

Getting Started#

Create a workflow to Create Scheduled Query with the Google Cloud API. When you configure and deploy the workflow, it will run on Pipedream's servers 24x7 for free.

Configure the Create Scheduled Query action
1. Connect your Google Cloud account
2. Select a Destination Dataset
3. Optional- Select a Dataset Region
4. Configure Display Name
5. Configure Query
6. Optional- Configure Schedule
7. Optional- Select a Write Disposition
8. Optional- Configure Destination Table Name Template
Select a trigger to run your workflow on HTTP requests, schedules or app events
Deploy the workflow
Send a test event to validate your setup
Turn on the trigger

Integrations#

Create Scheduled Query with Google Cloud API on New Requests (Payload Only) from HTTP / Webhook API

HTTP / Webhook + Google Cloud

Try it

Create Scheduled Query with Google Cloud API on New Submission from Typeform API

Typeform + Google Cloud

Try it

Create Scheduled Query with Google Cloud API on Custom Events from Zoom API

Zoom + Google Cloud

Try it

Create Scheduled Query with Google Cloud API on New Submission (Instant) from Jotform API

Jotform + Google Cloud

Try it

Create Scheduled Query with Google Cloud API on New Scheduled Tasks from Pipedream API

Pipedream + Google Cloud

Try it

Details#

This is a pre-built, source-available component from Pipedream's GitHub repo. The component is developed by Pipedream and the community, and verified and maintained by Pipedream.

To contribute an update to an existing component or create a new component, create a PR on GitHub. If you're new to Pipedream component development, you can start with quickstarts for trigger span and action development, and then review the component API reference.

Create Scheduled Query on Google Cloud

Description:Creates a scheduled query in Google Cloud. [See the documentation](https://cloud.google.com/bigquery/docs/scheduling-queries)

Version:0.0.1

Key:google_cloud-create-scheduled-query

View on GitHub

Code#

import { protos } from "@google-cloud/bigquery-data-transfer";
import googleCloud from "../../google_cloud.app.mjs";
import constants from "../../common/constants.mjs";
import regions from "../../common/regions.mjs";

const {
  CreateTransferConfigRequest,
  TransferConfig,
} = protos.google.cloud.bigquery.datatransfer.v1;

export default {
  key: "google_cloud-create-scheduled-query",
  name: "Create Scheduled Query",
  description: "Creates a scheduled query in Google Cloud. [See the documentation](https://cloud.google.com/bigquery/docs/scheduling-queries)",
  version: "0.0.1",
  type: "action",
  props: {
    googleCloud,
    destinationDatasetId: {
      label: "Destination Dataset",
      description: "The name of the dataset to create the table in. If the dataset does not exist, it will be created.",
      propDefinition: [
        googleCloud,
        "datasetId",
      ],
    },
    datasetRegion: {
      type: "string",
      label: "Dataset Region",
      description: "The geographic location where the dataset should reside. [See the documentation here](https://cloud.google.com/bigquery/docs/locations#specifying_your_location)",
      default: "us",
      options: regions,
      optional: true,
    },
    displayName: {
      type: "string",
      label: "Display Name",
      description: "The user-friendly display name for the transfer config.",
    },
    query: {
      type: "string",
      label: "Query",
      description: "The GoogleSQL query to execute. Eg. ``SELECT @run_time AS time, * FROM `my_dataset.my_table` LIMIT 1000``. [See the documentation here](https://cloud.google.com/bigquery/docs/scheduling-queries#query_string).",
    },
    schedule: {
      type: "string",
      label: "Schedule",
      description: "Data transfer schedule. If the data source does not support a custom schedule, this should be empty. If it is empty, the default value for the data source will be used. The specified times are in UTC. Examples of valid format: `every 24 hours`, `1st,3rd monday of month 15:30`, `every wed,fri of jan,jun 13:15`, and `first sunday of quarter 00:00`. [See more explanation about the format here](https://cloud.google.com/appengine/docs/flexible/python/scheduling-jobs-with-cron-yaml#the_schedule_format).",
      optional: true,
    },
    writeDisposition: {
      type: "string",
      label: "Write Disposition",
      description: "The write preference you select determines how your query results are written to an existing destination table. [See the documentation here](https://cloud.google.com/bigquery/docs/scheduling-queries#write_preference).",
      default: constants.WRITE_DISPOSITION.WRITE_TRUNCATE,
      options: Object.values(constants.WRITE_DISPOSITION),
      optional: true,
    },
    destinationTableNameTemplate: {
      type: "string",
      label: "Destination Table Name Template",
      description: "The destination table name template can contain template variables such as ``{run_date}`` or ``{run_time}``. [See the documentation here](https://cloud.google.com/bigquery/docs/scheduling-queries#templating-examples).",
      optional: true,
      default: "logs",
    },
  },
  methods: {
    getTransferConfig({
      query, writeDisposition, destinationTableNameTemplate, ...args
    } = {}) {
      return new TransferConfig({
        dataSourceId: constants.DATA_SOURCE_ID.SCHEDULED_QUERY,
        params: {
          fields: {
            query: {
              stringValue: query,
            },
            destination_table_name_template: {
              stringValue: destinationTableNameTemplate,
            },
            write_disposition: {
              stringValue: writeDisposition,
            },
          },
        },
        ...args,
      });
    },
    createTransferConfig(args = {}) {
      const {
        googleCloud,
        getTransferConfig,
      } = this;

      const {
        project_id: projectId,
        client_email: serviceAccountName,
      } = googleCloud.authKeyJson();

      const client = googleCloud.bigQueryDataTransferClient();
      const parent = client.projectPath(projectId);

      const request = new CreateTransferConfigRequest({
        parent,
        serviceAccountName,
        transferConfig: getTransferConfig(args),
      });

      return client.createTransferConfig(request);
    },
  },
  async run({ $ }) {
    const {
      createTransferConfig,
      ...props
    } = this;

    const [
      response,
    ] = await createTransferConfig(props);

    $.export("$summary", `Scheduled query created with name: \`${response.name}\``);

    return response;
  },
};

Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI and CLI.

Label	Prop	Type	Description
Google Cloud	`googleCloud`	`app`	This component uses the Google Cloud app.
Destination Dataset	`destinationDatasetId`	`string`	Select a value from the drop down menu.
Dataset Region	`datasetRegion`	`string`	Select a value from the drop down menu:`useuropeus-east1us-east4us-east5us-east7us-central1us-central2us-west1us-west2us-west3us-west4us-south1asia-east1asia-northeast1asia-southeast1asia-south1asia-east2asia-northeast2asia-northeast3asia-southeast2asia-south2europe-north1europe-west1europe-west2europe-west3europe-west4europe-west6europe-central2europe-west9europe-west8europe-southwest1europe-west10europe-west12northamerica-northeast1northamerica-northeast2southamerica-east1southamerica-west1australia-southeast1australia-southeast2me-central2me-west1me-central1africa-south1aws-us-east-1aws-us-west-2aws-ap-northeast-2aws-eu-west-1azure-eastus2`
Display Name	`displayName`	`string`	The user-friendly display name for the transfer config.
Query	`query`	`string`	The GoogleSQL query to execute. Eg. SELECT @run_time AS time, * FROM `my_dataset.my_table` LIMIT 1000. See the documentation here.
Schedule	`schedule`	`string`	Data transfer schedule. If the data source does not support a custom schedule, this should be empty. If it is empty, the default value for the data source will be used. The specified times are in UTC. Examples of valid format: `every 24 hours`, `1st,3rd monday of month 15:30`, `every wed,fri of jan,jun 13:15`, and `first sunday of quarter 00:00`. See more explanation about the format here.
Write Disposition	`writeDisposition`	`string`	Select a value from the drop down menu:`WRITE_TRUNCATEWRITE_APPEND`
Destination Table Name Template	`destinationTableNameTemplate`	`string`	The destination table name template can contain template variables such as `{run_date}` or `{run_time}`. See the documentation here.

Authentication#

Google Cloud uses API keys for authentication. When you connect your Google Cloud account, Pipedream securely stores the keys so you can easily authenticate to Google Cloud APIs in both code and no-code steps.

Create a service account in GCP and set the permissions you need for Pipedream workflows.
Generate a service account key
Download the key details in JSON format
Upload the key below.