Insert Multiple Rows with Snowflake API on New Matched Traffic Stats from Google Postmaster Tools API

Pipedream makes it easy to connect APIs for Snowflake, Google Postmaster Tools and 2,400+ other apps remarkably fast.

Trigger workflow on

New Matched Traffic Stats from the Google Postmaster Tools API

Next, do this

Insert Multiple Rows with the Snowflake API

No credit card required

▶

Watch us build a workflow

8 min

Watch now ➜

Trusted by 1,000,000+ developers from startups to Fortune 500 companies

Developers ♥ Pipedream

Getting Started#

This integration creates a workflow with a Google Postmaster Tools trigger and Snowflake action. When you configure and deploy the workflow, it will run on Pipedream's servers 24x7 for free.

Select this integration
Configure the New Matched Traffic Stats trigger
1. Connect your Google Postmaster Tools account
2. Configure timer
3. Select a Domain Name
4. Configure filterInfo
5. Optional- Configure Match All Filters
6. Optional- Select one or more IP Reputation
7. Optional- Select one or more Domain Reputation
8. Optional- Configure User Reported Spam Ratio Greater Than or Equal To
9. Optional- Configure SPF Success Ratio Less Than
10. Optional- Configure DKIM Success Ratio Less Than
11. Optional- Configure DMARC Success Ratio Less Than
12. Optional- Configure Outbound Encryption Ratio Less Than
13. Optional- Configure Inbound Encryption Ratio Less Than
14. Optional- Configure Error Ratio Greater Than or Equal To
15. Optional- Select a Error Categories
Configure the Insert Multiple Rows action
1. Connect your Snowflake account
2. Select a Database
3. Select a Schema
4. Select a Table Name
5. Select one or more Columns
6. Configure Row Values
Deploy the workflow
Send a test event to validate your setup
Turn on the trigger

Details#

This integration uses pre-built, source-available components from Pipedream's GitHub repo. These components are developed by Pipedream and the community, and verified and maintained by Pipedream.

To contribute an update to an existing component or create a new component, create a PR on GitHub. If you're new to Pipedream component development, you can start with quickstarts for trigger span and action development, and then review the component API reference.

Trigger#

New Matched Traffic Stats on Google Postmaster Tools

Description:Emits a new event when traffic stats match certain criteria. [See the documentation](https://developers.google.com/gmail/postmaster/reference/rest)

Version:0.1.0

Key:google_postmaster_tools_api-new-matched-traffic-stats

View on GitHub

Google Postmaster Tools Overview#

The Google Postmaster Tools API offers insights into the performance and deliverability of emails sent to Gmail users. It provides data on spam rates, feedback loops, delivery errors, and more, allowing senders to optimize email campaigns and enhance inbox placement. In Pipedream, you can harness this data to automate monitoring, analyze sending patterns, and integrate email performance metrics with other services for comprehensive analysis and action.

Trigger Code#

import { DEFAULT_POLLING_SOURCE_TIMER_INTERVAL } from "@pipedream/platform";
import googlePostmasterToolsApi from "../../google_postmaster_tools_api.app.mjs";
import sampleEmit from "./test-event.mjs";

export default {
  key: "google_postmaster_tools_api-new-matched-traffic-stats",
  name: "New Matched Traffic Stats",
  description:
    "Emits a new event when traffic stats match certain criteria. [See the documentation](https://developers.google.com/gmail/postmaster/reference/rest)",
  version: "0.1.0",
  type: "source",
  dedupe: "unique",
  sampleEmit,
  props: {
    googlePostmasterToolsApi,
    db: "$.service.db",
    timer: {
      type: "$.interface.timer",
      default: {
        intervalSeconds: DEFAULT_POLLING_SOURCE_TIMER_INTERVAL,
      },
    },
    domain: {
      propDefinition: [
        googlePostmasterToolsApi,
        "domain",
      ],
    },
    filterInfo: {
      type: "alert",
      alertType: "info",
      content: "By default, events will be emitted when matching **any** of the configured filters. If you want **all** configured filters to be required instead, you can use the `Match All Filters` prop below.",
    },
    matchAllFilters: {
      propDefinition: [
        googlePostmasterToolsApi,
        "matchAllFilters",
      ],
    },
    ipReputation: {
      propDefinition: [
        googlePostmasterToolsApi,
        "ipReputation",
      ],
    },
    domainReputation: {
      propDefinition: [
        googlePostmasterToolsApi,
        "domainReputation",
      ],
    },
    userReportedSpamRatio: {
      propDefinition: [
        googlePostmasterToolsApi,
        "userReportedSpamRatio",
      ],
    },
    spfSuccessRatio: {
      propDefinition: [
        googlePostmasterToolsApi,
        "spfSuccessRatio",
      ],
    },
    dkimSuccessRatio: {
      propDefinition: [
        googlePostmasterToolsApi,
        "dkimSuccessRatio",
      ],
    },
    dmarcSuccessRatio: {
      propDefinition: [
        googlePostmasterToolsApi,
        "dmarcSuccessRatio",
      ],
    },
    outboundEncryptionRatio: {
      propDefinition: [
        googlePostmasterToolsApi,
        "outboundEncryptionRatio",
      ],
    },
    inboundEncryptionRatio: {
      propDefinition: [
        googlePostmasterToolsApi,
        "inboundEncryptionRatio",
      ],
    },
    errorRatio: {
      propDefinition: [
        googlePostmasterToolsApi,
        "errorRatio",
      ],
    },
    errorCategories: {
      propDefinition: [
        googlePostmasterToolsApi,
        "errorCategories",
      ],
    },
  },
  methods: {
    _getSavedItems() {
      return this.db.get("savedItems") ?? [];
    },
    _setSavedItems(value) {
      this.db.set("savedItems", value);
    },
    getDateValues(date) {
      const [
        year,
        month,
        day,
      ] = date.toISOString().split("T")[0].split("-").map((i) => Number(i));
      return {
        year,
        month,
        day,
      };
    },
    async getTrafficStats() {
      const today = new Date();
      const oneDayAgo = new Date(today);
      oneDayAgo.setDate(oneDayAgo.getDate() - 7);
      const endDate = this.getDateValues(today);
      const startDate = this.getDateValues(oneDayAgo);

      return this.googlePostmasterToolsApi.getDomainTrafficStats({
        domainName: this.domain,
        params: {
          "startDate.day": startDate.day,
          "startDate.month": startDate.month,
          "startDate.year": startDate.year,
          "endDate.day": endDate.day,
          "endDate.month": endDate.month,
          "endDate.year": endDate.year,
        },
      });
    },
    filterIpReputation(item) {
      let { ipReputation } = this;
      if (typeof ipReputation === "string")
        ipReputation = ipReputation.split(",");
      if (!ipReputation?.length) return undefined;
      return item.ipReputations.some(
        ({
          reputation, ipCount,
        }) =>
          ipReputation.includes(reputation) && ipCount > 0,
      );
    },
    filterDomainReputation(item) {
      let { domainReputation } = this;
      if (typeof domainReputation === "string")
        domainReputation = domainReputation.split(",");
      if (!domainReputation?.length) return undefined;
      return domainReputation.includes(item.domainReputation);
    },
    filterRatio(prop, value, greaterOrEqual = false) {
      if (prop?.endsWith("%")) prop = Number(prop.slice(0, -1)) / 100;
      const ratio = Number(prop);
      if (isNaN(ratio) || value === undefined) return undefined;
      return greaterOrEqual
        ? value >= ratio
        : value < ratio;
    },
    filterSpamRatio(item) {
      return this.filterRatio(
        this.userReportedSpamRatio,
        item.userReportedSpamRatio,
        true,
      );
    },
    filterSpfSuccessRatio(item) {
      return this.filterRatio(this.spfSuccessRatio, item.spfSuccessRatio);
    },
    filterDkimSuccessRatio(item) {
      return this.filterRatio(this.dkimSuccessRatio, item.dkimSuccessRatio);
    },
    filterDmarcSuccessRatio(item) {
      return this.filterRatio(this.dmarcSuccessRatio, item.dmarcSuccessRatio);
    },
    filterOutboundEncryptionRatio(item) {
      return this.filterRatio(
        this.outboundEncryptionRatio,
        item.outboundEncryptionRatio,
      );
    },
    filterInboundEncryptionRatio(item) {
      return this.filterRatio(
        this.inboundEncryptionRatio,
        item.inboundEncryptionRatio,
      );
    },
    filterErrorRatio(item) {
      const ratio = Number(this.errorRatio);
      if (isNaN(ratio)) return undefined;

      let { errorCategories } = this;
      if (typeof errorCategories === "string")
        errorCategories = errorCategories.split(",");

      return item.deliveryErrors.some(
        ({
          errorType, errorRatio,
        }) =>
          (!errorCategories || errorCategories.includes(errorType)) && errorRatio > ratio,
      );
    },
    matchesCriteria(item) {
      // Filters return undefined if the prop is not set, or true/false otherwise
      // Filters are a logical AND - if any filter returns false, the item is not emitted
      const filters = [
        this.filterIpReputation,
        this.filterDomainReputation,
        this.filterSpamRatio,
        this.filterSpfSuccessRatio,
        this.filterDkimSuccessRatio,
        this.filterDmarcSuccessRatio,
        this.filterOutboundEncryptionRatio,
        this.filterInboundEncryptionRatio,
        this.filterErrorRatio,
      ];

      let hasMatch = false;

      for (let filter of filters) {
        const result = filter(item);
        if (result === true) {
          if (!this.matchAllFilters) return true;
          hasMatch = true;
        } else if (result === false && this.matchAllFilters) {
          return false;
        }
      }

      return hasMatch;
    },
  },
  async run() {
    const savedItems = this._getSavedItems();
    const ts = Date.now();
    const stats = await this.getTrafficStats();
    stats?.trafficStats?.filter(({ name }) => !savedItems.includes(name)).forEach((item) => {
      const id = item.name;
      if (this.matchesCriteria(item)) {
        this.$emit(item, {
          id,
          summary: `Matched Traffic Stats: ${id}`,
          ts,
        });
        savedItems.push(id);
      }
    });
    this._setSavedItems(savedItems);
  },
};

Trigger Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI and CLI.

Label	Prop	Type	Description
Google Postmaster Tools	`googlePostmasterToolsApi`	`app`	This component uses the Google Postmaster Tools app.
N/A	`db`	`$.service.db`	This component uses `$.service.db` to maintain state between executions.
	`timer`	`$.interface.timer`
Domain Name	`domain`	`string`	Select a value from the drop down menu.
Match All Filters	`matchAllFilters`	`boolean`	If `true`, events will only be emitted if they match all of the selected criteria. The default behavior is matching any of them.
IP Reputation	`ipReputation`	`string[]`	Select a value from the drop down menu:`{ "label": "High", "value": "HIGH" }{ "label": "Medium", "value": "MEDIUM" }{ "label": "Low", "value": "LOW" }{ "label": "Bad", "value": "BAD" }`
Domain Reputation	`domainReputation`	`string[]`	Select a value from the drop down menu:`{ "label": "High", "value": "HIGH" }{ "label": "Medium", "value": "MEDIUM" }{ "label": "Low", "value": "LOW" }{ "label": "Bad", "value": "BAD" }`
User Reported Spam Ratio Greater Than or Equal To	`userReportedSpamRatio`	`string`	Emit events only when the user reported spam ratio is greater than, or equal to, the specified value (can be a percentage, e.g. `10%` or a ratio between 0 and 1, e.g. `0.1`)
SPF Success Ratio Less Than	`spfSuccessRatio`	`string`	Emit events only when the SPF success ratio is less than the specified value (can be a percentage, e.g. `10%` or a ratio between 0 and 1, e.g. `0.1`)
DKIM Success Ratio Less Than	`dkimSuccessRatio`	`string`	Emit events only when the DKIM success ratio is less than the specified value (can be a percentage, e.g. `10%` or a ratio between 0 and 1, e.g. `0.1`)
DMARC Success Ratio Less Than	`dmarcSuccessRatio`	`string`	Emit events only when the DMARC success ratio is less than the specified value (can be a percentage, e.g. `10%` or a ratio between 0 and 1, e.g. `0.1`)
Outbound Encryption Ratio Less Than	`outboundEncryptionRatio`	`string`	Emit events only when the outbound encryption ratio is less than the specified value (can be a percentage, e.g. `10%` or a ratio between 0 and 1, e.g. `0.1`)
Inbound Encryption Ratio Less Than	`inboundEncryptionRatio`	`string`	Emit events only when the inbound encryption ratio is less than the specified value (can be a percentage, e.g. `10%` or a ratio between 0 and 1, e.g. `0.1`)
Error Ratio Greater Than or Equal To	`errorRatio`	`string`	Emit events only when the error ratio (in the specified `Error Categories`, or any if not specified) is greater than, or equal to, the specified value (can be a percentage, e.g. `10%` or a ratio between 0 and 1, e.g. `0.1`)
Error Categories	`errorCategories`	`string`	Select a value from the drop down menu:`{ "value": "RATE_LIMIT_EXCEEDED", "label": "The Domain or IP is sending traffic at a suspiciously high rate, due to which temporary rate limits have been imposed. The limit will be lifted when Gmail is confident enough of the nature of the traffic." }{ "value": "SUSPECTED_SPAM", "label": "The traffic is suspected to be spam, by Gmail, for various reasons." }{ "value": "CONTENT_SPAMMY", "label": "The traffic is suspected to be spammy, specific to the content." }{ "value": "BAD_ATTACHMENT", "label": "Traffic contains attachments not supported by Gmail." }{ "value": "BAD_DMARC_POLICY", "label": "The sender domain has set up a DMARC rejection policy." }{ "value": "LOW_IP_REPUTATION", "label": "The IP reputation of the sending IP is very low." }{ "value": "LOW_DOMAIN_REPUTATION", "label": "The Domain reputation of the sending domain is very low." }{ "value": "IP_IN_RBL", "label": "The IP is listed in one or more public Real-time Blackhole Lists. Work with the RBL to get your IP delisted." }{ "value": "DOMAIN_IN_RBL", "label": "The Domain is listed in one or more public Real-time Blackhole Lists. Work with the RBL to get your domain delisted." }{ "value": "BAD_PTR_RECORD", "label": "The sending IP is missing a PTR record." }`

Trigger Authentication#

Google Postmaster Tools uses OAuth authentication. When you connect your Google Postmaster Tools account, Pipedream will open a popup window where you can sign into Google Postmaster Tools and grant Pipedream permission to connect to your account. Pipedream securely stores and automatically refreshes the OAuth tokens so you can easily authenticate any Google Postmaster Tools API.

Pipedream requests the following authorization scopes when you connect your account:

emailprofilehttps://www.googleapis.com/auth/postmaster.readonly

About Google Postmaster Tools#

Use Postmaster Tools to analyze your email performance, and help Gmail route your messages to the right place

Action#

Insert Multiple Rows on Snowflake

Description:Insert multiple rows into a table

Version:0.1.2

Key:snowflake-insert-multiple-rows

View on GitHub

Snowflake Overview#

Snowflake offers a cloud database and related tools to help developers create robust, secure, and scalable data warehouses. See Snowflake's Key Concepts & Architecture.

Getting Started

1. Create a user, role and warehouse in Snowflake

Snowflake recommends you create a new user, role, and warehouse when you integrate a third-party tool like Pipedream. This way, you can control permissions via the user / role, and separate Pipedream compute and costs with the warehouse. You can do this directly in the Snowflake UI.

We recommend you create a read-only account if you only need to query Snowflake. If you need to insert data into Snowflake, add permissions on the appropriate objects after you create your user.

2. Enter those details in Pipedream

Visit https://pipedream.com/accounts. Click the button to Connect an App. Enter the required Snowflake account data.

You'll only need to connect your account once in Pipedream. You can connect this account to multiple workflows to run queries against Snowflake, insert data, and more.

3. Build your first workflow

Visit https://pipedream.com/new to build your first workflow. Pipedream workflows let you connect Snowflake with 1,000+ other apps. You can trigger workflows on Snowflake queries, sending results to Slack, Google Sheets, or any app that exposes an API. Or you can accept data from another app, transform it with Python, Node.js, Go or Bash code, and insert it into Snowflake.

Learn more at Pipedream University.

Action Code#

import snowflake from "../../snowflake.app.mjs";
import { ConfigurationError } from "@pipedream/platform";

export default {
  type: "action",
  key: "snowflake-insert-multiple-rows",
  name: "Insert Multiple Rows",
  description: "Insert multiple rows into a table",
  version: "0.1.2",
  props: {
    snowflake,
    database: {
      propDefinition: [
        snowflake,
        "database",
      ],
    },
    schema: {
      propDefinition: [
        snowflake,
        "schema",
        (c) =>  ({
          database: c.database,
        }),
      ],
    },
    tableName: {
      propDefinition: [
        snowflake,
        "tableName",
        (c) => ({
          database: c.database,
          schema: c.schema,
        }),
      ],
      description: "The table where you want to add rows",
    },
    columns: {
      propDefinition: [
        snowflake,
        "columns",
        (c) => ({
          tableName: c.tableName,
        }),
      ],
    },
    values: {
      propDefinition: [
        snowflake,
        "values",
      ],
    },
  },
  async run({ $ }) {
    let rows = this.values;

    let inputValidated = true;

    if (!Array.isArray(rows)) {
      rows = JSON.parse(rows);
    }

    if (!rows || !rows.length || !Array.isArray(rows)) {
      inputValidated = false;
    } else {
      rows.forEach((row) => { if (!Array.isArray(row)) { inputValidated = false; } });
    }

    // Throw an error if input validation failed
    if (!inputValidated) {
      throw new ConfigurationError("The row data you passed is not an array of arrays. Please enter an array of arrays in the `Values` parameter above. If you're trying to add a single row to Snowflake, select the **Insert Single Row** action.");
    }

    const response = await this.snowflake.insertRows(this.tableName, this.columns, this.values);
    $.export("$summary", `Successfully inserted ${this.values.length} rows in ${this.tableName}`);
    return response;
  },
};

Action Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI.

Label	Prop	Type	Description
Snowflake	`snowflake`	`app`	This component uses the Snowflake app.
Database	`database`	`string`	Select a value from the drop down menu.
Schema	`schema`	`string`	Select a value from the drop down menu.
Table Name	`tableName`	`string`	Select a value from the drop down menu.
Columns	`columns`	`string[]`	Select a value from the drop down menu.
Row Values	`values`	`string`	Provide an array of arrays. Each nested array should represent a row, with each element of the nested array representing a value (e.g., passing `[["Foo",1,2],["Bar",3,4]]` will insert two rows of data with three columns each). The most common pattern is to reference an array of arrays exported by a previous step (e.g., `{{steps.foo.$return_value}}`). You may also enter or construct a string that will `JSON.parse()` to an array of arrays.