Retrieve Site Performance Data with Google Search Console API on New Download Counts from npm API

Pipedream makes it easy to connect APIs for Google Search Console, npm and 2,500+ other apps remarkably fast.

Trigger workflow on

New Download Counts from the npm API

Next, do this

Retrieve Site Performance Data with the Google Search Console API

No credit card required

▶

Watch us build a workflow

8 min

Watch now ➜

Trusted by 1,000,000+ developers from startups to Fortune 500 companies

Developers ♥ Pipedream

Getting Started#

This integration creates a workflow with a npm trigger and Google Search Console action. When you configure and deploy the workflow, it will run on Pipedream's servers 24x7 for free.

Select this integration
Configure the New Download Counts trigger
1. Connect your npm account
2. Configure timer
3. Select a Period
4. Optional- Configure Package
Configure the Retrieve Site Performance Data action
1. Connect your Google Search Console account
2. Configure Verified Site URL
3. Configure Start Date (YYYY-MM-DD)
4. Configure End Date (YYYY-MM-DD)
5. Optional- Configure Dimensions
6. Optional- Select a Search Type
7. Optional- Select a Aggregation Type
8. Configure Max rows to return
9. Optional- Configure Start row (for pagination)
10. Optional- Configure Dimension Filters
11. Optional- Select a Data State
Deploy the workflow
Send a test event to validate your setup
Turn on the trigger

Details#

This integration uses pre-built, source-available components from Pipedream's GitHub repo. These components are developed by Pipedream and the community, and verified and maintained by Pipedream.

To contribute an update to an existing component or create a new component, create a PR on GitHub. If you're new to Pipedream component development, you can start with quickstarts for trigger span and action development, and then review the component API reference.

Trigger#

New Download Counts on npm

Description:Emit new event with the latest count of downloads for an npm package. [See the documentation](https://github.com/npm/registry/blob/main/docs/download-counts.md).

Version:0.1.0

Key:npm-download-counts

View on GitHub

Trigger Code#

import app from "../../npm.app.mjs";

export default {
  key: "npm-download-counts",
  name: "New Download Counts",
  description: "Emit new event with the latest count of downloads for an npm package. [See the documentation](https://github.com/npm/registry/blob/main/docs/download-counts.md).",
  version: "0.1.0",
  type: "source",
  props: {
    app,
    db: "$.service.db",
    timer: {
      type: "$.interface.timer",
      description: "One day interval time is recommended because NPM only update metrics once a day. [See the documentation](https://github.com/npm/registry/blob/main/docs/download-counts.md#data-source).",
      default: {
        intervalSeconds: 60 * 60 * 24,
      },
    },
    period: {
      type: "string",
      label: "Period",
      description: "Select last-day, last-week or last-month.",
      optional: false,
      default: "last-day",
      options: [
        "last-day",
        "last-week",
        "last-month",
      ],
    },
    packageName: {
      type: "string",
      label: "Package",
      description: "Enter an npm package name. Leave blank for all",
      optional: true,
    },
  },
  methods: {
    getDownloadCounts({
      period, packageName, ...args
    } = {}) {
      const basePath = `/downloads/point/${encodeURIComponent(period)}`;
      return this.app.makeRequest({
        path: packageName
          ? `${basePath}/${encodeURIComponent(packageName)}`
          : basePath,
        ...args,
      });
    },
  },
  async run({ timestamp: ts }) {
    const {
      getDownloadCounts,
      period,
      packageName,
    } = this;

    const response = await getDownloadCounts({
      period,
      packageName,
    });

    this.$emit(response, {
      id: ts,
      summary: `New Download Count ${response.downloads}`,
      ts,
    });
  },
};

Trigger Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI and CLI.

Label	Prop	Type	Description
npm	`app`	`app`	This component uses the npm app.
N/A	`db`	`$.service.db`	This component uses `$.service.db` to maintain state between executions.
	`timer`	`$.interface.timer`	One day interval time is recommended because NPM only update metrics once a day. See the documentation
Period	`period`	`string`	Select a value from the drop down menu:`last-daylast-weeklast-month`
Package	`packageName`	`string`	Enter an npm package name. Leave blank for all

Trigger Authentication#

The npm API does not require authentication.

About npm#

Node package manager

Action#

Retrieve Site Performance Data on Google Search Console

Description:Fetches search analytics from Google Search Console for a verified site.

Version:0.0.1

Key:google_search_console-retrieve-site-performance-data

View on GitHub

Google Search Console Overview#

The Google Search Console API opens a treasure trove of data and insights about your website’s presence in Google Search results. You can get detailed reports on your site's search traffic, manage and test your site's sitemaps and robots.txt files, and see which queries bring users to your site. On Pipedream, utilize this API to automate checks on site performance, integrate with other tools for deeper analysis, or keep tabs on your SEO strategy's effectiveness.

Action Code#

/**
 * Fetches performance data (clicks, impressions, etc.) for a verified site
 * via the Google Search Console Search Analytics API.
 *
 * Full usage docs in README.md
 */

import { axios } from "@pipedream/platform";
import gsConsole from "../../google_search_console.app.mjs";
import {
  removeCustomPropFields, trimIfString,
} from "../../common/utils.mjs";

/*
  Define prop metadata separately, including custom fields used for extended validation
  and runtime behavior.

  These extended fields (like `extendedType`, `postBody`, etc.) are not part of the standard
  Pipedream prop schema.

  A helper function (`removeCustomPropFields`) will later strip these non-standard fields,
  returning only valid Pipedream props for use in the UI.

  Keeping the full metadata in closure allows access to helpful context (e.g. validation rules)
  during runtime.
*/
const propsMeta = {

  siteUrl: {
    type: "string",
    extendedType: "url",
    label: "Verified Site URL",
    description: "Including https:// is strongly advised",
  },
  startDate: {
    type: "string",
    extendedType: "YYYY-MM-DD",
    label: "Start Date (YYYY-MM-DD)",
    postBody: true,
  },
  endDate: {
    type: "string",
    extendedType: "YYYY-MM-DD",
    label: "End Date (YYYY-MM-DD)",
    postBody: true,
  },
  dimensions: {
    type: "string[]",
    label: "Dimensions",
    optional: true,
    description: "e.g. ['query', 'page', 'country', 'device']",
    postBody: true,
  },
  searchType: {
    type: "string",
    label: "Search Type",
    optional: true,
    options: [
      "web",
      "image",
      "video",
      "news",
      "googleNews",
      "discover",
    ],
    default: "web",
    postBody: true,
  },
  aggregationType: {
    type: "string",
    label: "Aggregation Type",
    optional: true,
    options: [
      "auto",
      "byPage",
    ],
    postBody: true,
  },
  rowLimit: {
    type: "integer",
    label: "Max rows to return",
    default: 10,
    postBody: true,
  },
  startRow: {
    type: "integer",
    label: "Start row (for pagination)",
    optional: true,
    postBody: true,
  },
  dimensionFilterGroups: {
    type: "object",
    label: "Dimension Filters",
    optional: true,
    description: "Follow Search Console API structure for filters",
    postBody: true,
  },
  dataState: {
    type: "string",
    label: "Data State",
    optional: true,
    options: [
      "all",
      "final",
    ],
    default: "final",
    postBody: true,
  },
};

export default {
  name: "Retrieve Site Performance Data",
  description: "Fetches search analytics from Google Search Console for a verified site.",
  key: "google_search_console-retrieve-site-performance-data",
  version: "0.0.1",
  type: "action",
  props: {
    gsConsole,
    // Remove non-standard fields and expose only valid props to Pipedream UI
    ...removeCustomPropFields(propsMeta),
  },

  //=================== RUN ==============================
  //======================================================

  async run({ $ }) {

    /*
      `dimensionFilterGroups` is expected to be an object.
      If a JSON string is passed instead (e.g. from UI input), attempt to parse it.
      - Returns parsed object if successful
      - Returns original input if not a string
      - Throws a descriptive error if JSON parsing fails
    */
    this.dimensionFilterGroups = this.gsConsole.parseIfJsonString(this.dimensionFilterGroups);

    // Prepare the POST request payload
    const body = {};

    // Accumulator for non-blocking input warnings
    const warnings = [];

    /*
      This loop:
      - Trims and validates all defined props
      - Skips empty optional fields
      - Accumulates non-blocking warnings
      - Adds valid props to the POST request payload (`body`) if marked with `postBody: true`
    */
    for (let propName in propsMeta) {

      // Just for convenience.
      const meta = propsMeta[propName];

      // Trim the value if it's a string
      this[propName] = trimIfString(this[propName]);

      // Skip if the prop is optional and empty (null, undefined, or blank string)
      if (meta.optional === true && ((this[propName] ?? "") === "")) continue;

      // Validate input (may throw or return warning messages)
      const validationResult = this.gsConsole.validateUserInput(meta, this[propName]);

      // Push the warnings into warnings accumulator if any.
      if (validationResult.warnings) warnings.push(...validationResult.warnings);

      // Include prop in the request body if marked as postBody
      if (meta.postBody === true) body[propName] = this[propName];
    };

    // Already trimmed earlier
    const url = this.siteUrl;

    // Response of the POST request.
    let response;

    try {
      response = await axios($, {
        method: "POST",
        url: `https://searchconsole.googleapis.com/webmasters/v3/sites/${encodeURIComponent(url)}/searchAnalytics/query`,
        headers: {
          "Authorization": `Bearer ${this.gsConsole.$auth.oauth_access_token}`,
          "Content-Type": "application/json",
        },
        data: body,
      });

    } catch (error) {
      // Identify if the error was thrown by internal validation or by the API call
      const thrower = this.gsConsole.checkWhoThrewError(error);

      throw new Error(`Failed to fetch data ( ${thrower.whoThrew} error ) : ${error.message}. ` + warnings.join("\n- "));
    };

    // Output summary and any warnings for the user
    $.export("$summary", ` Fetched ${response.rows?.length || 0} rows of data. ` + warnings.join("\n- "));
    return response;
  },
};

Action Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI.

Label	Prop	Type	Description
Google Search Console	`gsConsole`	`app`	This component uses the Google Search Console app.
Verified Site URL	`siteUrl`	`string`	Including https:// is strongly advised
Start Date (YYYY-MM-DD)	`startDate`	`string`
End Date (YYYY-MM-DD)	`endDate`	`string`
Dimensions	`dimensions`	`string[]`	e.g. ['query', 'page', 'country', 'device']
Search Type	`searchType`	`string`	Select a value from the drop down menu:`webimagevideonewsgoogleNewsdiscover`
Aggregation Type	`aggregationType`	`string`	Select a value from the drop down menu:`autobyPage`
Max rows to return	`rowLimit`	`integer`
Start row (for pagination)	`startRow`	`integer`
Dimension Filters	`dimensionFilterGroups`	`object`	Follow Search Console API structure for filters
Data State	`dataState`	`string`	Select a value from the drop down menu:`allfinal`

Action Authentication#

Google Search Console uses OAuth authentication. When you connect your Google Search Console account, Pipedream will open a popup window where you can sign into Google Search Console and grant Pipedream permission to connect to your account. Pipedream securely stores and automatically refreshes the OAuth tokens so you can easily authenticate any Google Search Console API.

Pipedream requests the following authorization scopes when you connect your account:

emailprofilehttps://www.googleapis.com/auth/webmastershttps://www.googleapis.com/auth/webmasters.readonlyhttps://www.googleapis.com/auth/siteverificationhttps://www.googleapis.com/auth/indexing