← AWS

New Records Returned by CloudWatch Logs Insights Query from AWS API

Pipedream makes it easy to connect APIs for AWS and 900+ other apps remarkably fast.

Trigger workflow on
New Records Returned by CloudWatch Logs Insights Query from the AWS API
Next, do this
Connect to 900+ APIs using code and no-code building blocks
No credit card required
Into to Pipedream
Watch us build a workflow
Watch us build a workflow
7 min
Watch now ➜
Trusted by 250,000+ developers from startups to Fortune 500 companies:
Trusted by 250,000+ developers from startups to Fortune 500 companies

Developers Pipedream

Getting Started

Trigger a workflow on New Records Returned by CloudWatch Logs Insights Query with AWS API. When you configure and deploy the workflow, it will run on Pipedream's servers 24x7 for free.

  1. Configure the New Records Returned by CloudWatch Logs Insights Query trigger
    1. Connect your AWS account
    2. Select a AWS Region
    3. Select one or more CloudWatch Log Groups
    4. Configure Logs Insights Query
    5. Optional- Configure Emit query results as a single event
    6. Configure Polling schedule
  2. Add steps to connect to 900+ APIs using code and no-code building blocks
  3. Deploy the workflow
  4. Send a test event to validate your setup
  5. Turn on the trigger

Integrations

Send Message with Discord Webhook API on New Records Returned by CloudWatch Logs Insights Query from AWS API
AWS + Discord Webhook
 
Try it
Add Multiple Rows with Google Sheets API on New Records Returned by CloudWatch Logs Insights Query from AWS API
AWS + Google Sheets
 
Try it
Get Film with SWAPI API on New Records Returned by CloudWatch Logs Insights Query from AWS API
AWS + SWAPI - Star Wars
 
Try it
Create Multiple Records with Airtable API on New Records Returned by CloudWatch Logs Insights Query from AWS API
AWS + Airtable
 
Try it
Custom Request with HTTP / Webhook API on New Records Returned by CloudWatch Logs Insights Query from AWS API
AWS + HTTP / Webhook
 
Try it

Details

This is a pre-built, open source component from Pipedream's GitHub repo. The component is developed by Pipedream and the community, and verified and maintained by Pipedream.

To contribute an update to an existing component or create a new component, create a PR on GitHub. If you're new to Pipedream component development, you can start with quickstarts for trigger span and action development, and then review the component API reference.

New Records Returned by CloudWatch Logs Insights Query on AWS
Description:Executes a CloudWatch Logs Insights query on a schedule, and emits the records as invidual events (default) or in batch
Version:0.2.0
Key:aws-new-records-returned-by-cloudwatch-logs-insights-query

Code

import common from "../../common/common-cloudwatch-logs.mjs";

export default {
  ...common,
  key: "aws-new-records-returned-by-cloudwatch-logs-insights-query",
  name: "New Records Returned by CloudWatch Logs Insights Query",
  // eslint-disable-next-line pipedream/source-description
  description: "Executes a CloudWatch Logs Insights query on a schedule, and emits the records as invidual events (default) or in batch",
  version: "0.2.0",
  type: "source",
  props: {
    aws: common.props.aws,
    region: common.props.region,
    db: "$.service.db",
    // eslint-disable-next-line pipedream/props-label
    logGroupNames: {
      ...common.props.logGroupName,
      type: "string[]",
      description: "The log groups you'd like to query",
    },
    queryString: {
      label: "Logs Insights Query",
      description: "The query you'd like to run. See [this AWS doc](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html) for help with query syntax",
      type: "string",
    },
    emitResultsInBatch: {
      type: "boolean",
      label: "Emit query results as a single event",
      description: "If `true`, all events are emitted as an array, within a single Pipedream event. If `false`, each row of results is emitted as its own event. Defaults to `true`",
      optional: true,
      default: true,
    },
    timer: {
      label: "Polling schedule",
      description: "How often you want to query CloudWatch Logs Insights for results",
      type: "$.interface.timer",
      default: {
        intervalSeconds: 5 * 60,
      },
    },
  },
  async run() {
    const now = +new Date();
    const startTime = this.db.get("startTime") || now - 1000 * 60 * 60;

    // First, start the query
    const params = {
      queryString: this.queryString,
      startTime,
      endTime: now,
      logGroupNames: this.logGroupNames,
    };

    const { queryId } = await this.startQuery(params);

    // Then poll for its status, emitting each record as its own event when completed
    async function sleep(ms) {
      return new Promise((r) => setTimeout(r, ms));
    }

    let result, res;
    do {
      await sleep(1000);
      res = await this.getQueryResults({
        queryId,
      });
      result = res.status;
    } while (result === "Running" || result === "Scheduled");

    if (result !== "Complete") {
      throw new Error(`Query ${queryId} failed with status ${result}`);
    }

    console.log(JSON.stringify(res, null, 2));
    const { results } = res;

    if (!results || !results.length) {
      console.log("No results, exiting");
      this.db.set("startTime", now);
      return;
    }

    if (this.emitResultsInBatch === true) {
      this.$emit(results, {
        summary: JSON.stringify(results),
      });
    } else {
      for (const item of results) {
        this.$emit(item, {
          summary: JSON.stringify(item),
        });
      }
    }

    // The next time this source runs, query for data starting now
    this.db.set("startTime", now);
  },
};

Configuration

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI and CLI.
LabelPropTypeDescription
AWSawsappThis component uses the AWS app.
AWS RegionregionstringSelect a value from the drop down menu.
N/Adb$.service.dbThis component uses $.service.db to maintain state between component invocations.
CloudWatch Log GroupslogGroupNamesstring[]Select a value from the drop down menu.
Logs Insights QueryqueryStringstring

The query you'd like to run. See this AWS doc for help with query syntax

Emit query results as a single eventemitResultsInBatchboolean

If true, all events are emitted as an array, within a single Pipedream event. If false, each row of results is emitted as its own event. Defaults to true

Polling scheduletimer$.interface.timer

How often you want to query CloudWatch Logs Insights for results

Authentication

AWS uses API keys for authentication. When you connect your AWS account, Pipedream securely stores the keys so you can easily authenticate to AWS APIs in both code and no-code steps.

Follow the AWS Instructions for creating an IAM user with an associated access and secret key.

As a best practice, attach the minimum set of IAM permissions necessary to perform the specific task in Pipedream. If your workflow only needs to perform a single API call, you should create a user and associate an IAM group / policy with permission to do only that task. You can create as many linked AWS accounts in Pipedream as you'd like.

Enter your access and secret key below.

About AWS

On-demand cloud computing platforms and APIs

More Ways to Use AWS

Triggers

New Scheduled Tasks from the AWS API

Creates a Step Function State Machine to publish a message to an SNS topic at a specific timestamp. The SNS topic delivers the message to this Pipedream source, and the source emits it as a new event.

 
Try it
New SNS Messages from the AWS API

Creates an SNS topic in your AWS account. Messages published to this topic are emitted from the Pipedream source.

 
Try it
New Inbound SES Emails from the AWS API

The source subscribes to all emails delivered to a specific domain configured in AWS SES. When an email is sent to any address at the domain, this event source emits that email as a formatted event. These events can trigger a Pipedream workflow and can be consumed via SSE or REST API.

 
Try it
New Deleted S3 File from the AWS API

Emit new event when a file is deleted from a S3 bucket

 
Try it
New Restored S3 File from the AWS API

Emit new event when an file is restored into a S3 bucket

 
Try it

Actions

CloudWatch Logs - Put Log Event with the AWS API

Uploads a log event to the specified log stream. See docs

 
Try it
DynamoDB - Create Table with the AWS API

Creates a new table to your account. See docs

 
Try it
DynamoDB - Execute Statement with the AWS API

This operation allows you to perform transactional reads or writes on data stored in DynamoDB, using PartiQL. See docs

 
Try it
DynamoDB - Get Item with the AWS API

The Get Item operation returns a set of attributes for the item with the given primary key. If there is no matching item, Get Item does not return any data and there will be no Item element in the response. See docs

 
Try it
DynamoDB - Put Item with the AWS API

Creates a new item, or replaces an old item with a new item. If an item that has the same primary key as the new item already exists in the specified table, the new item completely replaces the existing item. See docs

 
Try it