S3 - Stream file to S3 from URL with AWS API

Pipedream makes it easy to connect APIs for AWS and 2,400+ other apps remarkably fast.

Trigger workflow on

HTTP requests, schedules and app events

Next, do this

S3 - Stream file to S3 from URL with the AWS API

No credit card required

▶

Watch us build a workflow

8 min

Watch now ➜

Trusted by 1,000,000+ developers from startups to Fortune 500 companies

Developers ♥ Pipedream

Getting Started#

Create a workflow to S3 - Stream file to S3 from URL with the AWS API. When you configure and deploy the workflow, it will run on Pipedream's servers 24x7 for free.

Configure the S3 - Stream file to S3 from URL action
1. Connect your AWS account
2. Select a AWS Region
3. Select a S3 Bucket Name
4. Configure File URL
5. Configure S3 Filename Key
Select a trigger to run your workflow on HTTP requests, schedules or app events
Deploy the workflow
Send a test event to validate your setup
Turn on the trigger

Integrations#

S3 - Stream file to S3 from URL with AWS API on New Requests (Payload Only) from HTTP / Webhook API

HTTP / Webhook + AWS

Try it

S3 - Stream file to S3 from URL with AWS API on New Message from Discord API

Discord + AWS

Try it

S3 - Stream file to S3 from URL with AWS API on New Message In Channels from Slack API

Slack + AWS

Try it

S3 - Stream file to S3 from URL with AWS API on New Message in Channel from Discord Bot API

Discord Bot + AWS

Try it

S3 - Stream file to S3 from URL with AWS API on New Submission from Typeform API

Typeform + AWS

Try it

Details#

This is a pre-built, source-available component from Pipedream's GitHub repo. The component is developed by Pipedream and the community, and verified and maintained by Pipedream.

To contribute an update to an existing component or create a new component, create a PR on GitHub. If you're new to Pipedream component development, you can start with quickstarts for trigger span and action development, and then review the component API reference.

S3 - Stream file to S3 from URL on AWS

Description:Accepts a file URL, and streams the file to the provided S3 bucket/key. [See the docs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html)

Version:0.4.3

Key:aws-s3-stream-file

View on GitHub

Code#

import fs from "fs";
import common from "../../common/common-s3.mjs";
import { toSingleLineString } from "../../common/utils.mjs";

export default {
  ...common,
  key: "aws-s3-stream-file",
  name: "S3 - Stream file to S3 from URL",
  description: toSingleLineString(`
    Accepts a file URL, and streams the file to the provided S3 bucket/key.
    [See the docs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html)
  `),
  version: "0.4.3",
  type: "action",
  props: {
    aws: common.props.aws,
    region: common.props.region,
    bucket: common.props.bucket,
    fileUrl: common.props.fileUrl,
    filename: common.props.key,
  },
  methods: {
    ...common.methods,
    async getStreamAndContentLength(fileResponse, absoluteFilePath) {
      if (fileResponse.headers["content-length"]) {
        return {
          stream: fileResponse.data,
          contentLength: fileResponse.headers["content-length"],
        };
      }
      const fileName = absoluteFilePath.split("/").pop();
      const tempLocalPath = `/tmp/${fileName}`;
      await (new Promise((resolve) => {
        const writeStream = fs.createWriteStream(tempLocalPath);
        fileResponse.data.pipe(writeStream);
        writeStream.on("finish", resolve);
      }));
      const { size } = fs.statSync(tempLocalPath);
      return {
        stream: fs.createReadStream(tempLocalPath),
        contentLength: size,
      };
    },
  },
  async run({ $ }) {
    const fileResponse = await this.streamFile(this.fileUrl);
    const filePath = this.filename.replace(/^\/+/, "");
    const {
      stream,
      contentLength,
    } = await this.getStreamAndContentLength(fileResponse, filePath);
    const response = await this.uploadFile({
      Bucket: this.bucket,
      Key: filePath,
      Body: stream,
      ContentType: fileResponse.headers["content-type"],
      ContentLength: contentLength,
    });
    $.export("$summary", `Streaming file ${this.filename} to ${this.bucket}`);
    return response;
  },
};

Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI and CLI.

Label	Prop	Type	Description
AWS	`aws`	`app`	This component uses the AWS app.
AWS Region	`region`	`string`	Select a value from the drop down menu.
S3 Bucket Name	`bucket`	`string`	Select a value from the drop down menu.
File URL	`fileUrl`	`string`	The absolute URL of the file you'd like to upload
S3 Filename Key	`filename`	`string`	The name of the S3 key with extension you'd like to upload this file to

Authentication#

AWS uses API keys for authentication. When you connect your AWS account, Pipedream securely stores the keys so you can easily authenticate to AWS APIs in both code and no-code steps.

Follow the AWS Instructions for creating an IAM user with an associated access and secret key.

As a best practice, attach the minimum set of IAM permissions necessary to perform the specific task in Pipedream. If your workflow only needs to perform a single API call, you should create a user and associate an IAM group / policy with permission to do only that task. You can create as many linked AWS accounts in Pipedream as you'd like.

Enter your access and secret key below.

About AWS#

Amazon Web Services (AWS) offers reliable, scalable, and inexpensive cloud computing services.

More Ways to Use AWS#

Triggers#

New Scheduled Tasks from the AWS API

Creates a Step Function State Machine to publish a message to an SNS topic at a specific timestamp. The SNS topic delivers the message to this Pipedream source, and the source emits it as a new event.

Try it

New SNS Messages from the AWS API

Creates an SNS topic in your AWS account. Messages published to this topic are emitted from the Pipedream source.

Try it

New Inbound SES Emails from the AWS API

The source subscribes to all emails delivered to a specific domain configured in AWS SES. When an email is sent to any address at the domain, this event source emits that email as a formatted event. These events can trigger a Pipedream workflow and can be consumed via SSE or REST API.

Try it

New Deleted S3 File from the AWS API

Emit new event when a file is deleted from a S3 bucket

Try it

New DynamoDB Stream Event from the AWS API

Emit new event when a DynamoDB stream receives new events. See the docs here

Try it

Actions#

CloudWatch Logs - Put Log Event with the AWS API

Uploads a log event to the specified log stream. See docs

Try it

DynamoDB - Create Table with the AWS API

Creates a new table to your account. See docs

Try it

DynamoDB - Execute Statement with the AWS API

This operation allows you to perform transactional reads or writes on data stored in DynamoDB, using PartiQL. See docs

Try it

DynamoDB - Get Item with the AWS API

The Get Item operation returns a set of attributes for the item with the given primary key. If there is no matching item, Get Item does not return any data and there will be no Item element in the response. See docs

Try it

DynamoDB - Put Item with the AWS API

Creates a new item, or replaces an old item with a new item. If an item that has the same primary key as the new item already exists in the specified table, the new item completely replaces the existing item. See docs

Try it