File Stores: Cloud File Storage for your Workflows

Since its inception, Pipedream workflows have been code first and developer friendly. Each workflow includes 2 gigabytes of storage at absolutely no cost within the /tmp directory.

In addition to writing Node.js or Python code to connect APIs together and transform data, you can also download and manipulate files with the flexibility of code and the entire PyPI and NPM package ecosystems.

But now within Pipedream you can use File Stores to store arbitrary files that are accessible not only between workflow executions but across your entire Pipedream project.

What are File Stores?

Files Stores are simple cloud storage for your Projects to upload, download and retrieve files.

All workflows within your Project will be able to access the files within the File Store. Unlike files stored in the workflow's /tmp directory, they are long lived between workflow executions.

You can store any type of file within a File Store and without any size limit. And you don't have to worry about losing them between workflow executions.

Your files stored within the File Store are:

• Stored within permanent long lived cloud storage
• Accessible to all workflows within your Pipedream project
• Accessible outside Pipedream with short lived URLs
• Managed programmatically in Node.js or within the Pipedream Dashboard

Getting to know the $.files Node.js helper

$.files is the helper within your workflows Node.js code steps to interact with your File Store.

It has built-in functions to list, upload, download, delete files in your File Store.

Let's go over a practical use case of resizing images. First we'll upload the original image to the File Store, then download it to the workflow, resize it and upload the newly resized copy.

Follow the guide below, or copy this workflow directly to your Pipedream account by opening this link:

Resizing Image Workflow

Uploading files to the File Store

First, we'll need to add a file to our File Store. Let's use the Pipedream logo available at this URL: https://pdrm.co/logo

It's 400x400 pixels, and let's convert into a 50x50 thumbnail in a Pipedream workflow.

You can upload this image a File Store with just two lines of code:

export default defineComponent({
  async run({ steps, $ }) {
	// create a new file in the File Store by opening it a path:
    const file = $.files.open('pipedream.jpg')
    
    // Upload a file to the File Store by a URL    
    await file.fromUrl('https://pdrm.co/logo')
  },
})

This example creates a new file stored at pipedream.jpg from this logo hosted at https://pdrm.co/logo.

And you can see the file downloaded within the Pipedream dashboard too:

Downloading a File Store file to your workflow

This same $.files helper offers tooling to easily download your files from File Stores to the /tmp directory for in-code processing.

Given pipedream.jpg is now uploaded to our File Store, it's trivial to download it to the current workflow's /tmp directory:

export default defineComponent({
  async run({ steps, $ }) {
    // Open the file stored in the File Store
    const file = $.files.open('pipedream.jpg')
    // Download the file from the File Store to the local /tmp/ directory
	await file.toFile('/tmp/pipedream.jpg')
  },
})

There are also helpers to work with streams to optimize on memory usage while downloading large files.

Resizing the logo

Now that the logo has been downloaded to /tmp/pipedream.jpg we can use any NPM or PyPI package to modify it.

For example, here's a Python code step that leverages the fantastic Pillow package to resize our newly downloaded pipedream.jpg down to 50x50 pixels:

# pipedream add-package Pillow
from PIL import Image

def handler(pd: "pipedream"):
    image = Image.open("/tmp/pipedream.jpg")
    resized = image.resize((50, 50))
    resized.save("/tmp/resized.jpg", "JPEG")

Storing the resized file into the File Store

Now we can upload this new resized.jpg in /tmp to the File Store using $.files:

export default defineComponent({
  async run({ steps, $ }) {
    // Open the File Store file to be populated
    const file = await $.files.open('resized.jpg')
    
    // Upload from the local /tmp/ directory to this new File Store file
    await file.fromFile('/tmp/resized.jpg')
  },
})

Volia, now both our original pipedream.jpg image and the resized.jpg are safely tucked away in the File Store:

The local version of the file in /tmp/pipedream.jpg and /tmp/resized.jpg are subject to deletion after the workflow goes cold. But now our stored file in the File Store will remain in cloud storage indefinitely.

What's the difference between /tmp and File Stores?

The /tmp directory within your workflow's filesystem is available for both reading and writing, and it's very useful for tasks like resizing images, downloading and uploading files to and from APIs or over FTP.

However, files in /tmp aren't accessible by URL and they're subject to be deleted between your workflow's executions as your workflow spins up and down workers to meet demand.

Files in /tmp are also subject the limits of the overall storage available in your workflows.

Join us on Tuesday Dec 5th for a live demo

We'll be giving a live demonstration of how to leverage File Stores with A.I. APIs tand more on Tuesday December 5.

Sign up here: Live File Stores Workflow Building Demo

Learn more and join our 4,000+ community of developers:

• 📚 Read the Docs
• 🤝 Join the Community
• ⚡️ Create a free Pipedream account