Hello.
What is the recommended way to store a bunch of files that persist across executions?
Let’s imagine that I want to buffer a set of file uploads and only process them when I have 5 already on the FS (or any other place). Is this possible? If not, any possible workaround?
@danielo515 how large are these files? You may have seen that you can store any JSON-serializable data in workflow state, up to 64KB in total size. Technically, if the file data can be serialized, you could store the data like so:
// Read the contents of $checkpoint from the last execution, or initialize it
const checkpoint = $checkpoint || {}
// Read file, save contents to checkpoint
checkpoint['filename'] = fileContents
// Write data back to $checkpoint
$checkpoint = checkpoint
If your files are large, this solution may not work. Do you use any cloud storage providers, like Amazon S3 or Google Cloud Storage? We support those integrations (and more), and that’s the best way to store data across executions. On each execution, you could list the files in remote storage and, if you have 5 files to process, execute the rest of the code. Otherwise, you can save a new file to the store.
Would either of those options work for you?
Yes, I can setup a s3 or google cloud bucket. The thing is that I didn’t saw documentation about how to “list” contents on those, just how to add new files. Does that mean that I should (and can) use the SDK of those services directly on a node script?
That’s right! If we don’t have a pre-built action for a specific operation, you can always add a new Node.js code step, require the SDK for your service, and list the files that way.
Are you comfortable with Node.js, and would that work for you?
Yes, I’m absolutely familiar with nodejs. It is nice to see that I can install any package. I’ll check the documentation to install packages.
1 Like