I’m new to Pipedream and currently using the Free version of the product. I’ve successfully created a JavaScript step that calls an API endpoint and downloads a JSON document that’s over 5MB in size. This JSON is then inserted into a PostgreSQL RDS table using a JSONB field.
I came across this thread and a related thread from August 2021 discussing the same issue, but I haven’t yet figured out how to overcome the limitations around large payloads.
Ideally, I’d prefer not to go down the route of storing the JSON as a file (e.g., storing the data as a file to the ‘/tmp’ directory), as that would turn my current one-step process into something more complex and involved.
I’d really appreciate any guidance, workarounds, or best practices the community can share.
You should visit Pricing - Pipedream for the latest pricing, but I’ll do my best to answer.
To handle large payloads in Pipedream, you can use the built-in support for sending large HTTP payloads. By default, Pipedream limits the size of incoming HTTP payloads, but you can bypass this limit by using one of the following methods:
Query String Method: Add the pipedream_upload_body=1 query string to your HTTP request URL.
HTTP Header Method: Include the x-pd-upload-body: 1 HTTP header in your request.
Here’s an example of how you can use these methods with curl:
When you use these methods, Pipedream saves the raw payload data in a file whose URL you can reference in the variable steps.trigger.event.body.raw_body_url. You can then download the contents of this data using the Send HTTP Request action or by saving the data as a file to the /tmp directory.
For more detailed information, please refer to the Pipedream documentation on sending large payloads. If you have further questions or need additional support, please visit Support - Pipedream.
I’m a bot powered by Pipedream and GPT-4. I’m still learning, so please double-check my answers and code! Learn how to build your own.
We’re encountering persistent step timeouts when attempting to load a 6.07MB JSON payload from an API into our PostgreSQL RDS instance using a Pipedream workflow on the free tier.
Use Case
Our workflow involves ingesting large JSON payloads (5MB+) from public APIs and loading them into a PostgreSQL database for downstream processing. A reliable, automated solution is critical—manual workarounds are not feasible.
Issue
Despite multiple implementation attempts, we consistently hit the 30-second step timeout limit, with our most recent run failing after only 36 rows were inserted into the database.
Attempts Made
We’ve tested several approaches, all resulting in timeouts:
/tmp Directory Write + Single Insert
Saved the full JSON to /tmp/content.json, then attempted to insert it into a staging.raw_payload table (raw_data as TEXT or JSONB).
Outcome: Step timed out or hit memory/payload limits due to the ~6MB size.
Chunked Insert + Append via UPDATE
Inserted the first 50,000-byte chunk and appended subsequent chunks via UPDATE statements (SET raw_data = COALESCE(raw_data, '') || $1).
With TEXT, we reached 1.75MB (35 updates) before timeout.
Increasing chunk size and splitting into multiple steps did not resolve the issue.
Chunked Inserts into Intermediary Table + Final Merge
Used a staging.raw_payload_chunk table with raw_data_chunk as VARCHAR(90000), planning to later assemble the JSON via a CTE.
Outcome: Timeout after inserting 36 chunks (~3.24MB).
Observations
The 30-second step limit and resource constraints appear to fundamentally restrict large-payload workflows.
Optimizing chunk sizes and step structure did not yield a viable solution.
Request
Could someone confirm whether this use case—reliably ingesting and loading 5MB+ JSON payloads into a database—is supported within the current architecture of Pipedream (especially on the free tier)? If so, what is the recommended pattern for doing this efficiently?