Shutting down the SQL Service
We've made the hard decision to shut down the Pipedream SQL Service on February 10th, 2022. We wanted to take the time to explain details about the deprecation timeline and why we made this decision. We'll also suggest services you can use in its place, all of which have built-in Pipedream integrations.
When will this happen?
The SQL Service will remain fully-operational for the next 90 days, until February 10th, 2022.
On February 10th, you will no longer be able to create new tables, or write records to existing tables, using $send.sql().
You will still be able to run SQL queries via the UI and the API for the following 30 days, between February 10th and March 10th, 2022. You can also export a full copy of your tables by running:
SELECT * FROM your_table_name;Since new records expire after 30 days automatically, all data in tables will be removed from the SQL Service by March 10th, 2022. On that day, you'll no longer be able to query or download data, and the service will be retired. We'll remove access to the UI and API, and references in docs, at that time.
Why?
Pipedream makes the hard parts of integrations easy: you don't need to set up infrastructure to host your workflow; you don't have to deal with the details of OAuth, concurrency, etc. The SQL Service aligned with this vision. As we developed the first versions of Pipedream and built our own workflows, we found that we had a lot of questions on event data: how often is this field NULL; how many events arrived within the last hour; what's the count of events by browser? We weren't yet sure if we needed to store this data in our data warehouse (Snowflake), and it would have taken time to build that ETL. So we needed a way to quickly create a table, stream JSON to it, and start to analyze it. That's how the SQL Service was born.
Many of y'all used the SQL Service in the same way. But as you gave us feedback, it was clear that the service didn't solve your core needs. You frequently needed to update and delete records like you could in a relational database, and we didn't support that. Improving the service in that direction would have required investment that took effort away from the core product: workflows. Many other services provide a better experience for storing and querying data. We prefer to focus on what we're good at (workflows) and develop best-in-class integrations with these services, instead.
What other services should I use?
It depends on your use case. We'll review a few of these cases and suggest appropriate alternatives below.
Are you storing data that you need to use within a specific workflow or component? Consider $checkpoint or $.service.db.
Are you just logging data, like incoming webhooks, that you need to search later? Check out:
Are you storing large amounts of unstructured data (like JSON from incoming webhooks) that you need to run basic queries on? Consider MongoDB / Mongo Atlas (example workflow).
Do you need to insert, update, and delete records, and ask complex questions about that data using SQL? Consider relational databases like:
If these options don't work for you, feel free to reach out at any time and we'll help you determine the best service to suit your needs.
Questions?
We understand this change has an impact on your existing workflows, and we're happy to help ease the transition in any way. Please reach out at https://pipedream.com/support if you have any questions or need support migrating to a new service.