This topic was automatically generated from Slack. You can find the original thread here.
It seems Pipedream recommends using Data Stores for caching of our Pipedream APIs, calls to other APIs etc yes? Does anyone have a link to a good overview of using Data Stores for caching - and using conditional branching logic to handle the “cache exists” cases?
Is if(condition === true){ $run.component("some_step_name")...} how to do branching within a workflow? Are there any docs out there for conditional step branching?
To run a code step only if a certain key exists in a data store, you can use the following example code. This example uses Node.js, as it’s the most common language for writing Pipedream components: import { axios } from "@pipedream/platform"; export default defineComponent({ props: { data: { type: "data_store" }, }, async run({ steps, $ }) { // Check if a specific key exists in your Data Store const keyExists = await this.data.has('your_key_name'); if (keyExists) { // Your code to run when the key exists console.log('The key exists in the data store.'); // Add any additional code you want to run here } else { // Optional: code to run when the key does not exist console.log('The key does not exist in the data store.'); } }, }) Replace 'your_key_name' with the actual key you’re checking for in the data store. If the key exists, the code within the if block will execute. If you have specific actions you want to perform when the key exists, you can place that code inside the if block.
thanks andrew. will have to see how i might hack this. i suspect it will be super hacky without conditional workflow branching.
(i know one step that interfaces a data source does not expose the code so i cant add any conditional logic to it - (unless i convert it to a direct api connection which sounds like a pain) - and possibly i can reorder tings to break the workflow before it… but sounds like a nightmare to tell you the truth. ha. )
So are there any docs out there for how to do caching on Piepdream? I’m not exactly sure how you’d do caching if onditional branching is so messy? Break workflows into more (messy) subparts maybe? Would be great to see how smarter folks than me tackle caching!
An aside: just curious: did you all base Pipedream on some kind of simplified, linear DAG workflow engine under the covers that makes more complex workflows super hard to implement?
the bulk of the work is in improving our UI to handle this use case and not in the backend. But it’s our #1 priority and we’re hoping to have something out to you soon!
Can you share more about the use case / what you’re intending to cache? I typically implement complex caching / early exit behavior in code steps, so I’m curious if that would help in your case (until we support conditional logic in the UI)
Hi Dylan, re: UI/UX, ah, yeah that makes sense. A lengthy art of finesse, edge cases and us fickle users? Ha.
Yeah, I’m trying to setup cache for Pipedream for the first time, so I’m curious about best ways to do it… with the purpose of mostly trying to limit slow AI calls if I can. And am 100% for making it work however it needs to be done. (Would love to check out how others on PD do it.)
Use case: I figured workflow might be:
• (Limited domain) search term → Pipedream webhook trigger
• If recent keyword not in Data Store, go grab some RAG data from Airtable (other DBs in future), then run some transformations via LangChain + OpenAI.
• Add those final results to the Data Store for that keyword search term to bypass the whole Airtable + slow AI process in the future.
• And whether Data Store had the keyword or not, deliver those results as the final HTTP response.
And so using the Airtable (or future db) step node = an all-config-UI-no-code PD component… though I assume I could go through the trouble of using the Airtable (or other DB) api directly via a standard PD node.js node, and wrap that with the Data Store conditional yeah?
I’d definitely recommend implementing this in code now. I can see exactly why you’d want to use the no-code primitives and do all of this with pre-built actions, and we should support this use case once we ship the conditional logic. But for now, you should be able to implement a code step:
Add a data store and Airtable app prop to the step and click Refresh at the top of the step, adding your data store and connecting your Airtable account.
Do a get to retrieve data by keyword from the data store. In the absence of a record, run the RAG logic.
)