This topic was automatically generated from Slack. You can find the original thread here.
Need help with a query. I’m trying to pull values from Data Store where keys are dates and pass combined contents to a step.
I want to pull the last 7 days. This is what get_record_keys is for, right? Is this a query or some custom code I need to write?
just to confirm, are you storing dates as a single array of values in your data store and just want to pull the last 7 elements of the array?
my data store looks like this. Btw if there’s a better way to store info from my daily runs, open to adjusting.
Also, do you always want to pull the date 7 days ago? Or are the dates going to have gaps / be longer than 7 days?
Context: I have a daily flow that stores a list of to-dos.
I now want a weekly flow that does a digest.
So I figured I’d store the results from daily runs and retrieve them in weekly.
Got it, that makes sense. And just to confirm - you want to pull both the dates and to dos (keys and values) for the last 7 days?
Actually just the values.
Cool thanks. Let me see if I can use our new Q&A bot Mario to help and show you how I’d do this
Oh yea… Mario! I wish I could just msg directly?
I’m storing daily to dos in a data store: the current date (YYYY-MM-DD) as the key, the to dos (a string) is the value. I’d like to retrieve the values from the last 7 days of data, i.e. the data from the keys corresponding to the last 7 days. Can you show me the Node.js code to return the last 7 days of values as an array?
Sure, here’s a Node.js code snippet that retrieves the values from the last 7 days of data in your data store: const { datastore } = require("@pipedream/platform"); async function getLast7DaysValues() { const today = new Date(); const values = []; for (let i = 0; i < 7; i++) { const date = new Date(today); date.setDate(today.getDate() - i); const key = date.toISOString().slice(0, 10); // Format date as YYYY-MM-DD const value = await datastore.get(key); if (value) { values.push(value); } } return values; } getLast7DaysValues().then((values) => { console.log("Last 7 days values:", values); }); This code defines an async function getLast7DaysValues that iterates through the last 7 days, formats the date as a key (YYYY-MM-DD), retrieves the value from the data store using that key, and pushes the value into an array if it exists. Finally, it logs the array of values from the last 7 days.
I thought Mario would do better on that one! Let me try once more
I’d like to store an array of values in a single key in a Pipedream data store. I’d like to keep appending data once a day to the array, storing the array with the new data. Can you show me the Node.js code to retrieve the array from the data store and then return only the last 7 elements of the array?
Sure, here’s a Node.js code snippet to retrieve the array from the data store and return only the last 7 elements of the array: const { datastore } = require("@pipedream/platform"); async function getLast7Elements() { // Replace 'your_key' with the key you used to store the array in the data store const storedArray = await datastore.get('your_key') ?? []; // Append new data to the array const newData = "your_new_data"; // Replace with the new data you want to append storedArray.push(newData); // Store the updated array back to the data store await datastore.set('your_key', storedArray); // Get the last 7 elements of the array const last7Elements = storedArray.slice(-7); return last7Elements; } getLast7Elements().then((last7Elements) => { console.log("Last 7 elements:", last7Elements); }); This code defines an async function getLast7Elements that retrieves the stored array from the data store using the specified key, appends new data to the array, stores the updated array back to the data store, and then returns the last 7 elements of the array. Finally, it logs the last 7 elements of the array.
Mario is new and this isn’t accurate. I was hoping to show you how I’d approach this: I’d recommend 1) saving the to dos to a single array of objects, where each object has a date / content field. Then 2) you can fetch the data at that single key and always just return the last 7 elements of that array for your weekly report. Does that make sense?
yup that makes sense too if pulling keys inside a range is less optimal.
should I be concerned about limitations of how much can be stored under a single key?
Typically no - you can store MBs of data just fine at a single key. But you can also truncate the data before writing it back to the data store if you no longer need it, e.g. only keep the last 90 days of elements of the array each time you process, writing the last 90 days to the data store
Nice! Thanks, will give this a try. My digital assistant bot is coming along thanks to these. I appreciate the help.
Have you ever worked with Supabase or another database before? When you want to run queries like this (e.g. get all the tasks matching this filter in the last 7 days), a relational database can be better than a key-value store, which is limited to key lookup operations.
I haven’t, but it’s something I’m willing to learn more about. A part of the magic in these GPT flows I’m building is having some access to history/memory.
