Loop through a Pipedream Data Store?

Help
1

Reviewing the Node.js for data stores, it seems like retrieving all the keys() is possible, but how would one loop through an entire data store?

I tried values() but that isn’t supported, nor is a function like all()

// To use previous step data, pass the `steps` object to the run() function
export default defineComponent({
  props: {
    groups: { type: "data_store" },
  },
  async run({ steps, $ }) {

    const groups = await this.groups.all();//This doesn't work but is there another method?
    
    for (const item of groups) {

      console.log(item);

    }
    return steps.trigger.event
  },
})
2

Hi @mckltech

What you could do is use the list of keys from your Data Store as the iterator:

// To use previous step data, pass the `steps` object to the run() function
export default defineComponent({
  props: {
    groups: { type: "data_store" },
  },
  async run({ steps, $ }) {

    const keys = await this.groups.keys();
    
    for await (const key of keys) {

        const value = await this.groups.get(key);
        console.log(`Retrieved ${value} associated with ${key}`);

    }

  },
})

But wouldn’t necessarily recommend this. You may run into performance issues and this isn’t very efficient use of memory or time in your workflows.

If you need to grab all records in a data store, that might be your sign to use a more robust database like MongoDB, Postgres, etc.

3

Thanks, Pierce. I see there is actually a method called ‘entries’, and this works too:

async run({ steps, $ }) {

    const groups = await this.groups.entries();

    for (const item of groups) {

      console.log(item[0]);
      console.log(item[1]);
  }

It’s not a very large data set so performance here isn’t critical. If we scale up, we’ll move to a SQL dB or similar.

1 Like
4

Sure thing, just wanted to share the disclaimer on Data Store limitations.

Best of luck with your project!