Why is the Key+Value Not Being Recorded Consistently in my Data Store?

Bugs
1

This topic was automatically generated from Slack. You can find the original thread here.

I’m experiencing a weird bug with data stores, where the key+value consistently does not get recorded in the store at all.

2

Fortunately, I’ve found a workaround: by adding a pause at the end of the step, it seems like it gives some more time for the to actually get saved. :sweat_smile: :man-shrugging:

3

Here’s the code:

export default defineComponent({
  props: {
    dbt_runs: {
      label: 'Data Store',
      type: 'data_store'
    }
  },
  async run({ steps, $ }) {
    this.dbt_runs.set(
      'resume_run_' + steps.run_job.$return_value.id,
      // Timeout after 30 minutes
      $.flow.suspend(1000 ** 60 ** 30).resume_url
    );

    await new Promise(resolve => setTimeout(resolve, 1000));
  }
})
4

When I run this step without the await 1 second at the end, the value isn’t saved in the data store at all. :man-shrugging:

5

Maybe there’s some kind of conflict or race condition between the two services? :face_with_monocle: (data store & flow suspend)

6

maybe you need to add await to dbt_runs.set?

    await this.dbt_runs.set(
      'resume_run_' + steps.run_job.$return_value.id,
      // Timeout after 30 minutes
      $.flow.suspend(1000 ** 60 ** 30).resume_url
    );
7

That makes too much sense.

8

Thank you!

9

I’ll try it now.

10

All datastore operation are asynchronous, so you might need to await whenever you use them

Ref: Using Data Stores

11

Might be good to call that out explicitly in the documentation! (near the top)

Otherwise, it’s easy to miss the await in the code samples (like I did :man-facepalming:)

12

Thanks for the suggestion , actually you can contribute to Pipedream doc yourself whenever you see fits. For example, you can checkout and contribute to the doc source here: https://github.com/PipedreamHQ/pipedream/tree/master/docs/docs/data-stores

13

Nice!

Will do! :ok_hand:

14

: I can’t find the documentation for using the data stores in Node.js code steps on GitHub. :face_with_monocle:

There only seems to be the generic documentation about data stores in general.

15

check it here: https://github.com/PipedreamHQ/pipedream/blob/master/docs/docs/code/nodejs/using-data-stores/README.md

16

Ah! It was in the code directory :man-facepalming: