This topic was automatically generated from Slack. You can find the original thread here.
Do Pipedream Data Stores have support for transactions or atomic operations? I have a key that I’m using as a counter that I’d like to increment but I’m worried that I’ll introduce a race condition. I don’t want to limit my worker count to 1, as that’s not really a solution to ensuring the operation is atomic. Surely there’s a way to read and update a record atomically in one workflow step.
Not that I’m aware of. I’ll share this with the team, but from my knowledge the only way to ensure atomic read/writes is limiting to 1 worker
This seems like a pretty useful thing to support out of the box.
Also something that you might want to warn people about. I’d imagine a lot of people will run into this as they don’t even know what a race condition is.
Would be cool to be able to assume that a step in the workflow editor that uses the data store is atomic.
By that logic, I’d expect a step for “transaction” and maybe some useful steps like increment or decrement… but I’m sure there are a ton of ways this could be done.
Good call, I’ll make sure we add that to the docs
Totally hear you on this
My workaround is that I’m changing to using append to record for my accumulator.
Hopefully that will be atomic
You’re storing an array for the counter and considering the last index as the current counter value?
I’m appending to an array every time an action happens, then at a later time counting the elements
Or is the accumulator the sum of all values?
Ah gotcha
My goal is to count the number of times a workflow runs in a recurring time period, which would also be a nice to have directly from pipedream metadata
might be possible, but didn’t see it anywhere obvious
Well actually either way if two workers get the array from the data store at the same time, a race condition will occur
Yeah the number is not available out of the box, but you could send a notification to an external service, like a database and count from there
Maybe my mental model is wrong, I was thinking of Data Stores like key value, where an “append” operation doesn’t have a race condition because it doesn’t require a read then write.
Under the hood, the append reads first and then writes