Error communicating with data store

Bugs
1

The Data Store was working fine while I was coding my workflow.
As soon as I went “production” mode, meaning, 500 rows fetched and then updated per workflow execution, I started seeing errors.


Error
Error communicating with data store

DETAILS
    at null.__node_internal_captureLargerStackTrace (internal/errors.js:412:5)
    at null.__node_internal_exceptionWithHostPort (internal/errors.js:590:12)
    at null.internalConnect (net.js:934:16)
    at null.defaultTriggerAsyncIdScope (internal/async_hooks.js:452:18)
    at GetAddrInfoReqWrap.emitLookup (net.js:1077:9)
    at GetAddrInfoReqWrap.onlookup (dns.js:73:8)

My workflow id is p_RRCDOD7

2

I split the step in 2 and optimised its code.
My code was not traversing only 500 items as I thought, but all the records (around 2.9k).

I have added a filter to limit the amount of records to something less aggressive (dozens instead of thousands), and the error is now gone.

3

Well… I fixed the initial problem, but now I have another use case that brought me back to this issue.
I would like to reset a field inside all the objects I store (e.g. setting outdated: true on all of them) but I could not find an efficient way to do so.

image
image1842×1496 227 KB

4

Hi @zvictor

Data Stores are still a Preview feature, and we’re working to improve the guard rails so you t won’t hit these limitations.

It’s intended to be a convenient key => value data store, not a full fledged database. If you find yourself needing to iterate over a large amount of records to perform updates, you may be exceeding the use case for Data Stores.

We have pre-built actions for popular databases such as Postgres, MySQL, MongoDb, and DynamoDB. Any of these can be connected to your Pipedream workflows and have conditional updating actions that might better fit your purpose.