This topic was automatically generated from Slack. You can find the original thread here.
Hello there!
Referring to my previous thread.
I’ve tested in-box PD data stores, and it seems to me that they cannot process more than 994 requests at a time.
I cannot fill in the single database with more than 994 records inside 1 step, neither can retrieve all records from the database with more than 994 records at a time.
You might try to take a look.
I apologize for the confusion earlier. To address your concern about Pipedream data stores, it seems you’re experiencing a limitation when processing more than 994 requests at a time. I’ll need more information about your use case and the specific components you’re using to better understand the issue and provide a solution. Can you please provide more details about your workflow and the steps you’re using?
I am generating a JSON object that way.
export default defineComponent({
async run({ steps, $ }) {
let myArr = new Array();
for(let i = 0; i != 3000; ++i)
{
let string = new Object();
string[i] = Math.floor(Math.random() * 10000);
myArr.push(string);
}
let json = JSON.stringify(myArr, null, 2);
return json;
},
})
Then, I am trying to push it into the data store with “add_update_multiple_records”.
Then, I get this error.
Error
Error communicating with data store
DETAILS
at null.__node_internal_captureLargerStackTrace (node:internal/errors:490:5)
at null.__node_internal_exceptionWithHostPort (node:internal/errors:668:12)
at null.internalConnect (node:net:1066:16)
at null.defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18)
at GetAddrInfoReqWrap.emitLookup (node:net:1323:9)
at GetAddrInfoReqWrap.onlookup (node:dns:109:8)
I understand that you’re trying to store a large JSON object with 3000 records in a Pipedream data store using the “add_update_multiple_records” action, but you’re encountering an error. To help you further, I need more information about the data store component you’re using and the specific steps in your workflow. Can you please provide more details about the data store component and the steps you’re using to store the data?
That’s it. I am just trying to pass a large amount of data to a data store.
I see that you’re trying to store a large JSON object with 3000 records in a Pipedream data store, and you’re encountering an error. One possible solution is to split the data into smaller chunks and store them separately in the data store. Then, when retrieving the data, you can fetch the chunks and combine them back into the original JSON object. Here’s an example of how you can split the JSON object into smaller chunks and store them using the add_update_multiple_records action: export default defineComponent({ async run({ steps, $ }) { let myArr = new Array(); for (let i = 0; i != 3000; ++i) { let string = new Object(); string[i] = Math.floor(Math.random() * 10000); myArr.push(string); } const chunkSize = 500; // Adjust this value based on the maximum size you want to store in a single record const chunks = []; for (let i = 0; i < myArr.length; i += chunkSize) { chunks.push(myArr.slice(i, i + chunkSize)); } // Use the add_update_multiple_records action to store each chunk separately for (const [index, chunk] of chunks.entries()) { await steps.add_update_multiple_records.execute({ records: JSON.stringify(chunk), record_key: `chunk_${index}`, }); } return `Stored ${chunks.length} chunks in the data store.`; }, }); When retrieving the data, you can fetch the chunks and combine them back into the original JSON object: ```
export default defineComponent({ async run({ steps, $ }) { const chunkCount = 6; // Replace this with the actual number of chunks you’ve stored let combinedData = []; for (let i = 0; i < chunkCount; i++) { const chunk = await steps.get_record.execute({ record_key: chunk_${i}, }); combinedData = combinedData.concat(JSON.parse(chunk.data)); } return combinedData; }, });
You are a very helpful bot =*
Thank you for your kind words! I’m here to help. If you have any more questions or need assistance with Pipedream, feel free to ask.