This topic was automatically generated from Slack. You can find the original thread here.
How can I use a Node.js step to delete a single record from the data store?
To delete a single record from the data store using a Node.js step, you can use the @pipedream/platform package to interact with the Pipedream KeyValue store. Here’s an example of how to delete a record with a specific key: import { KeyValueStore } from "@pipedream/platform"; export default defineComponent({ props: { key: { type: "string", label: "Key", description: "The key of the record you want to delete", }, }, async run({ steps, $ }) { const kv = new KeyValueStore($); await kv.delete(this.key); console.log(`Deleted record with key: ${this.key}`); }, }); This code defines a component with a key prop, which you’ll provide when you run the step. It then uses the KeyValueStore class from the @pipedream/platform package to delete the record with the specified key.
I use the built in Node.js for the data store, I just need to know the function for deleting it.
Example of what I use in order to get all values:
await this.myDataStore.get(“orders”)
To delete a single record from the data store using the built-in Node.js, you can use the delete method. Here’s an example: export default defineComponent({ props: { data: { type: "data_store" }, key: { type: "string", label: "Key" }, }, async run({ steps, $ }) { // Delete the record with the specified key await this.data.delete(this.key); console.log(`Deleted record with key: ${this.key}`); }, }); In this example, you’ll provide the key of the record you want to delete as a prop. The delete method is then used to delete the record with the specified key.
I have just one key called “orders”, inside I have an array.
My specific question is if there’s an easy way to delete just one collection from the array without getting all the values and use a filter because if have like 10K values then it will very slow to return it
I understand that you want to delete a specific item from the “orders” array without fetching the entire array. Unfortunately, the KeyValue store in Pipedream doesn’t support modifying the array directly. You’ll need to fetch the array, remove the item, and then save the updated array back to the KeyValue store. Here’s an example of how to do this: ```
export default defineComponent({ props: { data: { type: “data_store” }, indexToRemove: { type: “integer”, label: “Index to remove” }, }, async run({ steps, $ }) { // Fetch the orders array const orders = await this.data.get(“orders”); // Remove the item at the specified index orders.splice(this.indexToRemove, 1); // Save the updated array back to the KeyValue store await this.data.set(“orders”, orders); console.log(Removed item at index: ${this.indexToRemove}); }, });
This is what I do now, what happens if I have like 10K values? it won’t slow down the process?
Fetching and updating a large array with 10K values from the KeyValue store can indeed slow down the process, as it requires fetching the entire array, modifying it, and then saving it back. Unfortunately, the current KeyValue store in Pipedream doesn’t support modifying the array directly. If performance becomes an issue, you might want to consider using an external database that supports more efficient operations for your use case, such as Firebase, MongoDB, or PostgreSQL. You can then interact with the external database within your Pipedream workflow using their respective APIs or SDKs.
Yep that’s what I thought
I understand that fetching and updating a large array in the KeyValue store can be a concern for performance. If you find that it becomes a bottleneck, consider using an external database that better suits your needs. If you have any other questions or need assistance, feel free to ask.
Could you create a separate data store to use as your array?
Each key-value in the store would be like an item in the array.
But obviously, that would be slower to operate on when you want to manipulate the entire list.
Yes, I think the best way is to stay with MongoDB