How Can I Exclude Keys from a Set of Key/Value Pairs?

Help
1

This topic was automatically generated from Slack. You can find the original thread here.

I’m spending waaaay too long on what should be a simple idea I think.

I have a set of key/value pairs returned from a data store at steps.get_all_records.$return_value in format {“7593892020470”:22269763,“6643999572154”:18409612,“6625223114938”:18400579,“6740633125050”:19472524,“7844368777462”:24589619…
and I want to exclude the keys returned from another data store in steps.Completed_Metadata.$return_value format [“6740454604986”,“7869234643190”,“6754827174074”,“7486943166710”,“6909540008122”,“6618209386682”,“7869228843254”…
and be left with the remaining key/value pairs

how do I do that?

2

If this is a stupid way to do it when the sets are going to include approx 15,000 keys each then let me know!

3

Hello , I think Pipedream Datastore API doesn’t provide the efficient way to support your usecase.

I think you can add a Node.js code step with 2 data store prop, get all records from both, then implement the logic to filter the Datastore A using the keys from Data Store B. This might work, but it will potentially cause memory limit exceeded error on your workflow if your record grow.

To have an efficient version, I would suggest you use relational database (maybe Supabase can provide?) and use SQL statement to fetch/modify your data