This topic was automatically generated from Slack. You can find the original thread here.
need to compare two large json files and find a - b and b - a. what are different ways we can do it in pipedream
To compare two large JSON files and find the differences between them (a - b and b - a) in Pipedream, you can use custom code steps within a workflow. Here’s a step-by-step guide on how to achieve this using Node.js: 1. Trigger: Start with a trigger that initiates your workflow. This could be an HTTP trigger if you’re sending the JSON files via HTTP requests, or any other trigger that suits your use case. 2. Add Custom Code Step for Comparison: - Use a Node.js code step to compare the two JSON objects. - You’ll need to parse the JSON files into JavaScript objects if they’re not already in that format. Here’s an example of how you might write the code step: export default defineComponent({ async run({ steps, $ }) { // Assuming `jsonA` and `jsonB` are your JSON objects // This could be replaced with steps that fetch these JSON objects dynamically const jsonA = {/** JSON object A **/}; const jsonB = {/** JSON object B **/}; function compareJSON(obj1, obj2) { const result = { inAnotB: [], inBnotA: [], }; for (let key in obj1) { if (!(key in obj2)) { result.inAnotB.push(key); } } for (let key in obj2) { if (!(key in obj1)) { result.inBnotA.push(key); } } return result; } const comparisonResult = compareJSON(jsonA, jsonB); // Returning the comparison results return comparisonResult; }, }) This code compares two JSON objects at a high level, looking for keys that are present in one but not the other. It’s a basic example and might need to be adapted depending on the structure of your JSON files and the depth of comparison required. For deeply nested objects, you might need a more sophisticated approach to recursively compare each level. 3. Inspect the Results: - After running the workflow, inspect the results in the Pipedream workflow inspector. You’ll see the keys that are present in jsonA but not in jsonB and vice versa. 4. Further Steps: - Depending on your needs, you can add further steps to process the comparison results
If the JSON is a collection of arrays, then lodash’s difference is pretty handy: Lodash Documentation
But if it’s a complex object, you might need to rely on Object.entries(obj1).reduce to compare the two with some custom logic