user-1
November 24, 2023, 9:28pm
21
Or maybe to set a max, right?
user-1
November 24, 2023, 9:28pm
22
The default code only works with up to 25 contacts and then fails. I had to customise it. I opened a PR but honestly have not had the time to make the requested edits, Node isn’t my forte.
PipedreamHQ:master ← MCKLtech:patch-3
opened 03:50PM - 13 Oct 23 UTC
Slack Discussion: https://pipedream-users.slack.com/archives/CMZG4EBJ9/p16972109… 29916869
https://api.hubapi.com/crm/v3/objects/contacts/batch/read can only accept 50 inputs per call. If there are more than 50 contacts, it will fail with the following message:
"The maximum number of inputs supported in a batch request for property histories is 50. Please reduce the number of items and try again"
The code loops over the Contact IDs and limits the inputs to less than 50.
## WHAT
<!--
copilot:summary
-->
### <samp>🤖 Generated by Copilot at 3741e30</samp>
Improved the performance and reliability of the `new-contact-property-change` source by processing the updated contacts in batches. Added a `batchSize` variable and a for loop to `new-contact-property-change.mjs` to implement the batching logic.
<!--
copilot:poem
-->
### <samp>🤖 Generated by Copilot at 3741e30</samp>
> _`batchSize` splits contacts_
> _HubSpot API is spared_
> _`after` marks time_
## WHY
Fixes bug whereby the workflow will fail if there are more than 50 Contacts in the BATCH API call
## HOW
<!--
copilot:walkthrough
-->
### <samp>🤖 Generated by Copilot at 3741e30</samp>
* Reduce the number of API calls to HubSpot by processing updated contacts in batches of 45 ([link](https://github.com/PipedreamHQ/pipedream/pull/8462/files?diff=unified&w=0#diff-a744670a02ee30bc9376f1b9c828607195e5a580fbbc0b76f4a2519caef21550L110-R133))
user-1
November 24, 2023, 9:28pm
23
Ah ok. Sounds like you just need to update the versions on the action itself and the app package.json. But I can help with that if you’d like
user-1
November 24, 2023, 9:28pm
24
Actually, it’s more complicated that than I thought. It seems getPaginatedItems is the root cause, as it calls hubspot.searchCRM iteratively, so the sleep needs to be there. It’s part of the common functions, so it’s impact would be much wider ranging.
this.emitEvent(item);
const ts = this.getTs(item);
if (ts > maxTs) {
maxTs = ts;
this._setAfter(ts);
}
}
}
}
},
async getPaginatedItems(resourceFn, params) {
const items = [];
do {
const {
results, paging,
} = await resourceFn(params);
items.push(...results);
if (paging) {
params.after = paging.next.after;
} else {
delete params.after;
user-1
November 24, 2023, 9:28pm
25
Got it. I think think we could sneak an option delay param to that method
user-1
November 24, 2023, 9:28pm
26
I know this bug is niche, but I could probably see that being a good idea for customers at scale.
user-1
November 24, 2023, 9:28pm
27
Assuming of course I’m right
user-1
November 24, 2023, 9:28pm
28
Ok, I can pretty much confirm this issue was caused by the pagination function. I wrote a new one as a work around, and added a ‘hubSleep’ function, and it’s working again. Thanks all.
user-1
November 24, 2023, 9:28pm
29
Nice! Glad you were able to find a workaround
user-1
November 24, 2023, 9:28pm
30
I assume hubSleep’s default is 0? So that way it’s “speedy” by default.
user-1
November 24, 2023, 9:28pm
31
At the moment just pass in a ms value. It’s a hacky fix for now.
user-1
November 24, 2023, 9:28pm
32
Hey whatever works, especially if it’s your own private action
