Is there an easy way for me to handle a situation where I need to fire off a refresh on another REST API (i have this set)- however I need to query a “refreshes” API and wait for a status of “Completed” before moving on- is there a way to handle this in one workflow? I see a continue if condition is met- but this doesn’t sit in hold status continually pinging the api- and I’ve seen other documentation about pausing it, but this doesn’t fit my needs as this could be anywhere from 5-30 mins at times depending on the size of the refresh we’re triggering. My Less than ideal situation would be to set up some webhook trigger to look for those refreshes - but I’d like the process to be dependent on the steps previous as they can potentially have different post response tasks.
Unfortunately not - basically this workflow is triggered from a webhook after a DBT run is complete- I then force a powerBI Partial refresh. I would like to send a slack alert when the refresh is finished - but it could be anywhere from 5-30 mins depending, and the only way to know if it’s done is to poll the refreshes API and check. I get a refresh ID back from the post so can use that to check. What would be ideal is like a filter loop - Continue only if condition is met - otherwise return to previous step (with a potential application for a 5 minute wait period).
I was thinking about this approach- however, if I poll every 5 mins, and the refresh complete’s and I get the “success” flag- how do I stop it polling and slacking me every 5 mins after?
To avoid double notifications you can delete the refresh ID from the data store after it has been notified.
Another option - but I’m not exactly sure how this works - is to pause/suspend the workflow after the refresh ID has been notified. Please look at the docs here. Not sure if the cancellation URL stops only the workflow run or if it pauses the workflow.
@dylan do you know if there is a way to programmatically pause/resume workflows?
Money! Thank @dylburger ! @andrew Also didn’t think about removing it from the store- good thinking too! Thanks so much for your help on this!! #pipedreamfanboy (what my peers are calling me cause I won’t shut up about it!!!)
hey @dylburger – if I need to pass a bearer token through with this http get request - and i would have been received in a previous step, how would I handle it… something like :
Hey @dylburger. SO I got the code to work - however it never returns and actual value for data. It keeps pausing indefinitely despite the status being completed. I’ve validated this with tests, and I can see the value is Completed being returned- I’m guessing I’m not comparing the correct value? Any ideas, since It’s a bit hard to trouble shoot the node script (if I could find a way for it to return the values of Data, instead of “rerunning” it could help debug! (compared to the other easy UI processes!! )
I’m including the code I’m running.
import axios from ‘axios’
export default defineComponent({
async run({ $ }) {
const MAX_RETRIES = 120
// 60 seconds
const DELAY = 1000 * 60
const { run } = $.context
// $.context.run.runs starts at 1 and increments when the step is rerun
if (run.runs === 1) {
// $.flow.rerun(delay, context (discussed below), max retries)
$.flow.rerun(DELAY, null, MAX_RETRIES)
}
else if (run.runs === MAX_RETRIES + 1) {
throw new Error(“Max retries exceeded”)
}
else {
// Poll external API for status
const { data } = await axios({
method: “GET”,
url: “$top=1”,
headers: {
Authorization: ‘Bearer {{steps.Bearer_Token.$return_value.access_token}}’
}
})
// If we’re done, continue with the rest of the workflow
if (value[0].status = “Completed”) return data
// Else retry later
$.flow.rerun(DELAY, null, MAX_RETRIES)
}
},
})
This is the failure I got this morning- I’m guessing this may be a timeout issue?
AxiosError
Request failed with status code 403
at null.settle (file:///pipedream/dist/code/87a4709fd7d03cb4d7f403680f15e0c1fa0ae610e494a5fa2428dc81ccb13dd3/node_modules/.pnpm/axios@1.4.0/node_modules/axios/lib/core/settle.js:19:12)
at IncomingMessage.handleStreamEnd (file:///pipedream/dist/code/87a4709fd7d03cb4d7f403680f15e0c1fa0ae610e494a5fa2428dc81ccb13dd3/node_modules/.pnpm/axios@1.4.0/node_modules/axios/lib/adapters/http.js:570:11)
at IncomingMessage.emit (node:events:525:35)
at null.endReadableNT (node:internal/streams/readable:1359:12)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
@mike.melanson I see one issue with your code and wanted to offer a few other general troubleshooting suggestions that may help.
It looks like your code uses a single equals sign for comparison here:
if (value[0].status = “Completed”) return data
Which will try to assign the value of Completed instead of comparing the two. Try:
if (value[0].status === "Completed") return data
The axios error you’re seeing is different — it looks like the service you’re trying to hit is issuing a 403 Forbidden error (usually something to do with bad credentials / lack of permissions to work with the resource you’re requesting). axios doesn’t return great data by default, so I’d recommend using our version, which will give you better debugging data.
In general, these retries / big blocks of conditional logic can be difficult to troubleshoot, so I like to sprinkle console.log statements and use $.export to return data at specific parts of the code, so I see what’s happening along the way. It sounds like you may already be doing that, but I hope this all helps.
Much appreciated @dylburger - I’ll take a peak at pipedream version! I was trying to do console.log statements - but wasn’t exporting them back out (haven’t really dealt with node.js before – I was trying to mimic the hello world Node.js function you guys have- but didn’t see a export! If you have an example of a straight fw debug script- that would be super helpful!) so I’ll try and stack those in there too! And yeah sorry- I tried one two and 3 equals- Lol- neither made a diff! I guess I just pasted the version that was trying 1 !!
Well- just to close the book on this for now- I ended up utilizing the data store approach- and building a checker every 5 mins to see if a new entry has been added to the store. The good thing though- I ended up building a slack integration yelling “Pipedream Refresh” so I can also add manual “API checkers” for when we do manual refreshes. So worked out!
