Requesting a possible functionality that would allow for a trigger that could cycle on a set interval (5 mins for instance) during the existence of a key in a data store.
As it stands- I have a process that kicks off a refresh in PBI and pushes a value into a data store- I have another process that runs every 5 mins to check the REST API for a complete or failure status provided this key is there (if there is a condition at the beginning of the workflow to check for this value- other wise end) , once the failure or completed status is captured I delete the key. This runs EVERY 5 mins during working hours (for both failure and success), and consumes a lot of credits. What I’m proposing would be a trigger that would notice the key get added and basically run every 5 mins while the key is there- and then once the key is gone- the trigger wouldn’t fire until the next time I add a key cause of a refresh!
But for your use case, what I would suggest instead is to use $.flow.rerun().
After inserting the key into the data store, you can kickstart a workflow that will pause & rerun every 5 minutes, and complete once your PBI refresh is finished. Based on this, you might not even really need the data store at all, actually.
It will still use 1 credit every 5 minutes while it runs, but at least it won’t need to run all day.
And what would be even better is if PBI had webhooks for completed or failed refreshes (not sure if it does). That way, there would be no need to poll the API: you could just have a workflow waiting to receive those callbacks instead.
Yeah- I’ve tried using that before but was unable to get it to work! I don’t have a ton of js.node experience! My forte is more data modelling and Data architecture! I’m just also dabbling in api integrations to help streamline our notifications and keep our ETL’s tight as they can be! (I posted this back in July)
@mike.melanson: Ah, well that should still solve your problem! And the polling for those triggers is internal to the Pipedream components, and thus not using any credits.
Hey @michael — So I’m trying to set up something to use the powerBI refresh features above- but I don’t see my dataset- is there a setting in the dataset I need to change to allow pipedream to see it? Or is there a way to enter a “custom value” with the dataset ID ? I’m logged in as our admin account so can’t see any reason why it wouldn’t show
— I know the DS ID , so forcing it to be a custom ID (kinda like in slack when it can’t see the channel etc) — but also not 100% what kinda perms would allow some of these to be visible and not others
