Trigger based Existence of data in Data Store

Requesting a possible functionality that would allow for a trigger that could cycle on a set interval (5 mins for instance) during the existence of a key in a data store.

As it stands- I have a process that kicks off a refresh in PBI and pushes a value into a data store- I have another process that runs every 5 mins to check the REST API for a complete or failure status provided this key is there (if there is a condition at the beginning of the workflow to check for this value- other wise end) , once the failure or completed status is captured I delete the key. This runs EVERY 5 mins during working hours (for both failure and success), and consumes a lot of credits. What I’m proposing would be a trigger that would notice the key get added and basically run every 5 mins while the key is there- and then once the key is gone- the trigger wouldn’t fire until the next time I add a key cause of a refresh!


Here’s a GitHub issue for this feature: [TRIGGER] Pipedream Data Store - New Key in Data Store, Updated key in Data Store.

1 Like

But for your use case, what I would suggest instead is to use $.flow.rerun().

After inserting the key into the data store, you can kickstart a workflow that will pause & rerun every 5 minutes, and complete once your PBI refresh is finished. Based on this, you might not even really need the data store at all, actually.

It will still use 1 credit every 5 minutes while it runs, but at least it won’t need to run all day.

And what would be even better is if PBI had webhooks for completed or failed refreshes (not sure if it does). That way, there would be no need to poll the API: you could just have a workflow waiting to receive those callbacks instead.

For example, Tableau supports webhooks.

1 Like

Yeah- I’ve tried using that before but was unable to get it to work! I don’t have a ton of js.node experience! My forte is more data modelling and Data architecture! I’m just also dabbling in api integrations to help streamline our notifications and keep our ETL’s tight as they can be! (I posted this back in July)

I think your dev team is exploring those functions and integrating them as we speak! Power BI · Issue #9247 · PipedreamHQ/pipedream · GitHub :slight_smile:

Those functions are mostly to send API calls to PBI, not to receive webhooks from PBI! I’m not sure what triggers PBI makes available.

P.S. I don’t work for Pipedream! :sweat_smile:

1 Like

These prebuilt triggers and actions are now complete! Have a look here:

Unfortunately, it doesn’t look like Power BI has any webhook functionality available in their APIs, hence why we set these triggers up as polling.

1 Like

@mike.melanson: Ah, well that should still solve your problem! And the polling for those triggers is internal to the Pipedream components, and thus not using any credits. :ok_hand:

1 Like

These are awesome… I think most of the ones I requested are here! Awesome!

1 Like

Hey @michael — So I’m trying to set up something to use the powerBI refresh features above- but I don’t see my dataset- is there a setting in the dataset I need to change to allow pipedream to see it? Or is there a way to enter a “custom value” with the dataset ID ? I’m logged in as our admin account so can’t see any reason why it wouldn’t show

Hi Mike,

Can you share a screenshot? Are you able to see any datasets, or none at all? This is what I can see, as an example:

I see a few - but not the main DS that we use to power all our reporting…

— I know the DS ID , so forcing it to be a custom ID (kinda like in slack when it can’t see the channel etc) — but also not 100% what kinda perms would allow some of these to be visible and not others

Just to make sure… have you clicked on “Load more” until all of them got loaded?

A few times… :wink:

1 Like

was getting a similar thing happen with the salesforce account update action, not listing all the columns or objects

hey @michael — Were you able to find anything related to the missing datasets?

@michael — I know my datset ID so perhaps we could just add an option for a custom ID ?

Hi @mike.melanson , I’m looking into this now. Will follow up shortly with regard to adding the ability to add a custom ID to the sources!

Hi Mike,

I’ve opened up a ticket to update the triggers to allow for a custom dataset id to be entered. You can follow along the progress here! [FEATURE] Microsoft Power BI - Triggers - Add option for custom id for dataset · Issue #9878 · PipedreamHQ/pipedream · GitHub

Thanks for raising the issue.

1 Like