Update on Pipedream not picking up missed tasks after schedule change to 6 hours?

Help
61

disco_core

62

try this, please

63 
use disco_core;
SELECT * FROM TABLE(INFORMATION_SCHEMA.TASK_HISTORY()) 
where NAME = 'my_info_schema_task6'
order by COMPLETED_TIME DESC;
64

Still nothing

65

can we make a huddle?

66

Yes, sure

67

Cassiano caught me up on the current state. Do you have anyone else on your team who’s familiar with the Snowflake configuration of the task history table? I believe you can modify the retention on that table (perhaps on a per-role basis), and I also can’t reproduce the issue on our instance. The Snowflake team directed us to use that table as a reference for failed tasks, so you may need to adjust the retention on that table or modify the configuration in some way to get it working

68

But this does not look like an issue with the Pipedream source, so I’d recommend going that route — Snowflake can probably assist better than we can

69

OK. Let me give it a try

70

I did find that running this query:
SELECT *
FROM snowflake.account_usage.task_history
where NAME = ‘MY_INFO_SCHEMA_TASK6’
order by COMPLETED_TIME DESC;

71

Gave me all failed tasks

72

Here is a reference TASK_HISTORY | Snowflake Documentation

73

is that query subtly different than the one we’re constructing? It looks here like task history retention may not be the core issue if we’re able to see all failed tasks

74

the new sql sahred by is selecting for a table, we are using a function, maybe that function is using this table, need to investigate, the doc is the same

75

Ok. Should I stand by while you investigate?

76

Hey . I will comeback with some news today

77

Sounds good. Thank you

78

Hey . I can see all my executed tasks here using your query, I will update the action and adapt it to use the new query. Hopefully, this may fix your problem. I will let you know when the PR is done. But probably tomorrow. Thanks for you patience

79

One thing, this query just works using accountadminrole. So, when using it in Pipedream, make sure to select this role or some role with these permisions

80

Sounds good. Thanks