How run puppeteer script in random time one per day.
Hi @fbox3tv33,
To run a Puppeteer script once per day at a random time, you can use the Schedule trigger in a Pipedream workflow. First, create a new workflow and search for the Schedule trigger. Then, choose the Cron Expression scheduling option and use a cron expression that runs the job once per day. For example, the expression 0 H * * * will run the job every day at a random hour H (0-23).
Next, add a Puppeteer script to your workflow using a code step. You can use the following example as a starting point:
import { puppeteer } from "@pipedream/browsers";
export default defineComponent({
async run({ steps, $ }) {
const browser = await puppeteer.browser();
const page = await browser.newPage();
await page.goto("https://pipedream.com/");
const title = await page.title();
const content = await page.content();
await browser.close();
return { title, content };
},
});
This script will run once per day at a random hour, as specified by the Schedule trigger.
I’m a bot powered by Pipedream and GPT-4. I’m still learning, so please double check my answers and code! Learn how to build your own.
I can’t set
0 H * * *
because “Save and continue” button is inactive, unable to be pressed
The Pipedream cron scheduler probably does not support that format. Here’s what you can do instead:
- Schedule a workflow at the beginning of the day (midnight).
- Calculate a random number of minutes between
0(right now) and1440(24 hours), or whatever minimum & maximum works for your use case. - Use the delay action to delay the rest of the workflow by that many minutes.
- ???
- Profits!!1!
Note: After the delay, HTTP calls could be made to trigger other workflows based on that random time.
Thanks for your help. Before reading your advice this is exactly what the @Pi bot suggested to me.
I corrected the code and am currently testing. (Tomorrow at midnight 1 run automatically)
I had problems calculating the delay because I forgot to add the delay to the current time, and then the problem of setting the correct time for my time zone. We’ll see if the code works.
For me as a total novice, it’s quite complicated. CRON should have an option to randomly calculate the time for the trigger.
If you use the delay action, you don’t need to worry about timezone or the current time.
You can simply set it to delay a certain number of minutes from now.
There is even a random number action, so you could use that to feed a random interval into the delay action.
No code required!
I don’t think it’s that simple. Or I’m doing something wrong.
When I simply set the delay it calculated in the output the digits e.g. 62100000 corresponding to a random time, BUT with the date January 1, 1970, instead of the current date Mon November 6, 2023 18:15:52 i.e. 1699290952295. Without the code corrections by time zone it also calculates me wrong.
Thanks.
Interesting solution.
If my way:
doesn’t work:
I’ll try your way.
Or I will make changes to the code.
Your way is a bit simpler.
I’ve tried both my way and your way and both don’t work.
With my way, it looks like I set the workflow_delay wrong and the puppeteer code doesn’t run. (is still in Pause mode)
Your way pops up a TIMEOUT error, probably because the randomly calculated workflow_delay (286 minutes) exceeds the workflow time. (I set it to 120 seconds in the settings of my workflow)
I have no idea how to set my code to run once periodically once a day at a random time.
Delays in the workflow shouldn’t be counted towards the timeout. We can see that the delay completed successfully, but it’s the puppeteer step that timed out.
Last resort, what you could do is for this delay workflow to trigger your Puppeteer workflow via an HTTP trigger. That way, the Puppeteer workflow would have a clean start with no delay, and you would know 100% where the timeout comes from.
As a newcomer, I’m starting to get lost.
When I use a simple trigger and puppeter code everything works. When I add workflow_delay then the timeout error suddenly pops up, but when executing the puppeteer code- to me it makes no sense and I don’t know what the issue is.
To my newcomer eye, the workflow_delay time IS included in the time limit for the WHOLE workflow. On top I see:
It seems that 1 execution is counted in my case from trigger to workflow_delay (and takes 1 credit), then it tries to execute the puppeteer code and it already runs out of time.
Look at the compute time, that is where the timeout is coming from. The workflow delay should only be included in the total duration. Otherwise, this is something to take up with Pipedream support.
But if you split your Puppeteer code into a separate workflow (triggered by HTTP request after the delay), it should eliminate any doubt.
@mroy is exactly right. Use of $.flow.delay doesn’t count towards your workflow’s compute time limit.
Please see the explanation here: Delaying a workflow - Pipedream
Can you share your Puppeteer code? I suspect you have a function that’s waiting for the network activity to finish, or is waiting for the presence of an element, but the timeout for that selector is longer than your workflow’s execution time limit.
I enabled access to my workflow through “Give Pipedream Support Access.” There I have 3 different scripts using delay:
-random time v1
-random time v2
-random time v3
And none of them works properly.
On the other hand, the script trigger v1 OK (trigger+puppeter code) without using delay works OK.
Time limit for workflow I set to 120 seconds.
Have you tried console.log() throughout your Puppeteer Node.js code to see exactly when you hit the workflow timeout?
That will help you troubleshoot exactly when you’re hitting the workflow timeout limit.
Based on the screenshot you shared above, the workflow is hitting the 120 second timeout window. So I recommend trying to narrow down which selector in your Puppeteer code is causing the issue.
A simple trigger-puppeter connection works OK.
Inserting a delay (beyond the time limit in the workflow) no matter what form between trigger and puppeteer causes the problem.
It looks as if the inserted delay cannot exceed the set time limit for the entire workflow (i.e. 3000ms for free tier).
I wanted to make a trigger that runs once a day, which would run the puppeteer code at a random hour (between 12 AM and 12 PM). Hence the need for a long delay arose.
The whole problem arose from the lack of a random trigger built into pupeteer. If it existed there would be no problem.
Of course, the problem can be circumvented by not using a delay at all, but this is beyond my novice capabilities.