user-1
November 13, 2023, 5:01pm
1
This topic was automatically generated from Slack. You can find the original thread here .
Hello, first I would like to thank you for the availability of Puppeteer!
user-1
November 13, 2023, 5:01pm
2
Hi Marcelo, glad to hear you’ve found our Puppeteer native support.
A couple of follow questions:
Have you increased your workflow’s execution timeout settings? By default it’s 30 seconds per execution, so it’ll need to be increased in this case.
Can you share the code that’s attempting to locate this element? I just want to double check that you’re passing in a timeout amount.
user-1
November 13, 2023, 5:01pm
5
Thanks, unfortunately this seems like it may be a bug within this particular version of Puppeteer that we bundle in @pipedream/browsers (puppeteer v19.8.0)
opened 11:08AM - 28 Mar 23 UTC
closed 06:29AM - 04 Apr 23 UTC
bug
confirmed
### Bug expectation
Running the _gotTo_ or _waitForSelector_ methods with the… timeout option or with a default page timeout that is set to a value higher than 30000 will timeout after 30 seconds regardless of the timeout setting.
Expected behavior is to consider the timeout option or the default page timeout value.
### Minimal, reproducible example
Issue does not reproduced with the previous version 19.7.5 and with v19.8.0 it can be reproduce with the following example code:
package.json:
```
{
"name": "test-puppeteer-upgrade",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"dependencies": {
"express": "^4.18.2",
"puppeteer": "19.8.0",
"sleep-promise": "^9.1.0"
}
}
```
index.html
```
<!DOCTYPE html>
<html>
<body>
<h1 id="title" style="display: none">Test</h1>
</body>
<script>
setTimeout(() => {
document.getElementById('title').removeAttribute('style')
}, 35000)
</script>
</html>
```
index.js
```
const puppeteer = require('puppeteer');
const express = require('express');
const path = require('path');
const sleep = require('sleep-promise');
const app = express();
const port = process.env.PORT || 8080;
app.get('/', async function (req, res) {
await sleep(35000)
res.sendFile(path.join(__dirname, '/index.html'));
});
app.listen(port);
console.log('Server started at http://localhost:' + port);
(async () => {
const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
await page.setDefaultTimeout(180000)
try {
await page.goto(`http://localhost:${port}`);
await page.waitForSelector('#title', { visible: true/*, timeout: 180000*/ })
} catch (err) {
console.error(err);
}
await browser.close();
})();
```
This example will get this navigation timeout error after 30 seconds:
```
Page.navigate timed out.
```
When removing the sleep before responding with the HTML page, you can test for the waitForSelector issue that will timeout after 30 seconds:
```
Waiting for selector `#title` failed: Runtime.callFunctionOn timed out.
```
### Error string
Page.navigate timed out.
### Puppeteer configuration
```TypeScript
{
headless: false
}
```
### Puppeteer version
19.8.0
### Node version
18.12.1
### Package manager
npm
### Package manager version
8.19.2
### Operating system
macOS
The values you’re sharing match the original posters in the GH issue.
Have you noticed that the timeout occurs at exactly 30 seconds into the step’s execution?
user-1
November 13, 2023, 5:01pm
6
Also, did you try passing a timeout definition to the waitForXPath() method? Meaning await page.waitForXPath(selector7, { timeout: 120000 })
user-1
November 13, 2023, 5:01pm
7
yes I tried, it didn’t work
user-1
November 13, 2023, 5:01pm
8
user-1
November 13, 2023, 5:01pm
9
Nice find, that could potentially fix it. Yes, you can pass arbitrary options to the puppeteer.launch() command through the Pipedream @pipedream/browsers package:
Could you try passing { protocolTimeout: 240000 } to puppeteer.browser() like puppeteer.browser({ protocolTimeout: 240000 })?
user-1
November 13, 2023, 5:01pm
10
it works ! thanks a lot Pierce !
user-1
November 13, 2023, 5:01pm
11
Awesome
user-1
November 13, 2023, 5:01pm
12
Just an update for anyone finding this thread in the future, we have set the default protocol timeout on the constructor for both pre-built Puppeteer actions as well as the import { puppeteer } from '@pipedream/browsers' package.
You can still override it if you need a longer timeout time, but the default is 240000:
PipedreamHQ:master ← PipedreamHQ:pierce/DRE-528
opened 02:51PM - 06 Nov 23 UTC
## WHAT
<!--
copilot:summary
-->
### <samp>🤖 Generated by Copilot at 03e101e</s… amp>
Updated the `puppeteer` app and its actions to version `1.0.0` to support the latest changes in the `browsers` package. Added the `protocolTimeout` option to the `launch` method of both the `puppeteer` app and the `browsers` package to increase the browser connection timeout and prevent errors on slow pages.
<!--
copilot:poem
-->
### <samp>🤖 Generated by Copilot at 03e101e</samp>
> _`puppeteer` changes_
> _major version and timeout_
> _winter of slow pages_
## WHY
## HOW
<!--
copilot:walkthrough
-->
### <samp>🤖 Generated by Copilot at 03e101e</samp>
* Bump the version of the `puppeteer` package and its actions to `1.0.0` to reflect major changes in the app and the `browsers` package ([link](https://github.com/PipedreamHQ/pipedream/pull/8827/files?diff=unified&w=0#diff-ae1be2b9632b8fb3ff8f7c46bcdb32bb15bade47e542d49d9356295a9171611cL7-R7), [link](https://github.com/PipedreamHQ/pipedream/pull/8827/files?diff=unified&w=0#diff-9293050daaf377120514fd4576dde6625c9388148426be810fea8ce5869a2f14L7-R7), [link](https://github.com/PipedreamHQ/pipedream/pull/8827/files?diff=unified&w=0#diff-edeac74108f5fcfd4372f1d6d2fb8269ae22e83111c7f3da9bdcd486b97c7d96L9-R9), [link](https://github.com/PipedreamHQ/pipedream/pull/8827/files?diff=unified&w=0#diff-7c651bdabaca3622a521ba2fc19123bd7ddef4eeba614a5b98e6e07c11deb94aL10-R10), [link](https://github.com/PipedreamHQ/pipedream/pull/8827/files?diff=unified&w=0#diff-9a902b32400f187d4ef83890980c121f01c8d62616231a7a846b2e55d32156c6L3-R3))
* Add the `protocolTimeout` option to the `launch` method of the `puppeteer` app and the `browsers` package to increase the timeout for the browser connection from 30 to 240 seconds ([link](https://github.com/PipedreamHQ/pipedream/pull/8827/files?diff=unified&w=0#diff-dc255ad27cc4062520ec7c2eec5816c3045003fbef1ce0171b478e8cd6c831abR33), [link](https://github.com/PipedreamHQ/pipedream/pull/8827/files?diff=unified&w=0#diff-3c6823116f58743e9268d54102e61eec6cb2557f1cfae151e2777ff293829102R26))
* Bump the version of the `browsers` package to `1.0.2` to reflect the addition of the `protocolTimeout` option ([link](https://github.com/PipedreamHQ/pipedream/pull/8827/files?diff=unified&w=0#diff-8e01e5bf2a1332f403d9036f727d9e3db6ee8406b4b3430630160b086f59d0dfL3-R3))
glad to hear that. We’ll make an update to the package to increase this timeout by default