Why am I Getting '[Errno 30] Read-only File System' Error While Using Pyppeteer in Python?

Help
21

it says “The requested module ‘@pipedream/browsers’ does not provide an export named ‘launch’”

22

can you correct the import from the code sample above? @pipedream/browsers exports a puppeteer module, not a launch module

23

I apologize for the confusion earlier. You’re correct, the @pipedream/browsers package exports a puppeteer object, not a launch function. Here’s the corrected code: import puppeteer from "@pipedream/browsers"; export default defineComponent({ async run({ steps, $ }) { const websites = ['https://www.google.com', 'https://www.facebook.com']; const browser = await puppeteer.launch(); const promises = websites.map(async (url) => { const page = await browser.newPage(); await page.goto(url); return await page.content(); }); const htmlContents = await Promise.all(promises); await browser.close(); return htmlContents; }, }); In this corrected version, we’re importing puppeteer from @pipedream/browsers, and then using puppeteer.launch() to launch a new browser instance. The rest of the code remains the same.

24

That’s still not quite correct, it’s a named export:

import { puppeteer } from '@pipedream/browsers'
25

looks like it’s working, but when I actually add all my URLs to that array (there’s 17 of them), it returns this error: Navigation timeout of 30000 ms exceeded

26

Can you share your component code? Are you waiting for any specific selectors?

27

**import** { puppeteer } **from** '@pipedream/browsers';

export **default** defineComponent({
**async** run({ steps, $ }) {
**var** websites = ['https://chutzlaaretz.com','https://walcottradio.com','https://userway.org','https://eyeclick.com','https://rivkazaide.co.il','https://claimjusticerefund.com','https://totalmerchantresources.com','https://verbit.ai','https://verbit.co','https://go.verbit.ai','https://vitac.com','https://diplomatline.co.il','https://kmslh.com','https://aman.co.il','https://amandigital.co.il','https://rasner.co.il','https://dataloop.ai'];
**const** browser = **await** puppeteer.launch();

`**const** promises = websites.map(**async** (url) => {`
  `**const** page = **await** browser.newPage();`
  `**await** page.goto(url);`
  **`return`** `**await** page.content();`
`});`

`**const** htmlContents = **await** Promise.all(promises);`
`**await** browser.close();`

`**return** htmlContents;`

},
});

28

I wonder if one website in particular is slow to load or is not responding timely. Perhaps try adding a console.log(url) within the loop to identify which once fails

29

Or wrapping the scraping code with a try…catch to catch the exception and log which website is the culprit

30

But again, splitting the workflow into two helps diagnose these issues too, since an event is dedicated to one record

31

check it out

32

I added {timeout: 0} to the goto function

33

**await** page.goto(url, {timeout: 0});

34

now it works

image.png
image.png1104×676 66.2 KB

35

Nice find! I’m not entirely sure what that option does exactly, I’m not seeing a clear definition on the docs for that method:

36
37

Thanks for sharing!

By default, the maximum navigation timeout is 30 seconds. If for some reason, your target url requires more time to load (which seems unlikely), you can specify a timeout: 0 option.

So one of the URLs is taking longer than 30 seconds to load. Which probably means it’s down.