How can I sign in to access files from a non-API website and bypass the login wall using Pipedream?

Help
1

This topic was automatically generated from Slack. You can find the original thread here.

Is there a way to sign in to a website to grab files that only authorized users have access to? It’s not an API. it’s just a standard website and the files are offered by the wholesaler to retailers so we can showcase the products. But we need to sign in. When I try to grab those files via the API it’s obviously hitting a login wall. How do I navigate that? GPT is telling me to use Pupeteer but from the links I found we dont have access to that on pipedream? Puppeteer / Playwright support · Issue #209 · PipedreamHQ/pipedream · GitHub

2

Hi , unfortunately pupeteer is not compatible with Pipedream yet.

I think you can use other lib such as Cherio for crawling, note that this might breaks as the site changed their site; or you can contact with the site that provides the file for them to give them to you by batch?

3

see if you can reverse engineer and reproduce the network calls for login and then for querying the files to download using the login token