How to download zip files with varying URL formats and access requirements?

Help
1

This topic was automatically generated from Slack. You can find the original thread here.

I have a couple of issues regarding downloading a zip file to tmp. I have it working when the URL is http://www.favoriterec.com/pusher/preco/NG03-SalesKit.zip The link is publicly accessible and the filename is in the URL. Download/unzip to tmp works great. I’m having trouble with a couple of other use cases.

  1. The URL does not include the file name (and/or) redirects to a download link. Such as Redirecting to - (no zip in the URL, redirects to a dynamic/encoded/encrytped/limited access URL Kudos Download via another redirect URL I’m not fast enough to copy). None of the URLs feature “.zip” in the URL and the downloaded zip file is eventually “Psychosynthesis - Greg Foat.zip”

  2. The URL requires me to be logged in order to access the download link. User account | Rush Hour Distribution (sends me to a login page when I’m ingonito, downloads the file directly when I’m logged into my brower - again no file.zip in the URL and the file is ultimately downloaded as snippets_seaside_highway_20-04-2023_0921.zip)

What exactly do I need to look for to address either or both of these issues? Pupeteer seems to be the generally recommended option for #2 but it’s not available on pipedream (yet?) I’m not exactly experienced with how web downloads are supposed to work generally to know exactly what the issue is. Any tips?

2

, for both of the points, I think we will need sufficient coding experience to implement. Specifically, for 1), we will need to have the user to provide the file name, and the code need to be able to follow the redirect.

For the 2), I think they used some kind of cookies or local storage to store your credentials, the site automatically check those credentials for you to download. The code in this case need to have your credentials

The tips I have for now is you can try to learn more about Node.js and how browser download process works, then try to replicate it in Node.js, then ultimately port the code to Pipedream code action.

Or you can contact with the site, to have them send you the email with the files so that you don’t need to spend more effort needed

3

I’m doing it manually right now. Downloading the zip files to my machine and then uploading to AWS and using new AWS file as the trigger. Obviously I’d like to automate as much as possible. I’ll think about it. Thanks.