This topic was automatically generated from Slack. You can find the original thread here.
Using python, is there a way I can import my own library and model file?
Yes, you can use Python in Pipedream workflows and import your own library and model file. To do this, you’ll need to use the “Run Python Code” action within a workflow. Here’s how: 1. Search for the Python app from the step menu. 2. Select the “Run Python Code” action. 3. Connect your Python account. 4. Write your Python code, including the import statements for your library and model file. 5. At the bottom of the step, click “Test” to run the action and test the configuration. Please note that you might need to upload your library and model file to a storage service (e.g., S3, Google Cloud Storage) and then download them within your Python code before importing. For more information on using Python in Pipedream, visit https://pipedream.com/apps/python.
this idea of downloading before importing, can this be from github? Can you give an example code.
Yes, you can download a file from GitHub before importing it in your Python code within a Pipedream workflow. Here’s an example using the “Run Python Code” action: 1. Search for the Python app from the step menu. 2. Select the “Run Python Code” action. 3. Connect your Python account. 4. Write the following Python code in the provided editor: import requests # Replace the URL with the raw URL of the file you want to download from GitHub url = "https://raw.githubusercontent.com/user/repo/main/your_file.py" response = requests.get(url) # Save the content to a local file with open("/tmp/your_file.py", "w") as f: f.write(response.text) # Import the downloaded file import sys sys.path.insert(0, "/tmp") import your_file # Use the functions or classes from your_file as needed 5. At the bottom of the step, click “Test” to run the action and test the configuration. This example downloads a Python file from GitHub, saves it to a temporary location, and then imports it. Make sure to replace the url variable with the raw URL of the file you want to download from GitHub.
Hi , you can publish your Python files over PyPI can import them as normal modules.
Or pickle them and store them on private storage like Github or S3.
Hi Pierce, I don’t want the files public so PyPi is out as an option, is that right? Getting them from Github would be ideal
Correct, if you need these modules/data to be private, then uploading them to some private storage option should help.
The data just needs to be serializable/pickle-able so you can use it within your Python code steps after you retrieve it
OK - I need to learn and test this a bit.
Any reason I couldn’t use Attachments for the workflow and put the file/pickle there?
Thank you for the guidance, I appreciate it
You certainly could, but if the model data changes over time by other systems, it would be easier to retrieve the latest model from a storage that’s read/write programmatically. Whereas if you use attachments it would need to be updated manually with every change.
Agree, that is why I like the idea of github. Just looking at the options. I’ll jump into some testing and working this out. Thank you.
Sure thing 
You might be able to use GitHub’s Get Repository Content action to download your module. Although I’m surprised it doesn’t download to the /tmp directory. 
We’re planning to do something similar in the future, so please let us know how it goes! 
Thank you I’ll take a look. I suspect I can use this to download to a temp folder that the rest of the workflow has access to.
Marco Roy’s suggestion works well.
I used the Get Repository Content block to point to a folder in a specific repository. The output of that block is a list of files/folders and various URLs. The download URL is a one time use URL. I then have a python block have there with the following code to download the files. I’ve not done the import yet. I wanted to share this code to complete this solution.
github_import_files = pd.steps["get_repository_content"]["$return_value"]
for import_file in github_import_files:
print(import_file)
print(import_file["download_url"])
response = requests.get(import_file["download_url"])
if response.status_code == 200:
# Save the content to a local file
path = "/tmp/" + import_file["name"]
with open(path, "wb") as f:
f.write(response.content)
print("File downloaded successfully.")
else:
print(f"Failed to download file, status code: {response.status_code}")
And here is the next step, doing the import. One gotcha is that there are imports on this imported file. So I need to determine how to handle that. Those imports though are standard python libraries I believe.
github_import_files = pd.steps["get_repository_content"]["$return_value"]
sys.path.insert(0, "/tmp")
for import_file in github_import_files:
file_name = import_file['name']
response = requests.get(import_file["download_url"])
if response.status_code == 200:
# Save the content to a local file
path = "/tmp/" + import_file["name"]
with open(path, "wb") as f:
f.write(response.content)
print(f"{file_name} downloaded successfully.")
if file_name.endswith('.py'):
module_name = file_name.rstrip('.py')
imported_modules[module_name] = importlib.import_module(module_name)
# You can now use 'module' to access functions/classes within the imported module
else:
print(f"{file_name} is not a Python file and was not imported.")
else:
print(f"{file_name} failed to download, status code: {response.status_code}.")
: Is there a prebuilt action to download a file from a url, and store it in the /tmp directory?
There isn’t a specific pre-built action to download a file from a URL and store it in the /tmp directory. However, you can achieve this using a Python code step in your Pipedream workflow. Here’s an example of how you can download a file from a URL and save it to the /tmp directory: import requests url = "https://example.com/your-file" response = requests.get(url) if response.status_code == 200: # Save the content to a local file in /tmp directory path = "/tmp/your-file" with open(path, "wb") as f: f.write(response.content) print("File downloaded successfully.") else: print(f"Failed to download file, status code: {response.status_code}") Replace https://example.com/your-file with the URL of the file you want to download. This code will download the file and save it to the /tmp directory.
~: You shouldn’t need to download the file again in the last step, since the /tmp directory is preserved between steps.~
Ah, nvm, I get it. :man-facepalming: It’s the same step, but you just added more code to it.