Has Anyone Successfully Managed to Run It?

Bugs
1

This topic was automatically generated from Slack. You can find the original thread here.

Anyone managed to run it correctly ? thank you :slightly_smiling_face:

2

Hey , it seems like the library you are using tried to find a directory, but it is not available in Pipedream run time. In Pipedream, only the tmp directory is writable, please consult the doc here for more detail https://pipedream.com/docs/code/python/working-with-files

3

Hello that’s the issue :slightly_smiling_face: it’s not need any directory. And when I try it in any other Python environnement it works juste fine

4

The directory is needed when working with duckdb in memory database, but here I am using MotherDuck

5

, for Pipedream (which uses AWS Lambda), you need to specify /tmp dir as your working directory. Please consult your library API documentation on how to pass a custom path to your library.

6

I already tried passing /tmp path as a custom directory. Unfortunately it does not work ! But my code works in classic lambda function

7

I already tried passing /tmp path as a custom directory
May I ask for your code in this case?

8

sure, I tried many things:

9 


import os
import duckdb

def handler(pd: "pipedream"):
    # Set your MotherDuck token as an environment variable (replace 'your_token' with your actual token)
    motherduck_token = os.environ['motherduck_token']
    try:
        # Set the directory to /tmp
        home_directory = '/tmp'

        # Connect to MotherDuck using the service token
        con = duckdb.connect(f'md:?motherduck_token={motherduck_token}')

        # Set the home directory using the SET command
        con.execute(f"SET home_directory='{home_directory}'")

        # Specify the COPY command
        copy_command = """
        COPY loaddata.Temp_Days_fr
        FROM 's3://my_bucket/tempo_days/*.csv' 
        WITH (
            DELIMITER ',',
            FORMAT CSV,
            HEADER
        );
        """

        # Execute the COPY command
        con.execute(copy_command)

        print("Data loaded successfully!")
    except Exception as e:
        print(f"An error occurred: {e}")
10

Set the home directory using the SET command

    con.execute(f”SET home_directory=’{home_directory}’“)

I believe dynamically setting environment variable like this wouldn’t work. You’ll need to set Environment Variable in your workspace or project setting: Environment Variables - Pipedream

11

I tried this as well

import os
import duckdb

def handler(pd: "pipedream"):
    motherduck_token = os.environ['motherduck_token']
    try:
        home_directory = '/tmp'

        con = duckdb.connect(f'md:?motherduck_token={motherduck_token}&home_directory={home_directory}')


        # Specify the COPY command
        copy_command = """
        COPY loaddata.Temp_Days_fr
        FROM 's3://my_bucket/tempo_days/*.csv' 
        WITH (
            DELIMITER ',',
            FORMAT CSV,
            HEADER
        );
        """

        con.execute(copy_command)

        print("Data loaded successfully!")
    except Exception as e:
        print(f"An error occurred: {e}")
12

you mean to do it like this:

home_directory = os.environ[‘HOME’] ?

13

You’ll need to set Environment Variable in your workspace or project setting. Here’s how: Environment Variables - Pipedream

14

ok I will try it now

15

I don’ know if this what you meant:

import os
import duckdb

def handler(pd: "pipedream"):
    motherduck_token = os.environ['motherduck_token']
    try:
        home_directory = os.environ['home_directory'] 

                # Connect to MotherDuck using the service token
        con = duckdb.connect(f'md:?motherduck_token={motherduck_token}')

        # Set the home directory using the SET command
        con.execute(f"SET home_directory='{home_directory}'")

        # Specify the COPY command
        copy_command = """
        COPY loaddata.Temp_Days_fr
        FROM 's3://my_bucket/tempo_days/*.csv' 
        WITH (
            DELIMITER ',',
            FORMAT CSV,
            HEADER
        );
        """

        con.execute(copy_command)

        print("Data loaded successfully!")
    except Exception as e:
        print(f"An error occurred: {e}")
16

but here still receiving the same error. home_directory is an env variable in my project

17

Here’s what I tried:

  1. Set environment variable in my project (1st image)
  2. Refer it in the python code, in this case I printed it out (2nd image)

image.png
image.png2670×1878 200 KB

image.png
image.png2670×1878 257 KB

18

1.png
1.png2548×362 40.9 KB

2.png
2.png2942×1502 315 KB

3.png
3.png1320×806 66.6 KB

19

I tried google colab to double check and the requests work fine

20

Then I believe this is specific error to your library.

As you can see the environment has been set successfully on Pipedream.

I would encourage you to take a look at motherduck or duckdb lib that you’re using on how to pass custom directory (maybe without using environment variable)