How to resolve "Magic Comment" error when installing a PyPI package with a different import name?

Help
1

This topic was automatically generated from Slack. You can find the original thread here.

Trying to perform a “Magic Comment” to install a PyPI package that differs from it’s import name. I am trying to use pinecone with a langchain agent which is why I’m not using the direct pinecone integration/workflow step. I seem to be following the syntax, but i am getting an error still. Any suggestions?

# pipedream add-package pinecone-client
import pinecone

Error
Command failed: python3 -m pip install --target=/tmp/pdg/dist/python -r requirements.txt --upgrade WARNING: The directory ‘/home/sbx_user1051/.cache/pip’ or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo’s -H flag. ERROR: Could not find a version that satisfies the requirement pinecone==0.1.0 (from versions: none) ERROR: No matching distribution found for pinecone==0.1.0 WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available. You should consider upgrading via the ‘/var/lang/bin/python3 -m pip install --upgrade pip’ command.

2

I’m getting the same error Command failed: python3 -m pip install --target=/tmp/__pdg__/dist/python -r requirements.txt --upgrade WARNING: The directory '/home/sbx_user1051/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled.

3

The read-only OS makes working with these python packages quite difficult

4

@U02SX7ETBFB has there been any discussion about expanding the w permissions on the local disk? perhaps running the worker in a container?

5

Yes, the Python support is still in Beta and we’re working out the kinks on the importing system. We know it’s not in a fully stable state yet.

There has not been any discussion around opening up more of the permissions on the filesystem, for security purposes I don’t think that’s likely.

But bring-your-own-container has been discussed a few times, it’s not on the near term roadmap but I think it’s a possibility in the longer term just because it opens up the door to bringing your own binaries and more flexibility.

6

would you consider this a known issue, or should we create a bug report?

7

(Sorry for the @ spam, not sure if you’d see the message otherwise?)

8

The /tmp dir has full write permissions, but if you look at the error message, it’d trying to write to /home/sbx… which doesn’t exist, hence the error

9

Oh, I assumed sbx_user1051 was the Pipedream user

10

Does the package let you specify where to place /tmp files?

11

Nah I’m actually not sure where that’s coming from. It might be the user the Amazon Linux runtime automatically creates

12

That’s what I’m looking for right now, but this seems to occur during install

13

# pipedream add-package pinecone-client

14

I don’t think the worker got to the handler function

15

The error shows at the top of the builder

image.png
image.png1057×468 23 KB

16

The placement of the error is misleading, and it’s a known issue unfortunately.

The error is happening at workflow build time, when the workflow is being compiled with your PyPI dependencies.

This is another good candidate for an issue on our public GH: Sign in to GitHub · GitHub

17

Got it, I’ll make an issue

18
19

I’ve seen some other packages have gotten “hotfixes” basically in some of the closed issues on GitHub. Any chance that might happen with pinecone? Deciding on whether I continue down the pipedream path for my application or if I need to break out into lambda :face_vomiting:

20

yeah I’d love to support Pinecone, I can look into this more — just assigned myself the issue. I can’t promise an ETA but should have time to look into it next week and see what the blocker is