Why Does the "Wait for Path Existence" Step Stall in Deployed Workflow Despite Successful Testing?

Help
1

This topic was automatically generated from Slack. You can find the original thread here.

I am using a parallel structure to conditionally execute two possible paths. At the conclusion of the parallel structure the output is under steps.parallel.$return_value.python

I later use a step to wait until the file that is defined by that returned value exists. When testing the workflow this all works fine but when deployed it stalls at this “wait for path existence” step. Any ideas?

For example, the path output from a recent run is:
/tmp/export_Open PO-PR by Dept 1203.csv

2

To handle the output from a parallel structure in Pipedream, you can access the results of each branch using the steps.parallel.$return_value object. If you have a parallel step named parallel, and you want to access the output from a specific branch, you would typically reference it using the branch name or identifier. Here’s a general approach to accessing the output: 1. Identify the Branch: Determine the name or identifier of the branch whose output you need to access. In your case, it seems like you are referring to a branch named python. 2. Access the Output: Use the steps.parallel.$return_value object to access the output of the specific branch. For example, if the branch is named python, you would access it like this: const pythonOutput = steps.parallel.$return_value.python; 3. Use the Output: You can now use the pythonOutput variable in subsequent steps of your workflow to perform further actions based on the data returned from the parallel branch. If you need more specific guidance or have additional questions about handling outputs from parallel branches, feel free to ask!

3

Here’s the block for the “wait for path existence”

from pipedream.script_helpers import (steps, export)
import os
import time

while not os.path.exists(steps["parallel"]["$return_value"]["python"]):
    time.sleep(1)
4

any more feedback?

5

It seems like you’re asking about feedback, but your question is not very clear. Could you please provide more context or specify what kind of feedback you’re looking for? If you’re referring to feedback on a specific feature or component, please let me know so I can assist you better.

6

aC looks like the python code structure you’re using is outdated. Seems like it was generated by some AI. Can you create a new Python step and use that structure instead?

7

Hi can you give me a little more to go off of? You’re talking about the “Wait for Path Existence” code or something in the parallel structure?

8

I mean the python code should look something like this (note the handler function):

def handler(pd: "pipedream"):
    # Reference data from previous steps
    print(pd.steps["trigger"]["context"]["id"])
    # Return data for use in future steps
    return {"foo": {"test": True}}