How to Properly Delete Files in the Tmp Folder During a Workflow with Three Steps?

Help
1

This topic was automatically generated from Slack. You can find the original thread here.

Hi there! I have a question regarding files in the tmp folder and how to delete them properly. I have a workflow with three steps

  1. Node.js: Download file from AWS S3 and store in tmp folder (using the tmp-promise package), return path of tmp file (for next step)
  2. GDrive: Upload tmp file (return value of step 1) to GDrive
  3. Node.js: Delete tmp file (using return value of step 1): return **await** fs.promises.unlink(steps.aws.$return_value);
    All works fine for a while, until at some point step 1 (download from AWS) fails with ENOSPC: no space left on device, write
    It seems like the deletion in step 3 doesn’t work as expected and the re-used container overflows eventually. I tried a different approaches:
    • simply deleting everything in /tmp (not a good idea, because there seems to be workflow code as well :grimacing: )
    • deleting all files that start like /tmp/temp-* which seems to be the naming pattern for tmp-promise file paths -> turns out, there are no files with that pattern :thinking_face:
    Can someone explain that to me? I expected the tmp folder to be available in all steps, since step 2 seems to be able to upload a tmp file from step 1. But for some reason, I can’t clean-up at the end.
2

Hi It’s complicated, but the tmp directory can persist files between runs. It’s not guaranteed that you’ll have a clean tmp directory between executions of your workflow.

For this reason, we recommend deleting files stored to your workflow immediately after you’ve used them.

Maybe perhaps trying to download the file to the same filepath each time so that the past file is overwritten instead of having to manage deleting files of different names at different paths.

3

Hi , thanks for your quick response :pray:

> For this reason, we recommend deleting files stored to your workflow immediately after you’ve used them.
Yes, that’s why I added the 3rd step to delete them, but somehow these files seem to pile up…

I will try your suggested approach of using the same file name every time. Then overwrites should prevent the pile to grow.

4

Is it possible your flow is exiting early before the deletion step is reached?

5

I thought the same. Maybe the return in return **await** fs.promises.unlink(steps.aws.$return_value); creates the issue? I blindly copied it from here Working with the filesystem in Node.js
Changed it and ran a couple of hundred invocations since then. Seems to work fine so far

6

Technically if you return a Promise from a Node.js code step, the workflow should wait for that promise to resolve before proceeding to the next one.

Awaiting the promise within the step doesn’t hurt though.

7

Yes, probably no difference, But the strange thing is, that the workflow was running fine for >20k invocations (with interruptions, because I trigger them from my laptop). And then the error appeared multiple times. Even after stopping and letting everything cool down for ~20 minutes (to make sure all containers are killed)

8

Thanks for you help . I hope it keeps working now. If not, I will give the same-name approach a try. Have a nice day! :wave:

9

Thanks, hoping it does. Best of luck. Have a nice weekend.