Ffmpeg fails with a TIMEOUT error

Help
1

This topic was automatically generated from Slack. You can find the original thread here.

Kris Niles : Howdy! I am using FFMPEG to extract a clip from an audio file (a podcast) in the tmp/ directory. It works most of the time, but sometimes it fails with a TIMEOUT error, and I don’t know why - here’s what I know:

• When the step successfully runs, I get an ACTIVE_HANDLE console log, and the output is returned via stderr (I assume it should return via stdout?). But it works, nonetheless. Is this set up correctly? I thought return await was the correct method to avoid that?
• When the step fails with a TIMEOUT, if I run the same input a second time, it succeeds. So I’m not sure what’s causing it to intermittently timeout, rather than every time. I’ve increased the timeout to 60 seconds, but to no avail. What could be causing it to timeout sometimes?
Screenshots below of a successful run, and a timeout run (both on the same file input).

image.png
image.png2220×1398 314 KB

image.png
image.png2290×1438 294 KB

2

Kris Niles : (- I added you to the workflow a while back - it’s called 1. Create MP3 Clip , shared by @krisniles)

3

Dylan Sather (Pipedream) : would you mind trying to increase the workflow’s memory in the Settings tab?

I reviewed the workflow and it’s not clear to me exactly where it’s timing out. Increasing the memory also increases its compute, so if the bottleneck is with either, increasing the memory will help. I’d like to see if that works, first.

4

Kris Niles : Cool! I had increased it to 512mb about a week ago… should I go higher?

5

Kris Niles :

image.png
image.png966×274 18.6 KB

6

Dylan Sather (Pipedream) : yes, I would. How large are the files you’re processing? Do you happen to know how long those same files take to process locally?

7

Kris Niles : Most podcasts are less than 50mb - but there are some long podcasts that can get up to 100-200mb. I saw that was erroring out when the file size got close to 256mb, so I increased to 512.

8

Kris Niles : I have not tried to process them locally - but in theory it should be super fast, because I’m not transcoding with FFMPEG, just making a copy, from X seconds to X+30 seconds.

9

Kris Niles : I can run some local tests to confirm.

10

Kris Niles : Did you happen to make any changes to the workflow? I see that it’s erroring out pretty consistently now - I just wanted to roll back to a previous version, but didn’t want to revert any changes you might have made.

11

Dylan Sather (Pipedream) : I did not make any changes to the workflow

12

Kris Niles : Ok - this is interesting - I’m seeing both a command failed and a timeout error for the exact same podcast. My only guess here is that the FFMPEG command is erroring, but just ends up timing out, rather than returning an error? These are all the exact same input:

image.png
image.png680×932 42.7 KB

13

Kris Niles : So, it’s either failing in 10 seconds, or timing out after 60 seconds. I don’t know what could be causing the difference.

14

Dylan Sather (Pipedream) : Does the error make sense for the given input? I’m curious if you address that, if it may fix the timeout.

Can you also add a timeout on the call to exec? I’d like to see if, when the timeout is hit, you see anything in stdout / stderr from ffmpeg

15

Kris Niles : Ok! I was able to resolve the “command failed” errors above - that was indeed ffmpeg failing on certain files. Should be resolved :+1:

However, I’m still getting intermittent timeouts from the step. Most of the time it works, but sometimes it times out.

If I replay a previous event multiple times, sometimes it works, sometimes it times out, which is super curious… nothing is changing between replays, so I feel like I’ve isolated it to Pipedream, rather than the file, or ffmpeg itself.

I added a 5000ms timeout to the exec call as you asked, but when I do that, the step times out every single time (after ~80 seconds?). But ffmpeg is reporting success in the logs.

Perhaps I’ve set up the timeout wrong? Screenshot below…

ScreenShot 2021-09-23 at 09.20.10@2x.png
ScreenShot 2021-09-23 at 09.20.10@2x.png3138×1862 669 KB

16

Dylan Sather (Pipedream) : I’m currently out of the office but may have some ideas

17

Giao Phan : I’m not sure if this is entirely the problem, but using the callback with a promisify function, I don’t think is supported. From my testing, it appears that the promise won’t resolve in that case. Can you try without the callback?

18

Kris Niles : Thanks ! Sorry, I’m a bit of a novice - is this how I would remove the promisify function?

I changed:
**const** exec = promisify(require('child_process').exec);
to:
**const** { exec } = require('child_process');

image.png
image.png2480×1268 323 KB

19

Giao Phan : Sorry, I meant you want the promisify, you don’t want to call exec with a calback. The code would look something like

const rv = await exec("ffmpeg", {timeout: 5000})
console.log(return)
return rv
20

Giao Phan : Closer to your original code, but with the timeout