Workflow Payload Limit Exceeded error when uploading large files with axios

Help
21

Giao Phan : Are you using $respond by any chance?

22

Nacho Caballero : yes to disabling and no to $respond

23

Giao Phan : Mind reposting the entire code to that step?

24

Nacho Caballero : ```
this.id = $checkpoint[steps.checkpoint.id].buzzsproutEpisodeId;

if (!this.id) {
const axios = require("@pipedreamhq/platform").axios;
const fs = require(‘fs’);

const request = {
method: “POST”,
url: https://www.buzzsprout.com/api/${process.env.BUZZSPROUT_PODCAST_ID}/episodes.json,
headers: {
‘Authorization’: Bearer ${process.env.BUZZSPROUT_TOKEN}
},
data: {
title: steps.init.titleWithoutAuthors,
published_at: steps.init.publishAtTime,
description: steps.init.descriptionHtml,
summary: steps.init.teaser,
artist: steps.init.artist,
episode_number: steps.init.episodeNumber,
audio_file: fs.readFileSync(steps.downloadAudioFile.filePath).toString(),
}
};

const response = await axios(this, request);
this.id = response.id;
$checkpoint[steps.checkpoint.id].buzzsproutEpisodeId = this.id

25

Giao Phan : I don’t use the platform axios much, does it auto export or anything?

26

Dylan Sather (Pipedream) : nah you have to export manually, for example **await** require("@pipedreamhq/platform").axios(**this**, config) shouldn’t set any exports (just tested)

27

Giao Phan : One possible thing is that the $checkpoint is already huge, so updating it blows through the limit. Does it look reasonable in the settings?

28

Giao Phan : Oh, if the response errors out, we will export something to debug

29

Giao Phan : in the axios wrapper

30

Dylan Sather (Pipedream) : oh yes good call

31

Nacho Caballero : I just tried running with plain axios and without $checkpoint and the error persists.

32

Nacho Caballero : I can’t make it any smaller :sweat_smile:

image.png
image.png3038×896 272 KB

33

Nacho Caballero : If I disable that step, the test workflow runs.

image.png
image.png2314×862 117 KB

34

Nacho Caballero : The last step shows the file size of the file used in the disabled step (30 MB)

image.png
image.png1354×1166 122 KB

35

Giao Phan : hmm, one more test, can we put a try catch around the axios call?

36

Nacho Caballero :

image.png
image.png2356×608 53.9 KB

37

Nacho Caballero : I can bypass that error adding:
‘maxContentLength’: Infinity,
‘maxBodyLength’: Infinity`

38

Giao Phan : Yeah, so I think what is happening is that the error the axios throws includes the original request with its big payload. The platform one cleans it up a bit, but not enough.

39

Nacho Caballero : You’re right. In fact, if I use the platform axios, it fails. It only works if I use the standard axios library

40

Nacho Caballero : Thanks for the help, I was struggling to figure it out :sweat_smile: