Uploading media files

Hello dear Pipedream Community!

I am trying for a while now to create files in Akeneo with Pipedream. It is also the first time for me to work with file streams at all.

I made some nice progress, but it seems like I can’t do the final leap without help.

The use case is, that I create a barcode in PDF_co and upload the created image to a dedicated media attribute in Akeneo.

What already works:

  • Create a PNG of the barcode in PDF_co
  • Download that image to TMP
  • Read that file again from TMP

The steps are:

download_image_to_tmp

import stream from "stream";
import { promisify } from "util";
import fs from "fs";
import got from "got";

export default defineComponent({
  async run({ steps, $ }) {
    // Download the barcode image file to /tmp
    const pipeline = promisify(stream.pipeline);
    return await pipeline(
      got.stream(steps.pdf_co_barcode_generator.$return_value.url),
      fs.createWriteStream('/tmp/barcode.png')
    );
  }
})

read_file_from_tmp

import fs from "fs";

export default defineComponent({
  async run({ steps, $ }) {
    var myFile = fs.createReadStream('/tmp/barcode.png');
    return myFile;
  }
});

The documentation of Akeneo’s API endpoint is here and this is the Postman collection of Akeneo.

In Postman it works like this:

In Pipedream, I tried two different ways to upload the file.

Upload via HTTP POST Request step
I wanted to add a screenshot here, but I am allowed to upload only one. :slightly_frowning_face:

Body
Content-Type: multipart/form-data

Name: product
Value: {“identifier”:“{{steps.trigger.event.body.events[0].data.resource.identifier}}”, “attribute”:“barcode”, “scope”: null,“locale”:null}

Name: file
Value: {{steps.read_file_from_tmp.$return_value}}

But I got the error:

source.on is not a function

As far as I could find out it seems to be related to the way the form is structured. But with this step, I can’t really influence that, right?

So I also tried to build the upload step myself with Axios, based on the code that was provided by Postman.

Upload with a Node step

import fs from "fs";
import { axios } from "@pipedream/platform"
import FormData from "form-data"

// To use previous step data, pass the `steps` object to the run() function
export default defineComponent({
  async run({ steps, $ }) {

    var data = new FormData();
    data.append('product', '{"identifier":"' + steps.get_product_information.$return_value.identifier + '", "attribute":"barcode", "scope": null,"locale":null}');
    data.append('file', await fs.createReadStream('/tmp/barcode.png'));

    var config = {
      method: 'post',
      url: steps.akeneo.$return_value.host + '/api/rest/v1/media-files',
      headers: { 
        'Authorization': 'Bearer '+ steps.akeneo.$return_value.token, 
        'Content-Type': 'multipart/form-data', 
        ...data.getHeaders()
      },
      data : data
    };

    await axios(config)
    .then(function (response) {
      console.log(JSON.stringify(response.data));
    })
    .catch(function (error) {
      console.log(error);
    });

  },
})

Here I get a timeout and a warning:

Code was still running when the step ended. Make sure to await all Promises, or promisify callback functions. May be a false positive

Has anyone some idea how to improve this? Preferably I would like to use the HTTP request step. So config in the UI wherever possible.

Thanks in advance!
Anton

Hello @anzglr, I suspect the error you faced is because of this line right here

Could remove the await keyword and try again?

    data.append('file', fs.createReadStream('/tmp/barcode.png'));

In more detail, the data.file() function accept the read stream param, but adding the await keyword before the read stream param wraps your read stream inside a Promise which is not expected.

You can refer to this site for how the form-data is structured: form-data - npm

Thanks for your reply, @vunguyenhung.

But unfortunately this is not changing anything. The await wasn’t there in the beginning. I added more of those at some point, as an answer to the error message. I am doing try & error here :smile:

I’ll take it out again.

Even though I have a good technical understanding, I am not a programmer. So most helpful for an answer would be a piece of code I could try. :slight_smile:

Hello @anzglr,

I think you can try this

import fs from 'fs';
import { axios } from '@pipedream/platform';
import FormData from 'form-data';

// To use previous step data, pass the `steps` object to the run() function
export default defineComponent({
  async run({ steps, $ }) {
    const data = new FormData();

    const product = JSON.stringify({
      identifier: steps.get_product_information.$return_value.identifier,
      attribute: 'barcode',
    });
    const fileStream = fs.createReadStream('/tmp/barcode.png');
    data.append('product', product);
    data.append('file', fileStream);

    var config = {
      method: 'post',
      url: steps.akeneo.$return_value.host + '/api/rest/v1/media-files',
      headers: {
        Authorization: 'Bearer ' + steps.akeneo.$return_value.token,
        'Content-Type': 'multipart/form-data',
        ...data.getHeaders(),
      },
      data: data,
    };

    return await axios($, config)
      .then(function (response) {
        console.log(JSON.stringify(response.data));
      })
      .catch(function (error) {
        console.log(error);
      });
  },
});

Note that the step akeneo should have correct value, since this action depends on it

Thanks!

This still gives this error:

Code was still running when the step ended. Make sure to await all Promises, or promisify callback functions. May be a false positive

The Akeneo step gives correct values. I also use it in other workflows.

But maybe I should let you confirm that the file writing and reading from and to tmp is done correctly.

Because I just noticed that the error from above also can be found on the reading step. I don’t know if it wasn’t there before or if I just missed it.

Oh apologize @anzglr, I’ve updated the code, could you copy and paste the new code to your step and try again?

import fs from 'fs';
import { axios } from '@pipedream/platform';
import FormData from 'form-data';

// To use previous step data, pass the `steps` object to the run() function
export default defineComponent({
  async run({ steps, $ }) {
    const data = new FormData();

    const product = JSON.stringify({
      identifier: steps.get_product_information.$return_value.identifier,
      attribute: 'barcode',
    });
    const fileStream = fs.createReadStream('/tmp/barcode.png');
    data.append('product', product);
    data.append('file', fileStream);

    var config = {
      method: 'post',
      url: steps.akeneo.$return_value.host + '/api/rest/v1/media-files',
      headers: {
        Authorization: 'Bearer ' + steps.akeneo.$return_value.token,
        'Content-Type': 'multipart/form-data',
        ...data.getHeaders(),
      },
      data: data,
    };

    return await axios($, config)
      .then(function (response) {
        console.log(JSON.stringify(response.data));
      })
      .catch(function (error) {
        console.log(error);
      });
  },
});

Specifically, I’ve updated this part

    return await axios($, config)

Unfortunately still the same.

But as I said, in the step before (reading from tmp) there is the same issue. So maybe it is not because there is something wrong in your code, but because the file can’t be loaded in the previous step.

I just notice that this part is identical to your last code. Did you post the wrong one?

Hello @anzglr,

I think maybe this is the case, could you test the whole workflow by deploying the workflow and trigger it from Postman?

On another note, did you see the file in Akeneo? If the file is uploaded successfully on Akeneo then we can ignore the error (false positive)

I deployed it and let the whole workflow run. No real errors, but the warning from above. The file is not in Akeneo. Not before and not after the complete workflow run.

Hello @anzglr,

Apologize for the late reply. I think it would be better if Pipedream has an action for you to use directly, instead of using code.

I have created a ticket here to request for the action you used on Akeneo. I’ve put it to our prioritized backlog: [ACTION] Akeneo: Create a new Product Media file · Issue #4824 · PipedreamHQ/pipedream · GitHub

After this ticket is resolved, you’ll be able to use the Akeneo action without touching the code. Kindly subscribe to the action if you want :pray:

2 Likes

Thanks @vunguyenhung, that’s awesome! :slight_smile: :clap:

The new action works like a charm! Thank you!

It would be absolutely fantastic, if we could have something like this for asset media files as well, so that our paying customers on the Akeneo Enterprise Edition (who mostly use assets) have as much fun to create workflows in Pipedream. :slight_smile:

We can also discuss this on on Slack. I joined your workspace there. :wave: