Processing data with prebuilt modules between steps

[vanetreg]

Hello,

I’d like to process tweets and save modified/processed results to Google Sheets.
I also want to process hashtags, which count can be from zero to the max Twitter limit.
I’d like to save all hashtags of a tweet as a single cell value in a GS row, ( as a string like: “#hashtag1, #hashtag2, …”; optionally with or without “#” )
I checked Pipedream’s Node module whether I find a solution but I haven’t.
Not being a coder but working with some no-code/low-code tools ( eg. Integromat ) I have an idea that it would be great from Pipedream to provide us some pre-built modules (GUI) which process/convert data between objects, arrays and strings, optionally with functions available like trim spaces or replace chars ( like replacing “#” with “” when taking hashtags )

• Have I missed something and they exists? Could you link me to that?

• Could you recommend me a full working and/or low-code solution or a code sample where data is processed between steps (like Twitter Search Mentions and Google Sheets Add a Row) in a new intermediary step?

Thank you!

[pierce]

Hi @vanetreg

First off, welcome to the Pipedream community. Happy to have you!

Great questions, great questions.

We do have a Helper Functions app that contains useful utilities like converting HTML to Markdown, Base64 encoding strings, pretty printing JSON objects, etc.

But unfortunately we do not yet have a Text Helpers type of action, but I love this idea and shared this feedback with the team.

In the meantime here are some example Node.js code steps you can copy & paste so you can at least get unblocked:

Extract Hashtags Action

// To use previous step data, pass the `steps` object to the run() function
export default defineComponent({
  props: {
    text: {
      type: 'string',
      label: 'Text to extract hashtags from',
    }
  },
  async run({ steps, $ }) {
    return this.text.match(/#\w+/g);
  },
});

[vanetreg]

Hello @pierce

thank you for your response.
I have a path of:
steps.trigger.event.entities.hashtags
which is extracted to Google Sheets originally as:
[{"text":"SmallStepsToABetterPlanet","indices":[11,37]},{"text":"Starbucks","indices":[38,48]},{"text":"LineFriends","indices":[49,61]},{"text":"Brown","indices":[62,68]},{"text":"MiiR","indices":[69,74]}]
in this case with 5 hashtags, so it should be saved as a string of 5 hashtags separated with comma and I’m not sure your snippet gonna do it.
But thanks a lot, later I’ll check both your snippets and existing Helper Functions.
Thanks again!

[pierce]

@vanetreg

[{“text”:“SmallStepsToABetterPlanet”,“indices”:[11,37]},{“text”:“Starbucks”,“indices”:[38,48]},{“text”:“LineFriends”,“indices”:[49,61]},{“text”:“Brown”,“indices”:[62,68]},{“text”:“MiiR”,“indices”:[69,74]}]

Oh I see, thanks for the data snippet.

In that case you’d looking to collapse an array into a comma separated string, here’s a snippet to build a Node.js step to do that:

// To use previous step data, pass the `steps` object to the run() function
export default defineComponent({
  props: {
    records: {
      type: 'string[]',
      label: 'Array of records to extract hashtags from',
    }
  },
  async run({ steps, $ }) {
    return this.records.map(record => record.text).join(',');
  },
});

[vanetreg]

Hello @pierce

thank you for your help and code!

• regarding PD Helper Functions, I haven’t found one action I need.
• regarding 2nd code snippet you gave, it doesn’t return any value.
  the module’s Results/Exports" section shows:

steps.node {1}
  $return_value:

the Results/Inputs tab shows:

[ [ { "text": "SmallStepsToABetterPlanet", "indices": [ 11, 37 ] }, { "text": "Starbucks", "indices": [ 38, 48 ] }, { "text": "LineFriends", "indices": [ 49, 61 ] }, { "text": "Brown", "indices": [ 62, 68 ] }, { "text": "MiiR", "indices": [ 69, 74 ] } ] ]

as “Array of records to extract hashtags from”, so it looks the original array of 5 objects is wrapped into another array and I don’t know why we need this.

If I add your first snippet before your second (in a separate Node module) and set the input correctly, first snippet module gives a:

TypeError
this.text.match is not a function

with details:

    at Object.run (file:///tmp/ee/c_EpfldAwy/index.mjs:10:22)
    at global.executeComponent (/var/task/launch_worker.js:171:53)
    at MessagePort.messageHandler (/var/task/launch_worker.js:653:28)

I have this scenario working at Integromat, but I consider moving most of my scenarios to Pipedream where possible, but their no-code approach really works eg. in this case.

I’ll dig myself more into PD’s doc’s code section, because step exports and using props etc. is not clear, but I’m not sure I’m gonna get it all.

Thank you!

[pierce]

@vanetreg

[ [ { "text": "SmallStepsToABetterPlanet", "indices": [ 11, 37 ] }, { "text": "Starbucks", "indices": [ 38, 48 ] }, { "text": "LineFriends", "indices": [ 49, 61 ] }, { "text": "Brown", "indices": [ 62, 68 ] }, { "text": "MiiR", "indices": [ 69, 74 ] } ] ]

Hi @vanetreg,

I just noticed that your output contains two arrays not just one like my original code snippet expected.

Here’s an updated Node.js component to handle that exact data format you shared:

export default defineComponent({
  props: {
    records: {
      type: 'string[]',
      label: 'Array of records to extract hashtags from',
    }
  },
  async run({ steps, $ }) {
    return this.records[0][0].map(record => record.text).join(',');
  },
});

And here’s the results from my test:

If it continues to be a problem, could you please screenshot the Node step and the exports data of the step that’s producing these hash tags?

Also, we just published a video this morning that might help explain how to connect steps together in the workflow builder:

Passing data between Workflow Steps

Let me know if this helps you!

[vanetreg]

Hello @pierce

thank you for the help, I really appreciate it!

It still doesn’t work with your latest code.
Pls. find my screenshot attached.
Pls. note my trigger is a Twitter search,
second step is your latest code “with 2 arrays”.

The copied path is:
steps.trigger.event.entities.hashtags

The copied value is:
[{"text":"SmallStepsToABetterPlanet","indices":[11,37]},{"text":"Starbucks","indices":[38,48]},{"text":"LineFriends","indices":[49,61]},{"text":"Brown","indices":[62,68]},{"text":"MiiR","indices":[69,74]}]

Example Twitter search response in JSON is here…

Best regards,
V.

[pierce]

Hi @vanetreg ,

Looks like I neglected to tell you to replace the component outright, not add another step.

Your workflow should look like this:

1. Twitter Search Trigger
2. Node.js action (with the first code example I shared)
3. Add a single row to a Google Sheet

The Node.js step code does work with the Twitter search trigger, I tested it myself with the {{steps.trigger.event.entities.hashtags}}. It produces one string of text hashtag1, hashtag2, etc.

Hopefully this helps!

[vanetreg]

Hello @pierce

I watched your new videos at YT and retried this but my code still doesn’t work!

I shared this workflow with PD Support, could you have a look at pls.?

Thank you!

[pierce]

Hi @vanetreg,

I’m sorry we do not offer direct one on one consulting for free Developer Tier accounts.

Our Teams customers have access to priority direct Slack support, but we can help guide here if you could share more information about the problem step, the data given, produced and expected during your workflow’s execution.

Or if you need help from an expert, you can ask for help on our job board here:

Hire an expert

Thanks

[vanetreg]

For others, the correct Node code step code is:

export default defineComponent({
  props: {
    records: {
      type: 'string[]',
      label: 'Array of objects to extract hashtags from',
    }
  },
  async run({ steps, $ }) {
    return this.records.map(record => record.text).join(',');
  },
});