Is there a way to ensure only new RSS Feed items are sent to a Custom Request (POST)?

This topic was automatically generated from Slack. You can find the original thread here.

Michael Eades : Not sure if this is the right channel for this; please feel free to point me in the right direction if not.
Having an issue with a workflow. New RSS Feed Item -> Custom Request (POST). It runs wonderfully when a new item is added but triggers the POST for every item in the RSS feed, not just the new one.

I understand there is some de-duplication efforts happening to not do this but I’m not clear on how I can setup my workflow to ensure that only the new item is being sent to the custom request.

Dylan Pierce : Hi Michael, yes under the hood all sources like the the New RSS Feed Item have a dedupe strategy defined.

RSS feeds are tricky, since there’s not a defined data structure and index to use to dedupe against.

Here’s the exact definition of the deduping strategy on this source:[…]f32/components/rss/sources/new-item-in-feed/new-item-in-feed.ts

You can see that it uses the unique strategy, based on the ID emitted:[…]f32/components/rss/sources/new-item-in-feed/new-item-in-feed.ts

But, you can also use code + a data store to dedupe yourself:

Michael Eades : thank you. that is helpful.

when you say

You can see that it uses the unique strategy, based on the ID emitted:

am I able to define which field from the RSS feed should be considered for the ID? I’d prefer not to have to write custom code / use a data store if I can avoid it.

Michael Eades : my RSS feed example is here:

the unique id would be the <dc:identifier> but I agree that may not be standard naming for your deduping to recognize

Michael Eades : is there any sort of dedupeKey that can be defined with the new-item-in-feed approach or is the custom data store the only way to go?

Dylan Pierce : You can inspect the code of any trigger on Pipedream in our public registry.

Here’s the specific code for the new item in feed: pipedream/new-item-in-feed.ts at master · PipedreamHQ/pipedream · GitHub

You can see that this particular method is responsible for parsing feeds and attempting to extract an ID:

Here’s where I believe the ID is generated based on attributes available in the feed: pipedream/ at ad6a9ecb8681a2e0cba9e8b480498e7efd68ad35 · PipedreamHQ/pipedream · GitHub

If modifying the RSS feed content is possible, you could potentially add an id field which would help with this deduping issue without a datastore or forking this source’s code.

A new update to this RSS trigger was made to resolve this issue.