Is there a way to ensure only new RSS Feed items are sent to a Custom Request (POST)?

user-1 · December 11, 2022, 2:26am

This topic was automatically generated from Slack. You can find the original thread here.

Michael Eades : Not sure if this is the right channel for this; please feel free to point me in the right direction if not.
Having an issue with a workflow. New RSS Feed Item -> Custom Request (POST). It runs wonderfully when a new item is added but triggers the POST for every item in the RSS feed, not just the new one.

I understand there is some de-duplication efforts happening to not do this but I’m not clear on how I can setup my workflow to ensure that only the new item is being sent to the custom request.

user-2 · December 11, 2022, 2:26am

Dylan Pierce : Hi Michael, yes under the hood all sources like the the New RSS Feed Item have a dedupe strategy defined.

RSS feeds are tricky, since there’s not a defined data structure and index to use to dedupe against.

Here’s the exact definition of the deduping strategy on this source: https://github.com/PipedreamHQ/pipedream/blob/1b2f492496388a7d870edbcdb9b11f0d77d5[…]f32/components/rss/sources/new-item-in-feed/new-item-in-feed.ts

You can see that it uses the unique strategy, based on the ID emitted: https://github.com/PipedreamHQ/pipedream/blob/1b2f492496388a7d870edbcdb9b11f0d77d5[…]f32/components/rss/sources/new-item-in-feed/new-item-in-feed.ts

But, you can also use code + a data store to dedupe yourself:

user-1 · December 11, 2022, 2:26am

Michael Eades : thank you. that is helpful.

when you say

You can see that it uses the unique strategy, based on the ID emitted:

am I able to define which field from the RSS feed should be considered for the ID? I’d prefer not to have to write custom code / use a data store if I can avoid it.

user-1 · December 11, 2022, 2:26am

Michael Eades : my RSS feed example is here:
https://feeds.pinboard.in/rss/u:yewknee/t:quicklinks/

the unique id would be the <dc:identifier> but I agree that may not be standard naming for your deduping to recognize

user-1 · December 11, 2022, 2:26am

Michael Eades : is there any sort of dedupeKey that can be defined with the new-item-in-feed approach or is the custom data store the only way to go?

user-2 · December 11, 2022, 2:26am

Dylan Pierce : You can inspect the code of any trigger on Pipedream in our public registry.

Here’s the specific code for the new item in feed: pipedream/new-item-in-feed.ts at master · PipedreamHQ/pipedream · GitHub

You can see that this particular method is responsible for parsing feeds and attempting to extract an ID:

github.com

PipedreamHQ/pipedream/blob/ad6a9ecb8681a2e0cba9e8b480498e7efd68ad35/components/rss/app/rss.app.ts#L121


      
            const buffer = await new Promise<Buffer>((resolve, reject) => {
              const _buf = [];
              stream.on("data", (chunk) => _buf.push(chunk));
              stream.on("end", () => resolve(Buffer.concat(_buf)));
              stream.on("error", (err) => reject(err));
            });
            const contentString = buffer.toString();
            const feed = JSON.parse(contentString);
            return feed?.items || [];
          },
          async fetchAndParseFeed(u: string) {
            const url = this.validateAndFixFeedURL(u);
            const response = await this.fetchFeed(url);
            if (this.isJSONFeed(response)) {
              return await this.parseJSONFeed(response.data);
            } else {
              return await this.parseFeed(response.data);
            }
          },
          validateAndFixFeedURL(u: string) {
            if (!u) throw new ConfigurationError("No URL provided. Please enter an RSS URL to fetch");

Here’s where I believe the ID is generated based on attributes available in the feed: pipedream/rss.app.ts at ad6a9ecb8681a2e0cba9e8b480498e7efd68ad35 · PipedreamHQ/pipedream · GitHub

If modifying the RSS feed content is possible, you could potentially add an id field which would help with this deduping issue without a datastore or forking this source’s code.

pierce · December 12, 2022, 2:18pm

A new update to this RSS trigger was made to resolve this issue.