How to Configure External Sorting Property in Mongo Connector to Resolve Out-of-Memory Exception?

Bugs
1

This topic was automatically generated from Slack. You can find the original thread here.

hi :wave: i’m having issues with the Mongo connector
when using the On new document trigger, i’m getting an out-of-memory exception

Sort exceeded memory limit of 33554432 bytes, but did not opt in to external sorting.

not sure where I can configure the external sorting property in the trigger properties.

thanks!

2

I’m not sure which version of MongoDB you’re using, but it looks like it’s limited to 100mb of storage per stage:

https://www.mongodb.com/docs/manual/reference/operator/aggregation/sort/?_ga=2.118209871.735973999.1723645320-1791210708.1723645320#-sort-[…]ictions

Under the hood, the trigger is probably querying too much data.

It would be cool if you could define a projection to trim this query down, and/or pass options like allowDiskUse

3

Pipedream can help make these changes if you submit an integration request here: Support - Pipedream, or you can modify the source code with a fork of the Pipedream Github Repo

4

i think I’ve identified the issue w/ the plugin

the way it queries mongo is returning a huge response, which is throwing the mongo exception

      const collection = this.mongodb.getCollection(client, this.database, this.collection);
      const sort = {
        [this.timestampField]: -1,
      };
      const documents = await collection.find().sort(sort)
        .toArray();

the collection.find() without any filter criteria returns all docs in the collection, and the filter is applied in-memory after the response is received

is the appropriate path here:

  1. open a PR?
  2. file an issue?
    meanwhile, i’m trying to execute the same code in a custom NodeJS action from the source to unblock the query, and running into an error where props: { db: $.service.db } is throwing a null exception when trying to access this.db.set("field", value)
5

Yup that’s what I suspected. My prior message links to Pipedream’s support page where you can file a feature request for existing components.

meanwhile, i’m trying to execute the same code in a custom NodeJS action from the source to unblock the query, and running into an error where props: { db: $.service.db } is throwing a null exception when trying to access this.db.set("field", value)
I believe the problem here is that the $.service.db is only available in sources not actions.

The equivalent for actions & regular Node.js code steps are using Data Stores: Using Data Stores - Pipedream

6

I believe the problem here is that the $.service.db is only available in sources not actions.
really great info - thanks for this

7

i created a small PR to fix the issue

i’m seeing some Vercel permission failures though
is that something that should be fixed on my end?

8

Vercel is only for the public facing docs, it’s not related to components

9

A Pipedream dev will reach out to you if there’s any issues. Thanks for contributing a fix :slightly_smiling_face:

10

Helps me out - I’m a fellow mongo user too