This topic was automatically generated from Slack. You can find the original thread here.
is there a way to stop a mongodb trigger from fetching all previous records and just listen to new records from the present date onwards so we don’t have to use credits on old records?
Yes, you can configure the MongoDB trigger to only listen to new records from the present date onwards. When setting up the trigger, you’ll find an option to “Skip the first (N) events”. Set this value to a large number (e.g., 1000000) to ensure that it skips all previous records and only listens to new records. Please follow these steps to set up the MongoDB trigger: 1. Search for the MongoDB app from the trigger menu. 2. Select the appropriate trigger (e.g., New Document, New Field in Document, etc.). 3. Connect your MongoDB account. 4. Select a Database or pass a string from a previous step. 5. Select a Collection or pass a string from a previous step. 6. Configure the trigger with the “Skip the first (N) events” option set to a large number. 7. Follow the instructions on the trigger configuration screen. You may need to generate a test event from MongoDB to get real data to test your workflow. Remember to monitor your credit usage to ensure you’re not consuming credits on old records.
I don’t see an option to Configure the trigger with the “Skip the first (N) events” option set to a large number.
I apologize for the confusion earlier. It seems I provided incorrect information. The MongoDB triggers in Pipedream do not have a “Skip the first (N) events” option. However, you can add a condition in your workflow to check the timestamp of the incoming event and process only the events that occur after a specific date. Here’s an example of how you can do this: 1. Add a code step after the MongoDB trigger in your workflow. 2. Use the following code snippet to check the timestamp of the incoming event: const currentDate = new Date(); const eventDate = new Date(steps.trigger.event.timestamp); // Replace 'timestamp' with the appropriate field from the MongoDB event const specificDate = new Date("2022-01-01T00:00:00Z"); // Replace with your desired date if (eventDate > specificDate && eventDate <= currentDate) { // Process the event } else { // Ignore the event } Replace the timestamp field with the appropriate field from the MongoDB event, and set the specificDate variable to the date you want to start processing events from. This way, your workflow will only process events that occur after the specified date.
are you using this trigger? And to confirm, you’re saying it’s emitting events for existing documents and not just new ones?
Hi that’s correct, I’m using the Mongo New Document trigger. I’m hoping to have the trigger only initiate when a new document appears in the collection, rather than trigger on every document that already exists in the collection. If I try to test the workflow, it basically triggers on every document that already exists and then immediately makes us run out of credits.
That sounds like sn unintended bug with the trigger. can you help triage the issue with this MongoDB trigger?
Sure, I’ll check it now
Hi @U02A06JPKEH, after checking the source, I see that there’s some improvement needed to resolve this issue.
I’ve created a ticket here and added it to Component backlog
thanks
can you let me know your workspace ID? I’d like to look at your account to figure out if we can give you some credits back, since it sounds like this bug got in your way of actually making use of Pipedream. You can find your workspace ID here: https://pipedream.com/settings/account