Request to MongoDB times out

Help
, ,
1

This topic was automatically generated from Slack. You can find the original thread here.

Najmuzzaman Mohammad : Hey all. Nazz here from Clipboard Health. I am trying to get data from MongoDB but because the dataset is huge, the request times out. It works great on Retool/Appsmith though. Any suggestions?

2

Dylan Sather (Pipedream) : Hi , just to confirm, are you using a Pipedream workflow to fetch the data from Mongo?

If so, is your execution timeout limit set to the default of 30 seconds, or have you raised it further to see if the query works?

3

Najmuzzaman Mohammad : Thanks Dylan. Now, I am facing the issue of payload limit. I tried to store the resultant array in a tmp file and retrieve it but the file turns out blank

4

Dylan Sather (Pipedream) : Would you mind sharing your workflow with dylan@pipedream.com?

5

Najmuzzaman Mohammad : my bad. didn’t stringify and parse

6

Dylan Sather (Pipedream) : Gotcha, working ok now?

7

Najmuzzaman Mohammad : yes. but question. I am sending the data to Parabola. Is there a pipedream API I can hit to get a step’s exported result?

8

Dylan Sather (Pipedream) : We don’t have a public API for that specifically, but as we build out the API for workflows I expect that’ll come in the future. What’s the core use case? We might be able to get you the same data in a different way.

9

Najmuzzaman Mohammad : I want to do some transformations on the mongo data and then send it to Salesforce

10

Najmuzzaman Mohammad : Doing tabular transforms are much easier in Parabola. For example, I want only the first order placed of each customer in the table to send that data to Salesforce

11

Dylan Sather (Pipedream) : interesting, that’s good feedback. Could you send the results of the query in a webhook to Parabola?

12

Dylan Sather (Pipedream) : such that Pipedream triggers the Parabola workflow?

13

Dylan Sather (Pipedream) : also, there is an API that’ll function closer to what you wanted originally, and I’m writing up some docs that should clarify how to use this for you.

The tl;dr is: within a workflow, you can call $send.emit() to “emit” an arbitrary event:

$send.emit({
  raw_event: {
    name: "Luke",
    title: "Jedi",
  }
})

and retrieve that data using this endpoint:

curl -H "Authorization: Bearer <pipedream api key>" 'https://api.pipedream.com/v1/workflows/[WORKFLOW ID]/event_summaries?expand=event'