Receive third party API's object data

Dev
1

Hi guys, good day.

I have the following workflow:
-An HTTP API step
-A Node.js step to filter_favicon_requests
-Another Node.js step.

In this last one I’m trying to get an object given by a third party API here, but exchanging the vars for consts:

async (event,steps) => {
  const ParseHub = require('parsehub');
  const api = new ParseHub(yourApiKey);

  api.getAllJobs(function(err, jobs)
  {
	console.log(jobs);
  });

  api.getAllJobs({ include_last_run: true }, function(err, jobs)
  {
	console.log(jobs);
  });

  await $respond({
    status: 200,
    body: '${steps.respond}',
  })
}

I get an issue with the await as I don’t seem to be using it right:

This step was still trying to run code when the step ended. Make sure you promisify callback functions and await all Promises. (Reason: GetAddrInfoReqWrap

I feel I’m missing something rather obvious which I’m not seeing. I’m new to the app and Node.js. I have javascript/jQuery knowledge.

I would appreciate a helping hand with this.

Thank you!
Adrián.

2

@adriang2099 you’re seeing that error because you’re passing callback functions to the getAllJobs method, which will run after the step finishes. Pipedream can’t wait for callback functions that run in this manner, so we’re warning you that they’re still running by the time the step ends. More about this problem + solutions here.

In your case you’ll need to wrap those api.getAllJobs calls with a Promise that resolves when the callback function is executed:

await new Promise((resolve) => api.getAllJobs(function(err, jobs) {
  console.log(jobs);
  resolve()
}));

This will wait for the API call to finish, log the jobs, and the resolve() call finishes execution and we move on to the next chunk of code.

Normally an npm package would support Promises out of the box, which would let you do something like this, conceptually:

const jobs = await api.getAllJobs()

This library is unfortunately a few years old and doesn’t have support for that. But in general if you see this callback-style example for another npm package on the web, try to look for “Promise” support in the docs for the package, and that should work on Pipedream.

Let me know if that helps!

3

Hi @dylburger, thank you for the fast reply.

I still can’t see my data I’m afraid.

As the ParseHub package
const ParseHub = require('parsehub');
doesn’t have Promise support I would need to use your suggestion

await new Promise((resolve) => api.getAllJobs(function(err, jobs) {
  console.log(jobs);
  resolve()
}));

Is that correct?

I tried running just:

  const ParseHub = require('parsehub');
  const api = new ParseHub(myToken);
  await new Promise((resolve) => api.getAllJobs(function(err, jobs) {
    console.log(jobs);
    resolve()
  }));

But I just get Console (1) undefined

I ran console.log(api); and showed ParseHub { _apiKey: 'myToken' } on the console so I get the glimpse that it’s correctly getting the ParseHub Node.js package but it’s not really getting my data (there’s no error if I just type ‘22’ on myToken).

I also ran console.log(ParseHub); and got [Function: ParseHub] I’m used to php’s var_dump to see everything behind an object/function/var and console.log isn’t helping me to see anything further than this. I found some functions online that can mimic var_dump on js but I figured I might be diverting now.

I feel I’m missing something again or maybe I didn’t fully understand your suggestion or technique.

Thank you again,
Adrián.

4

Thanks. The fact that you’re seeing the Console (1) undefined message below your workflow actually suggests that this might be working — the console.log(jobs) statement appears to be running. Are you sure there are jobs that should be returned in this case? You can also try running:

console.log(err)

within that section to see if an error was returned from the API.

I’d also recommend reaching out to someone from the ParseHub community to see if they may be able to help you troubleshoot. Based on the docs for the npm package, this code should work, but they may be able to help you figure out why it’s not.

5

Using the promisify util this way will throw the error (which pipedream should show you) or get the jobs:

const jobs = await require('util').promisify(api.getAllJobs)()
console.log(jobs)
6

Hi @dylburger.

Yeah I apologize, I omitted to add that I get these in all console.log(err):

    at Request.null (/opt/ee/node_modules/parsehub/parsehub.js:42:20)
    at Request.self.callback (/opt/ee/node_modules/request/request.js:199:22)
    at Request.emit (events.js:376:20)
    at Request.null (/opt/ee/node_modules/request/request.js:1160:14)
    at Request.emit (events.js:388:22)
    at IncomingMessage.null (/opt/ee/node_modules/request/request.js:1111:12)
    at IncomingMessage.emit (events.js:388:22)
    at null.endReadableNT (internal/streams/readable.js:1336:12)
    at process.processTicksAndRejections (internal/process/task_queues.js:82:21)

I agree this means I’m getting something. But I still get undefined after changing the account, project and run tokens to one I know has results.
After that change I got errors like “please specify job token” and there’s no job token in the apps documentation or package docs, it might have changed to “Run” after some time but I’m guessing. This I can understand to be a ParseHub thing now.

I have reached out to them but they’re not familiar with pipedream and the package is an unofficial third party library, they’re giving me other kind of solutions like using a static php page to deal with what I need which sounds good enough.

What I want:
After each ParseHub Run finishes it sends a callback function with the ok data or error to the webhook I specify. I intended to use Pipedream as such webhook in which I would save the data gathered (google docs or csv export) and then trigger then next run, save that too and go for the next and so on till the last.
It would be a one button thing whenever I need that data so no cron jobs needed.

7

Hi @g.
I tried running your code and I get TypeErrorCannot read property '_apiKey' of undefined but I can’t tell how or where should I specify the _apiKey, I tried:

const jobs = await require('util').promisify(api.getAllJobs)({
  _apiKey:'apiKey'
})
console.log(jobs)

without success.

8

Found out on my own:

I only needed to give the api_key through the URL (?api_key=API_KEY) and then use GET or POST depending on what method I’m using, no extra headers needed.

I’ll try to achieve my thing from here, thanks.