Axios response is not what I expect

This topic was automatically generated from Slack. You can find the original thread here.

Neil Lovelace : Good Afternoon -
I’m developing in multiple Node.js platforms currently. I’ve noticed that when I make calls using axios within Pipedream, it consistently returns the full response that I am expecting. In Azure, or in my local VSCode application, it’s hit or miss. Is there some special header setting that Pipedream has in place that ensures the consistent results that I see within the platform? I’m not asking or expecting anyone to divulge proprietary information. Just trying to gain an understanding as to why the different results between two platforms using the same npm package, api call, Bearer token, etc.

Danny Roosevelt : Hey Neil, we don’t do anything special beyond how we display the data in step exports. But we just return await axios(opts) in most cases, which returns the standard axios response.

Neil Lovelace : Thanks . I was hoping that you were going to point me to some specific settings. Since that is not the case, I’ll keep looking to figure out why my promises aren’t being fulfilled 100%.

I do appreciate your prompt reply!

Neil Lovelace : Good Morning All,
I’m running a step in my workflow using good ol’ axios. I’m not able to retrieve a full response payload despite my best efforts. Anyone else having issues like this?

I get a response status of 200, but the payload is not complete - see below.

{
  status: 200,
  statusText: 'OK',
  headers: {
    date: 'Wed, 15 Sep 2021 16:34:16 GMT',
    'content-type': 'application/json; charset=utf-8',
    'transfer-encoding': 'chunked',
    connection: 'close',
    'set-cookie': [
      'AWSALB=eWfER5FcpK/GvY+8mTtcmTfEwCPf3NyijgIA5vSwEBr8nslBbC6IzAZQCPjv0RnOqZSNGkGbV9Us364lYLQwd8XqykXvV3bA7kom5IceQS4KDh40tndy78odN+FJ; Expires=Wed, 22 Sep 2021 16:34:15 GMT; Path=/',
      'AWSALBCORS=eWfER5FcpK/GvY+8mTtcmTfEwCPf3NyijgIA5vSwEBr8nslBbC6IzAZQCPjv0RnOqZSNGkGbV9Us364lYLQwd8XqykXvV3bA7kom5IceQS4KDh40tndy78odN+FJ; Expires=Wed, 22 Sep 2021 16:34:15 GMT; Path=/; SameSite=None; Secure',
      'AWSALBTG=L6y+9a3BRfJNuJ6ewiPFAayOOAvG80OX1q+DojE3b/rQaU8rfcwHw11Z6eSI4ZAxTweot5Iund5ztqTf3QiMJSGl1UjiYOuZBk3RzBDqImTkRr4hBBRcWRj0DsC3EBJQrBpykWkY7+/mEgMBcZb00aVbbO1eh5jrk+s4ZjWv2gjR7BpGt70=; Expires=Wed, 22 Sep 2021 16:34:15 GMT; Path=/',
      'AWSALBTGCORS=L6y+9a3BRfJNuJ6ewiPFAayOOAvG80OX1q+DojE3b/rQaU8rfcwHw11Z6eSI4ZAxTweot5Iund5ztqTf3QiMJSGl1UjiYOuZBk3RzBDqImTkRr4hBBRcWRj0DsC3EBJQrBpykWkY7+/mEgMBcZb00aVbbO1eh5jrk+s4ZjWv2gjR7BpGt70=; Expires=Wed, 22 Sep 2021 16:34:15 GMT; Path=/; SameSite=None; Secure'
    ],
    'x-frame-options': 'SAMEORIGIN',
    'x-xss-protection': '1; mode=block',
    'x-content-type-options': 'nosniff',
    'per-page': '10000',
    total: '9',
    'x-rate-limit-limit': '3600',
    'x-rate-limit-remaining': '3596',
    'x-rate-limit-reset': '1631726092',
    etag: 'W/"0640594046722873fa6676187fb04229"',
    'cache-control': 'max-age=0, private, must-revalidate',
    'x-request-id': '25ec04a2-2fc7-4006-9114-70eaee74a588',
    'x-runtime': '0.481138',
    'strict-transport-security': 'max-age=15552000; includeSubDomains',
    'x-envoy-upstream-service-time': '491',
    server: 'envoy'
  },
  config: {
    url: 'https://api.procore.com/rest/v1.0/projects/448081/users?filters[employee]=true',
    method: 'get',
    headers: {
      Accept: 'application/json, text/plain, **/**',
      Authorization: 'Bearer eyJh...cmr3',
      'Procore-Company-Id': '4923',
      'Content-Type': 'application/json',
      'User-Agent': 'axios/0.21.4'
    },

The data returned in the response is as follows:

data: [
    { id: 945366, name: 'Neil Lovelace', vendor: [Object] },
    { id: 1012129, name: 'John Evans', vendor: [Object] },
    { id: 3607712, name: 'Jessica Freese', vendor: [Object] },
    { id: 5961345, name: 'Chris Williams', vendor: [Object] },
    { id: 6684024, name: 'Harrold Stewart', vendor: [Object] },
    { id: 6863821, name: 'Brittany Moss', vendor: [Object] },
    { id: 7196952, name: 'Todd Hogrefe', vendor: [Object] },
    { id: 7202959, name: 'Hajime Yokoyama', vendor: [Object] },
    { id: 7285171, name: 'Heidi Harrison', vendor: [Object] }
  ]

Each object in the array should contain ~ 30 key-value pairs, but as you can see there are only 3. In Postman, I get the complete response. Need some help figuring out why.

Danny Roosevelt : Hey Neil, can you share your workflow with me so I can take a closer look? You can hit the blue Share button at the top right and share with [danny@pipedream.com](mailto:danny@pipedream.com)

Danny Roosevelt : And can you also share the full response you’re getting from Postman? (DM works, and please make sure to omit any sensitive info)

Neil Lovelace : Sure thing. See below.

{
  config: {
    url: 'https://api.procore.com/rest/v1.0/projects/448081/users/945366',
    method: 'get',
    headers: {
      Accept: 'application/json, text/plain, **/**',
      Authorization: 'Bearer eyJh....c7OX',
      'Content-Type': 'application/json',
      'User-Agent': 'axios/0.21.4'
    },
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    timeout: 0,
    adapter: [Function: httpAdapter],
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    maxBodyLength: -1,
    validateStatus: [Function: validateStatus],
    transitional: {
      silentJSONParsing: true,
      forcedJSONParsing: true,
      clarifyTimeoutError: false
    },
    data: undefined
  },
  request: <ref **1> ClientRequest {
    _events: [Object: null prototype] {
      abort: [Function (anonymous)],
      aborted: [Function (anonymous)],
      connect: [Function (anonymous)],
      error: [Function (anonymous)],
      socket: [Function (anonymous)],
      timeout: [Function (anonymous)],
      prefinish: [Function: requestOnPrefinish]
    },
    _eventsCount: 7,
    _maxListeners: undefined,
    outputData: [],
    outputSize: 0,
    writable: true,
    destroyed: false,
    _last: true,
    chunkedEncoding: false,
    shouldKeepAlive: false,
    _defaultKeepAlive: true,
    useChunkedEncodingByDefault: false,
    sendDate: false,
    _removedConnection: false,
    _removedContLen: false,
    _removedTE: false,
    _contentLength: 0,
    _hasBody: true,
    _trailer: '',
    finished: true,
    _headerSent: true,
    socket: TLSSocket {
      _tlsOptions: [Object],
      _secureEstablished: true,
      _securePending: false,
      _newSessionPending: false,
      _controlReleased: true,
      secureConnecting: false,
      _SNICallback: null,
      servername: '[api.procore.com](http://api.procore.com)',
      alpnProtocol: false,
      authorized: true,
      authorizationError: null,
      encrypted: true,
      _events: [Object: null prototype],
      _eventsCount: 10,
      connecting: false,
      _hadError: false,
      _parent: null,
      _host: '[api.procore.com](http://api.procore.com)',
      _readableState: [ReadableState],
      _maxListeners: undefined,
      _writableState: [WritableState],
      allowHalfOpen: false,
      _sockname: null,
      _pendingData: null,
      _pendingEncoding: '',
      server: undefined,
      _server: null,
      ssl: [TLSWrap],
      _requestCert: true,
      _rejectUnauthorized: true,
      parser: null,
      _httpMessage: [Circular **1],
      [Symbol(res)]: [TLSWrap],
      [Symbol(verified)]: true,
      [Symbol(pendingSession)]: null,
      [Symbol(async_id_symbol)]: 7,
      [Symbol(kHandle)]: [TLSWrap],
      [Symbol(kSetNoDelay)]: false,
      [Symbol(lastWriteQueueSize)]: 0,
      [Symbol(timeout)]: null,
      [Symbol(kBuffer)]: null,
      [Symbol(kBufferCb)]: null,
      [Symbol(kBufferGen)]: null,
      [Symbol(kCapture)]: false,
      [Symbol(kBytesRead)]: 0,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(connect-options)]: [Object],
      [Symbol(RequestTimeout)]: undefined
    },
    _header: 'GET /rest/v1.0/projects/448081/users/945366 HTTP/1.1\r\n' +
      'Accept: application/json, text/plain, **/**\r\n' +
      'Authorization: Bearer eyJh...c7OX\r\n' +
      'Content-Type: application/json\r\n' +
      'User-Agent: axios/0.21.4\r\n' +
      'Host: [api.procore.com](http://api.procore.com)\r\n' +
      'Connection: close\r\n' +
      '\r\n',
    _keepAliveTimeout: 0,
    _onPendingData: [Function: noopPendingOutput],
    agent: Agent {
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      defaultPort: 443,
      protocol: 'https:',
      options: [Object],
      requests: {},
      sockets: [Object],
      freeSockets: {},
      keepAliveMsecs: 1000,
      keepAlive: false,
      maxSockets: Infinity,
      maxFreeSockets: 256,
      scheduling: 'lifo',
      maxTotalSockets: Infinity,
      totalSocketCount: 1,
      maxCachedSessions: 100,
      _sessionCache: [Object],
      [Symbol(kCapture)]: false
    },
    socketPath: undefined,
    method: 'GET',
    maxHeaderSize: undefined,
    insecureHTTPParser: undefined,
    path: '/rest/v1.0/projects/448081/users/945366',
    _ended: true,
    res: IncomingMessage {
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      socket: [TLSSocket],
      httpVersionMajor: 1,
      httpVersionMinor: 1,
      httpVersion: '1.1',
      complete: true,
      headers: [Object],
      rawHeaders: [Array],
      trailers: {},
      rawTrailers: [],
      aborted: false,
      upgrade: false,
      url: '',
      method: null,
      statusCode: 403,
      statusMessage: 'Forbidden',
      client: [TLSSocket],
      _consuming: true,
      _dumped: false,
      req: [Circular **1],
      responseUrl: 'https://api.procore.com/rest/v1.0/projects/448081/users/945366',
      redirects: [],
      [Symbol(kCapture)]: false,
      [Symbol(RequestTimeout)]: undefined
    },
    aborted: false,
    timeoutCb: null,
    upgradeOrConnect: false,
    parser: null,
    maxHeadersCount: null,
    reusedSocket: false,
    host: '[api.procore.com](http://api.procore.com)',
    protocol: 'https:',
    _redirectable: Writable {
      _writableState: [WritableState],
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      _options: [Object],
      _ended: true,
      _ending: true,
      _redirectCount: 0,
      _redirects: [],
      _requestBodyLength: 0,
      _requestBodyBuffers: [],
      _onNativeResponse: [Function (anonymous)],
      _currentRequest: [Circular **1],
      _currentUrl: 'https://api.procore.com/rest/v1.0/projects/448081/users/945366',
      [Symbol(kCapture)]: false
    },
    [Symbol(kCapture)]: false,
    [Symbol(kNeedDrain)]: false,
    [Symbol(corked)]: 0,
    [Symbol(kOutHeaders)]: [Object: null prototype] {
      accept: [Array],
      authorization: [Array],
      'content-type': [Array],
      'user-agent': [Array],
      host: [Array]
    }
  },
  response: {
    status: 403,
    statusText: 'Forbidden',
    headers: {
      date: 'Wed, 15 Sep 2021 18:02:40 GMT',
      'content-type': 'application/json; charset=utf-8',
      'transfer-encoding': 'chunked',
      connection: 'close',
      'set-cookie': [Array],
      'x-frame-options': 'SAMEORIGIN',
      'x-xss-protection': '1; mode=block',
      'x-content-type-options': 'nosniff',
      'x-rate-limit-limit': '3600',
      'x-rate-limit-remaining': '3599',
      'x-rate-limit-reset': '1631732560',
      'cache-control': 'no-cache',
      'x-request-id': '78bb47a3-adfa-401c-8717-4f15d2d07952',
      'x-runtime': '0.170700',
      'strict-transport-security': 'max-age=15552000; includeSubDomains',
      'x-envoy-upstream-service-time': '178',
      server: 'envoy'
    },
    config: {
      url: 'https://api.procore.com/rest/v1.0/projects/448081/users/945366',
      method: 'get',
      headers: [Object],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      adapter: [Function: httpAdapter],
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: -1,
      maxBodyLength: -1,
      validateStatus: [Function: validateStatus],
      transitional: [Object],
      data: undefined
    },
    request: <ref **1> ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 7,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: false,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: false,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      _contentLength: 0,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      socket: [TLSSocket],
      _header: 'GET /rest/v1.0/projects/448081/users/945366 HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, **/*\r\n' +
        'Authorization: Bearer eyJh...qc7OX\r\n' +
        'Content-Type: application/json\r\n' +
        'User-Agent: axios/0.21.4\r\n' +
        'Host: [api.procore.com](http://api.procore.com)\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: noopPendingOutput],
      agent: [Agent],
      socketPath: undefined,
      method: 'GET',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      path: '/rest/v1.0/projects/448081/users/945366',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: '[api.procore.com](http://api.procore.com)',
      protocol: 'https:',
      _redirectable: [Writable],
      [Symbol(kCapture)]: false,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype]
    },
    data: {
      errors: 'You do not have sufficient access to complete that action.'
    }
  },
  isAxiosError: true,
  toJSON: [Function: toJSON]
}

Neil Lovelace : Invite sent. Here’s the link HERE

Neil Lovelace : Additionally, I’m getting a 403 response when my client credentials have full admin permissions in the platform that I’m calling.

Danny Roosevelt : Btw I see we have Procore as in integrated app, which might save you some time and simplify some of your code steps. We don’t have any actions built out other than the auth step, but you can write and submit your own for everyone to use if you’re interested!

Danny Roosevelt : Here is more info on using auth in your code steps: Auth in code steps

Danny Roosevelt : And here’s more info on building out additional component actions: Contributing to the Pipedream Registry

Danny Roosevelt : Okay I see what you’re referencing and I think console.log() might be truncating some of their response data. I’d recommend giving something like this a shot:

const procore_token = params.procore_token;

const config = {
  method: "get",
  url: `${process.env.procore_BaseURL}/${process.env.procore_APIVer}/projects/${event.body.project_id}/users/945366`,
  headers: {
    'Authorization': `Bearer ${procore_token}`,
    'Content-Type': 'application/json'
  }
};
return await require("@pipedream/platform").axios(this, config)

Giao Phan : Looks like you are console.log ing the object. console.log will not descend beyond a certain depth. You can either use console.dir(data, {depth: null}) or export it this.data = data and our json inspector to explore the object.

Danny Roosevelt : This should help in 2 ways:
• Return the correct data when the request succeeds
• Raise a well-formatted error when it fails (@pipedream/platform version of axios helps do this)

Danny Roosevelt : Hah, thank you Giao :pray::skin-tone-2:

Neil Lovelace : @UMZ235TK6 I appreciate your help.
Unfortunately, even after implementing your suggestions, I’m not receiving the full payload.

Neil Lovelace : Can I choose the version of axios in Pipedream, like I can using VSCode?

Dylan Sather (Pipedream) : can you clarify what payload you’re expecting, and what specifically is missing / incorrect about this payload?

Unfortunately you can’t set the version of axios within workflows, but we plan to support that in the future and we’re tracking here. Do you have reason to suspect that the version we’re using would be affecting the payload?

Neil Lovelace : - After spending an hour on the phone with Procore, it was found that something was amiss with my clientID and/or clientSecret and therefore my token was malformed. Once we figured that out, everything started working as expected.