Revisiting the Canvas 401 issue

Bugs
1

Hi team,

About a year ago I reported an issue with Canvas (Instructure) OAuth2 authentication causing intermittent 401 errors. See issue on GitHub.

Basically we get the following intermittent error:

Error — Request failed with status code 401

I replay the workflow in question and all is fine.

I don’t think I have ever seen it happen twice in a row. I am guessing it’s to do with the 1 hour lifespan on the access tokens:

According to the docs:

Access tokens have a 1 hour lifespan. When the refresh flow is taken, Canvas will update the access token to a new value, reset the expiration timer, and return the new access token as part of the response. When refreshing tokens the user will not be asked to authorize the application again.

The resolution last time was that there’s not much else you’re able to do, that it may be related to a nuance with the Canvas API.

I wasn’t too bothered by that as it was just one every couple of days or so. But the more we build with Pipedream the more frequently this error occurs and it’s getting a little more frustrating.

I was hoping we could take another look at this issue?

I have some workflows using the same API on the Zapier Developer Platform and have not had this issue happen at all, same with a custom solution we have running in PHP.

Is anyone able to help me explore this a little deeper? I am happy to go to Canvas tech support and seek assistance but I don’t have much technical data and can’t see the code used to authenticate it so it’s a little difficult for me to give them much more than “intermittent 401s.”

Would really love to see this one running a bit more smoothly!

Thanks for any help you can offer.

2

Thank you for raising this issue again @accellier,

As this issue is related to Authentication, maybe @sergioeliot, @pierce, @danny from Pipedream team can help to advice a way to solve it?

On top of my mind right now, I think we can catch this 401 error in Canvas component code, then retry it one to three time? I have created a ticket here: [IMPROVEMENT] Canvas actions - Retry requests when facing 401 error from Canvas · Issue #5511 · PipedreamHQ/pipedream · GitHub

3

Many thanks!

4

Hello @accellier,

We noticed that we don’t have Canvas action to improve. May I ask if you’re using code step to communicate with Canvas API?

image
image1468×732 51.9 KB

5

Yes we just use code steps.

The issue is with the Pipedream built in process for authenticating using OAuth2.

I can assist with any testing required.

Kind regards,
Paul

6

@accellier, could you try adding the code to retry the request if it failed with status 401? Also which API you’re using? Maybe you can create a ticket here to request the action?

7

Speaking out of my guts, I understand Pipedream components perform exponential backoff retrying failed request, before giving app. Perhaps the config on this strategy could be twaked so that it can retry taking the time enough for the access token to be refreshed properly instead of bailing out quickly.

8

Thanks to you both.

I guess the persistence and frequency of this issue lead me to see if there was a way to resolve it in the way the authentication works.

To be honest with you I was being lazy and not handling for this error on our side. So I have used the following approach. Do you think it will be okay?

    try {
      return await canvasRequest(requestData)
    } catch (error) {
        if (error.response.status === 401) {
          //try again if it's a 401...
          return await canvasRequest(requestData)
      } else {
          throw error
          }
    }

Is there perhaps a better way to handle it?

9

Hey @accellier

Yes I think there is likely a better way to do this, for example. I think you should wait certain interval before doing the retry. Pipedream components leverage the "async-retry" npm library, why don’t give a try?

You can take a look at the several Pipedream components, for instance, I myself was involved in the Mailchimp app.

Check how we wrap the request we want to be attempted, in this case ailchimp.lists.createListWebhook inside an “_withRetries”

async createWebhook(listId, config) {
  const mailchimp = this.api();
  const { id } = await this._withRetries(() =>
    mailchimp.lists.createListWebhook(listId, config));
  return id;
},

on the _withRetries function we do the retry logic:

async _withRetries(apiCall) {
  const retryOpts = {
    retries: 5,
    factor: 2,
  };
  return retry(async (bail) => {
    try {
      return await apiCall();
    } catch (err) {
      const { status = 500 } = err;
      if (!this._isRetriableStatusCode(status)) {
        bail(`
          Unexpected error (status code: ${status}):
          ${JSON.stringify(err.response)}
        `);
      }

      throw err;
    }
  }, retryOpts);
},

see how we define certain options on retryOpts to be passed over Retry from the retry-async package.

The “retry-async” package page at npm has an usage example too: async-retry - npm

the “Factor” option will help you to increase the time interval between attemps

  • factor: The exponential factor to use. Default is 2.
10

This looks a little out of my depth Sergio but I will give it a try!

I noticed this new “error reruns” feature roll out today: Settings

It hasn’t showed up in my account yet but perhaps this might be an easier option for now.

11

@accellier sorry to hear this issue is still plaguing you! I agree we should ideally be handling any refresh inconsistencies as a part of the auth process, so we’ll try to look into this again.

In the meantime though, there are a copule options to work around the 401s you’re seeing.

Like you saw, we recently rolled out auto-retry for the Advanced plan in workspaces (a new version of “orgs”). If you’re interested in checking it out, you can create a new workspace (top left of your Pipedream account), then you’ll have the option to enable auto-retry in any workflow after you upgrade to the Advanced plan.

Another approach you can try out via code (and available on all plans) is $.flow.rerun(): Pause, resume, and rerun a workflow

12

Thanks Danny! I’m the only Pipedream user in our org so we don’t have a need for workspaces.
$.flow.rerun() looks like a simple work-around in this case - thank you!