Design idea for custom OAuth services or Secret

While researching how to use AWS tools to avoid creating my own OAuth client in an AWS Lambda, I discovered that AWS Secrets Manager can be used as an OAuth client by Using a Lambda Function to Rotate Secrets, with a blog post tutorial here: How to rotate your Twitter API key and bearer token automatically with AWS Secrets Manager | AWS Security Blog

I know Pipedream has an open issue to add support for Pipedream users to submit their own OAuth clients for specific services, here: I want to create my own OAuth app, while still having Pipedream manage the authorization / token refresh process · Issue #163 · PipedreamHQ/pipedream (github.com)

Though I don’t need a custom auth provider in Pipedream right now, I think the design decisions taken by AWS Secrets Manager could be used by Pipedream.

As a first stab at a design for this, Pipedream could do this by adding:

  • A data store for credentials with standard fields “access_token”, “token_type”, and perhaps “refresh_token” and “authorization_token” to suggest best practices.
  • A new option in “Accounts > Connect an App” named “Custom OAuth Provider”.
  • Selecting “Custom OAuth Provider” would have you populate parameters used by an OAuth flow process, like “access_token_url”, “scopes”, and perhaps “authorize_url”, “client_id”, and “client_secret”.
  • Popping open the code block below the parameters, like in Workflow Actions, would allow you to view the JavaScript which implements it. and customize it.
  • The JavaScript behind those parameters appropriately handles each step of a secret token rotation:
    - CreateSecret: Invalidates existing bearer token, if the auth provider supports external invalidation, and retrieves a new one. If a secret version with PENDING stage exists, update it with the newly retrieved bearer token and if the PENDING stage does not exist, create a new version of the secret with that stage label.
    - SetSecret: Checks whether the bearer token is the same as the one in the version with PENDING stage.
    - TestSecret: Try to use the bearer token in the secret version with PENDING stage.
    - FinishSecret: Finish the rotation by moving the secret from the PENDING stage to the CURRENT stage.

Open questions:

  • Should these custom auth providers be able to be published to a shared repository, to be shared with your team, your workflows, or the wider Pipedream userbase?
  • How should this relate to defining Global Variables, such as “Settings > Environment Variables”, or Workflow-scoped Variables to complement component-specific parameter values?

@chexxor I love this approach and your notes. This shares similarities to the method we use to ship integrations internally.

When we implement custom auth, we do plan to optionally allow users to share the generic configuration for an app (either a key-based auth configuration, or OAuth) so that users can add integrations to apps that make sense for other users. e.g. if we didn’t have a Slack integration, anyone should be able to conceivably add that integration, use it with their own custom client, but share the general config for anyone to use. It’s likely these configurations will be serialized to e.g. JSON so that they can be stored on GitHub.

I’ll add a link to this topic in the GitHub issue, so we have this feedback recorded there.

You may also consider how others are handling this: