Using Pipedream to verify my Workflow does what I expect?

1

The workflow I’m writing is starting to get relatively complex. For example, if PropA changed, do X, else if PropB changed, do X and Y, else if PropA and B changed, do Z. I would feel better if I could write a script which verifies the behavior of my workflow, to ensure it’s doing what I intend for it to do.

What do other people do to verify their workflow’s behavior/implementation?

At first, I started writing a stand-alone JavaScript app which does this for each case I want to verify:

  • Send 1 or more events to my workflow.
  • Query the data in the system I expect the workflow to change to ensure the end state matches my expectation.

After spending a few hours writing code to handle OAuth, I stepped back and reconsidered my decision. Pipedream has a pretty good OAuth client - would be nice to reuse that. Maybe I can write a Pipedream workflow for verifying my workflow?

While I might be able to shoe-horn a Pipedream workflow to do this, I’m pretty sure it would be more awkward than I would like. For example:

  • We can’t verify the messages sent by a workflow, or similarly, run a workflow in an effect-less “dry-run mode”. If we can’t verify the output of a workflow, we must verify the effects a workflow produces. This means querying each target system to see if workflow’s effects are as expected.

  • Pipedream doesn’t have a nice place for storing events in a structured or labelled way to allow a test case to use them:

    • I could make a new event source which has all of them in the Configuration, but, as I imagine the only easy way to make that work, it would emit all events for every case to verify all at once rather than the event(s) for a single case at a time. While this might be ok, I would rather run one case at a time.
    • Event Sources don’t support Attachments, like a Workflow does, so I can’t use that feature to hide away sample events. Also, Attachments are relatively opaque and can’t quickly be viewed or edited, which is not ideal.

  • My workflow uses a custom event source. There’s no clear way for my test script to execute my workflow directly:

    • Workflows only support a single event source (right now).
    • The REST API doesn’t provide secret functionality to simply execute a workflow with an arbitrary event.
    • I could edit the workflow I want to verify to change its event source to a new one which emits a set of events to test a certain case, but it seems like bad practice to modify the unit to verify in the process of verifying it.

After this initial investigation of the possibility of using Pipedream as a tool for verifying the handling of events, it seems that not only is it not easy, it seems impossible to implement basic best-practice code verification techniques.

What other verification technique could be used with Pipedream if not traditional test cases? I guess static analysis could be applicable, but that seems unlikely, as each connected app/system is a black box, which static analysis systems likely can’t tackle. However, static analysis might work if we forget querying the target system to see how it’s been changed, and instead verify the messages being sent to the target app/system.

2

@chexxor I appreciate all the detailed feedback here. We have a long way to go to provide all of the testing functionality you’d like to see, but this is on our radar. I’ll share a few resources and open issues related to some of your questions. On others, I have some follow-up questions for you.

Like event sources, Pipedream workflow primitives (like actions and code steps) are moving towards the component model. We’re actually just about to release component actions (see the beta docs here). We don’t have a standard testing framework in place for components today, but since they’re just Node modules, they can be unit tested locally. In certain cases (e.g. for components that expose an HTTP interface), you can technically deploy them, invoke them, and retrieve the emitted event data programmatically, so you could develop a script that runs a basic integration test on a component in that manner. We’re planning to invest in better tooling here in the future.

As we think about an API / CLI for workflows, we’ll need to think more about how tests for workflows work.

Workflows technically do support multiple triggers today via subscriptions. We’ve actually added this functionality to the UI in a new version of the workflow builder we’re working on, so expect better support for multiple triggers moving forward.

Some follow up questions for you:

  • How would you envision a “dry run” of a workflow would work? e.g. would you want to mock API calls yourself, or would you expect Pipedream to provide those mocks for each action as a part of the service?
  • That’s great feedback re: Attachments. Would you mind opening a feature request for what API you’d specifically like to see there?
1 Like
3

For the workflow I have been working on, the implementation of the primary action had so many nested conditionals that I decided having some test cases would be very important. I spent a day translating it to the language my team already uses, which already has support for defining and running test suites. (I used Salesforce.) Pipedream, then, will only need to translate from the event to the inputs of this custom code.

This path might be a good recommendation for other Pipedream users for the next few years - use Pipedream to move your data, and do any complicated processing you need to do in a language or platform which has a better testing framework and code analysis than a data moving tool like Pipedream provides.

To answer your question about how a dry-run would work, I think that mocks would be fine. This is the strategy used by Salesforce for its traditional programming language and its new visual programming language inspired by the dataflow paradigm.

Example JavaScript language for Pipedream testers:

let workflowToTest = 'SomePipedreamWorkflowName';
let Workflow = require('@pipedream/TestFramework/Workflow');
let inputVariables = {};
let myWorkflow = Workflow.create(workflowToTest, inputVariables);

let Attachments = require('@pipedream/AttachmentsAPI');
myWorkflow.setHTTPMock((httpRequest) => {
  // https://docs.github.com/en/rest/reference/users#get-a-user
  if (httpRequest.path.matches('/users/{username}')) {
    return Attachments.folder(workflowToTest).find('fakeGitHubUserResponse');
  }
  // https://docs.github.com/en/rest/reference/orgs#get-an-organization
  if (httpRequest.path.matches('/orgs/{org}')) {
    return Attachments.folder(workflowToTest).find('fakeGitHubOrgResponse');
  }
});
myWorkflow.run();

let Test = require('@pipedream/TestFramework');
Test.assertEquals('abc123', myWorkflow.steps.getGitHubUserProfile.result.id);
Test.assertEquals('abc123', myWorkflow.steps.getGitHubUserProfile.result.id);
1 Like