Queues Compatible with Pipedream for Text Message Handling

Help
1

This topic was automatically generated from Slack. You can find the original thread here.

Are there any queues that work with pipedream, e.g. I want to put a text message on a queue and have someone pull it off the queue. Do I need to do that outside of pipedream?

2

Can you describe the mechanics of “someone else pulling off the queue” more? Meaning another system or script? Or a person interacting with a webpage?

3

Another system. Not a human. We’d like to use a Q like SQS, RabbitMQ or MuleSoft. The idea is that Pipedream would add the result (some variable value) of the previous step to the queue. The 3rd party’s system would then get the next message off the Q when it is ready. The 3rd party retrieval time is variable and not suitable for synchronous operation. e.g. they could get the next message 10 minutes later.

4

Thanks for clarifying, just wanted to make sure.

You can use any of those queue systems you listed with Pipedream.

I prefer the queuing systems that rely on HTTP and HTTP callbacks like SQS, Inngest or QStash. Since you can then configure an Pipedream workflow endpoint to also to consume events in addition to producing them:

5

With QStash specifically, we have a Publish Endpoint Message and Publish Topic pre-built actions you can use to quickly emit messages to a queue:

6

Thanks Pierce! I will look at QStash. We might be required to use one of the other Q’s that don’t have HTTP callbacks. If we must do that, how can we use a Q with Pipedream for at least sending queued messages for a particular topic?

7

As long as the Queue has an interface you can publish messages to, then you can use Pipedream workflows with Node.js or Python code steps to publish messages.

I would recommend choosing a Queue system that has a well maintained Node.js or Python package that can emit events. Then you simply import that package into your Pipedream workflow and publish messages to the queue.

8

Let’s see if Pi knows about queue packages. I haven’t tested this before.

9

can you show me how to queue a message to AWS SQS in a Node.js code step?

10

To queue a message to AWS SQS in a Node.js code step, you can use the @pipedream/platform package and the SQS sendMessage method. Here’s an example code snippet: import { axios } from "@pipedream/platform"; export default defineComponent({ async run({ steps, $ }) { const queueUrl = "YOUR_QUEUE_URL"; const message = "YOUR_MESSAGE"; const response = await axios($, { method: "POST", url: queueUrl, data: { MessageBody: message, }, }); return response.data; }, }); Replace "YOUR_QUEUE_URL" with the URL of your SQS queue and "YOUR_MESSAGE" with the message you want to send. This code will send the message to the specified SQS queue. Remember to connect your AWS account to Pipedream and configure the necessary AWS credentials for the code to work properly.

11

that is close, but can you use the official AWS Node.js package instead? And also use the authentication from the AWS app prop?

12

Yes, you can use the official AWS Node.js package in a Pipedream workflow. To do this, you would need to import the AWS SDK package and use the authentication credentials from the AWS app prop. Here’s an example of how you can achieve this: import AWS from "aws-sdk"; export default defineComponent({ props: { aws: { type: "app", app: "aws", }, }, async run({ steps, $ }) { const { accessKeyId, secretAccessKey, region } = this.aws.$auth; // Configure AWS SDK with authentication credentials AWS.config.update({ accessKeyId, secretAccessKey, region, }); // Use AWS SDK to interact with AWS services const sqs = new AWS.SQS(); const queueUrl = "YOUR_QUEUE_URL"; const message = "YOUR_MESSAGE"; const params = { MessageBody: message, QueueUrl: queueUrl, }; const response = await sqs.sendMessage(params).promise(); return response; }, }); Replace "YOUR_QUEUE_URL" with the URL of your SQS queue and "YOUR_MESSAGE" with the message you want to send. This code uses the AWS SDK to send a message to the specified SQS queue using the authentication credentials from the AWS app prop. Remember to connect your AWS account to Pipedream and configure the necessary AWS credentials for the code to work properly.

13

Understood about Node to publish. Not sure how this will be in the Q system we’ll be forced to use (TBD). I will have to find what kind of interfaces it supports. Node.js would be the easiest for us.

14

There are no restrictions on outgoing requests, so in theory as long as the queuing system accepts network requests, then you can use the corresponding Node.js package or craft a fetch/axios call to enqueue a message manually.

15

That’s what I figured. Thanks !