What do you want to automate
with Fivetran and Kafka?
Prompt, edit and deploy AI agents that connect to Fivetran, Kafka and 2,800+ other apps in seconds.
Trusted by 1,000,000+ developers from startups to Fortune 500 companies
Start for free — no credit card required
Watch a demo
Integrate the Fivetran API with the Kafka API
Setup the Fivetran API trigger to run a workflow which integrates with the Kafka API. Pipedream's integration platform allows you to integrate Fivetran and Kafka remarkably fast. Free for developers.
Overview of Fivetran#
The Fivetran API enables automated, effortless replication of data from various sources into a cloud warehouse. By leveraging Fivetran in Pipedream, you can programmatically manage your Fivetran connectors, set up and control data pipelines, and trigger data syncs. Pipedream's serverless platform allows for the creation of workflows that can respond to HTTP requests, process events on a schedule, and interact with numerous other services to create complex automation solutions.
Connect Fivetran#
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
import { axios } from "@pipedream/platform"
export default defineComponent({
props: {
fivetran: {
type: "app",
app: "fivetran",
}
},
async run({steps, $}) {
return await axios($, {
url: `https://api.fivetran.com/v1/users`,
headers: {
"Content-Type": `application/json`,
},
auth: {
username: `${this.fivetran.$auth.api_key}`,
password: `${this.fivetran.$auth.api_secret}`,
},
})
},
})
Connect Kafka#
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
import { Kafka } from "kafkajs"
export default defineComponent({
props: {
kafka: {
type: "app",
app: "kafka",
}
},
async run({steps, $}) {
const kafka = new Kafka({
brokers: [`${this.kafka.$auth.host}:${this.kafka.$auth.port}`],
});
const consumer = kafka.consumer({ groupId: 'TestGroup' });
await consumer.connect()
await consumer.subscribe({ topic: 'SampleTopic', fromBeginning: true });
let consumedMessage = "";
const eachMessage = async function({ topic, partition, message }){
consumedMessage = message.value.toString();
return consumedMessage;
};
await consumer.run({
eachMessage,
});
const producer = kafka.producer();
await producer.connect()
await producer.send({
topic: 'SampleTopic',
messages: [
{ value: 'Welcome KafkaJS + Pipedream users! '+ new Date().toISOString() },
],
});
await producer.disconnect();
const data = await consumer.describeGroup();
return { consumedMessage, groupDescription: data };
},
})
Start for free — no credit card required
Watch a demo
Trusted by 1,000,000+ developers from startups to Fortune 500 companies