Integrate the Kafka API with the Python API

Setup the Kafka API trigger to run a workflow which integrates with the Python API. Pipedream's integration platform allows you to integrate Kafka and Python remarkably fast. Free for developers.

Start for free — no credit card required

Watch a demo

Trusted by 1,000,000+ developers from startups to Fortune 500 companies

Popular Kafka and Python Actions#

Run Python Code with the Python API

Write Python and use any of the 350k+ PyPi packages available. Refer to the Pipedream Python docs to learn more.

Try it

Connect Kafka#

import { Kafka } from "kafkajs"
export default defineComponent({
  props: {
    kafka: {
      type: "app",
      app: "kafka",
    }
  },
  async run({steps, $}) {

    const kafka = new Kafka({
      brokers: [`${this.kafka.$auth.host}:${this.kafka.$auth.port}`],
    });

    const consumer = kafka.consumer({ groupId: 'TestGroup' });
    await consumer.connect()
    await consumer.subscribe({ topic: 'SampleTopic', fromBeginning: true });
    let consumedMessage = "";
    const eachMessage = async function({ topic, partition, message }){
      consumedMessage = message.value.toString();
      return consumedMessage;
    };  

    await consumer.run({
      eachMessage,
    });

    const producer = kafka.producer();  
    await producer.connect()
    await producer.send({
      topic: 'SampleTopic',
      messages: [
        { value: 'Welcome KafkaJS + Pipedream users! '+ new Date().toISOString() },
      ],
    });
    await producer.disconnect();

    const data = await consumer.describeGroup();
    return { consumedMessage, groupDescription: data };
  },
})

Overview of Python#

Develop, run and deploy your Python code in Pipedream workflows. Integrate seamlessly between no-code steps, with connected accounts, or integrate Data Stores and manipulate files within a workflow.

This includes installing PyPI packages, within your code without having to manage a requirements.txt file or running pip.

Below is an example of using Python to access data from the trigger of the workflow, and sharing it with subsequent workflow steps:

Connect Python#

def handler(pd: "pipedream"):
  # Reference data from previous steps
  print(pd.steps["trigger"]["context"]["id"])
  # Return data for use in future steps
  return {"foo": {"test":True}}

Start for free — no credit card required

Watch a demo