Building Event-Driven Microservices with Apache Kafka and Node.js

Event-driven architecture decouples services, enabling independent scaling and deployment. Apache Kafka provides the backbone for reliable event streaming at any scale.

Why Event-Driven Architecture?

Traditional request-response communication creates tight coupling. If Service B is down, Service A fails. Event-driven systems are resilient—events are stored until consumers are ready.

Kafka Core Concepts

Topics: Named feeds of messages (like database tables).

Partitions: Enable parallelism and ordering within partition.

Consumer Groups: Allow horizontal scaling of consumers.

Offsets: Track consumption progress for reliability.

Setting Up Kafka with Docker

# docker-compose.yml
services:
  kafka:
    image: confluentinc/cp-kafka:latest
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
    ports:
      - "9092:9092"

Producer Implementation (Node.js)

import { Kafka, CompressionTypes } from "kafkajs";

const kafka = new Kafka({
  clientId: "order-service",
  brokers: ["localhost:9092"],
});

const producer = kafka.producer();

async function publishOrderCreated(order: Order) {
  await producer.send({
    topic: "orders",
    messages: [{
      key: order.id,
      value: JSON.stringify({
        eventType: "ORDER_CREATED",
        data: order,
        timestamp: Date.now(),
      }),
      headers: {
        "correlation-id": order.correlationId,
      },
    }],
    compression: CompressionTypes.GZIP,
  });
}

Consumer Implementation

const consumer = kafka.consumer({ groupId: "inventory-service" });

async function startConsumer() {
  await consumer.connect();
  await consumer.subscribe({ topic: "orders", fromBeginning: false });
  
  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      const event = JSON.parse(message.value!.toString());
      
      switch (event.eventType) {
        case "ORDER_CREATED":
          await reserveInventory(event.data);
          break;
        case "ORDER_CANCELLED":
          await releaseInventory(event.data);
          break;
      }
    },
  });
}

Event Schema Design

interface DomainEvent<T> {
  eventId: string;
  eventType: string;
  aggregateId: string;
  aggregateType: string;
  data: T;
  metadata: {
    correlationId: string;
    causationId: string;
    timestamp: number;
    version: number;
  };
}

Handling Failures

Implement dead letter queues for poison messages:

async function processWithRetry(message: Message, retries = 3) {
  try {
    await processMessage(message);
  } catch (error) {
    if (retries > 0) {
      await processWithRetry(message, retries - 1);
    } else {
      await publishToDeadLetter(message, error);
    }
  }
}

Exactly-Once Semantics

Kafka supports exactly-once semantics with idempotent producers and transactional messaging. Enable with: `enable.idempotence=true` and `transactional.id=your-service-id`.

Monitoring Kafka

Monitor: consumer lag (are consumers keeping up?), broker health, partition distribution, and throughput metrics. Use Kafka UI or Grafana dashboards.

NyxaLabs Event-Driven Architecture Services

We design and implement event-driven systems that scale to millions of events per second. From Kafka cluster setup to microservices implementation, we deliver reliable architectures. Contact us for architecture consultation.