Kafka streaming data pipeline diagram

A streaming pipeline diagram shows where events start, which job reads them in real time, and where they land for analytics. Wavelength drew this Kafka pipeline from a three sentence description.

What was said

The checkout service publishes order events to Kafka. A Flink job reads the stream and flags orders that look like fraud. Flagged orders go to the fraud review team. All events also land in S3 through Kafka Connect, and Databricks processes them into Delta tables for the analytics team.

The diagram

Kafka, Flink, Kafka Connect, and Databricks are hexagons. S3 and the Delta tables are cylinders. The fraud review team and the analytics team are rounded orange boxes.

Open this diagram in Wavelength

Opens an editable copy you can change by voice or by hand. Free account, no card.

What is in this flow

Systems: Kafka, Flink Job, Kafka Connect and Databricks.
Data stores: S3 and Delta Tables.
People and teams: Fraud Review Team and Analytics Team.
Size: 9 steps and 8 connections.

Mermaid code

Copy this into any Markdown file, GitHub README, or Mermaid Live to render the same flow.

flowchart TD
  n1(["Checkout Service"])
  n2{{"Kafka"}}
  n3{{"Flink Job"}}
  n4("Fraud Review Team")
  n5[("S3"])
  n6{{"Kafka Connect"}}
  n7{{"Databricks"}}
  n8[("Delta Tables"])
  n9("Analytics Team")
  n1 -->|publishes order events| n2
  n2 -->|reads stream| n3
  n3 -->|flags orders| n4
  n2 -->|through Kafka Connect| n6
  n6 -->|lands events| n5
  n5 -->|processes into Delta tables| n7
  n7 -->|stores| n8
  n8 -->|used by analytics team| n9

Read this pipeline

Two paths leave Kafka. The fast path goes through the Flink job, which flags suspect orders for the fraud review team while the order is still fresh. The slow path goes through Kafka Connect to S3, and Databricks turns the raw events into Delta tables for the analytics team.

Questions that a data engineer asks next: