LinkedIn and Apache Kafka: 7 trillion messages a day

Apache Kafka, an open-source platform for real-time data stream processing, has become an essential tool for companies operating at large scale. LinkedIn, which originally developed Kafka, is today one of its most advanced users, handling over 7 trillion messages a day. To support this volume of data, LinkedIn has built a tailor-made Kafka ecosystem, facing scalability challenges that are rarely encountered elsewhere and actively contributing the solutions it finds back to the open-source community.

Kafka was born in 2010 within LinkedIn to manage real-time data flows generated by users. The system had such success that it was open-sourced in 2011 and is today an industry standard adopted by thousands of companies worldwide. Despite its widespread adoption, LinkedIn continues to be one of the most advanced users, with an infrastructure that includes dozens upon dozens of Kafka clusters, more than 4,000 brokers and 7 million partitions, an operational scale that poses engineering challenges of the highest order.

Kafka and the messaging system

To understand why Kafka is so central to LinkedIn’s infrastructure, it helps to grasp how the system works at a conceptual level. Kafka is built on the idea of the distributed commit log: every message produced by an application is written in an ordered and immutable fashion to a structure called a topic, divided into partitions that can be spread across multiple machines. This design allows many producers to write data in parallel and many consumers to read it independently, without interfering with one another. Consumer groups allow multiple instances of an application to process the same stream by dividing the load among themselves, while ordering within each individual partition is guaranteed, offering the predictability required for use cases such as recording user activity or synchronizing between microservices. The result is a system capable of handling high traffic peaks with very low latency, equally suited to real-time event streaming and to the historical analysis of data archived in the log.

LinkedIn’s Kafka ecosystem

LinkedIn’s Kafka clusters are designed to handle billions of events a day, ensuring that every piece of information is processed in real time. Internal applications rely on Kafka to record user activities, transmit messages between different services, and collect system metrics, allowing the company to continuously analyze and optimize its operations. To facilitate communication between the various components of the system, LinkedIn uses tools that improve data integration and management: the REST Proxy allows even non-Java-based services to interact with Kafka, while a schema management system ensures that sent and received data are always consistent. To optimize data flow between the company’s various operational centers, LinkedIn uses a replication technology that ensures information is always available and up to date wherever needed, and an internal software monitors resource usage and automatically redistributes the load when necessary.

Challenges and innovations for scalability

Managing Kafka at this scale presents concrete technical challenges that go well beyond those faced by most organizations. One of the most complex concerns ordering guarantees: when a stream of events is distributed across thousands of partitions, ensuring that correlated messages are processed in the correct sequence requires careful design of partitioning schemes and, in some cases, dedicated application-level logic. LinkedIn has also had to contend with the rebalancing problem, meaning the redistribution of partitions among brokers during maintenance operations or following failures: inside very large clusters, these operations can generate significant latency spikes that impact production applications. To contain these effects, the engineering team developed internal automation tools capable of scheduling rebalancing during low-traffic windows and monitoring the state of every partition in real time. Many of these innovations, such as improvements to partition management and rebalancing throttling mechanisms, were subsequently contributed back to the Apache community, allowing other companies to benefit from them.

The use of Apache Kafka by LinkedIn represents an example of how an open-source system can be adapted and scaled to support global operations of the highest level. Thanks to a customized ecosystem and careful management of operational challenges, LinkedIn is able to process enormous amounts of data every day, while guaranteeing high performance and contributing to the growth of the Kafka platform for the entire technology community.