13.4. Real-Time Analytics Platforms

Real-time analytics platforms sit at the hardest intersection in data engineering: sub-second query latency over data that is still arriving. Meeting that demand requires specialized joins across streams, purpose-built OLAP engines, and an architectural evolution away from the batch/speed split toward decentralized data ownership. This chapter covers stream-join strategies, low-latency OLAP engines like Pinot and Druid, the progression from Lambda to Data Mesh, and the feature-store and model-registry infrastructure that serves machine learning at scale.

Topics Covered

• 13.4.1. Stream Join Challenges: Temporal Join Strategies: Covers the challenges of joining streams and the temporal join strategies that bound state.
• 13.4.2. Apache Pinot / Druid: OLAP at Low Latency: Compares Apache Pinot and Druid, the OLAP engines built for sub-second queries over fresh data.
• 13.4.3. From Lambda to Data Mesh: The Evolution of Data Architecture: Traces the evolution of data architecture from Lambda’s dual layers toward decentralized ownership.
• 13.4.4. Data Mesh Principles: Domain-Oriented Data Ownership: Covers Data Mesh principles: domain-oriented ownership treating data as a product.
• 13.4.5. ML Serving at Scale: Feature Store and Model Registry: Explains serving machine learning at scale with feature stores and model registries.