Gleam Lab · Blog Archive

Blog Page 26

Technical exploration and engineering notes, 655 articles in total.

Gleam Lab technical blog cover: AI Engineering, Java backend, and long-form writing
All Articles Java243Backend50Microservices10AI Engineering86LLM35Big Data271Data Engineering57Kubernetes / Cloud Native3Real-time Voice1Robotics40Personal Growth29 More Tags →
Tutorial Series 6 min read Big Data Engineering

Apache Kylin Cube Practice: Hive Load & Pre-computation Acceleration

Apache Kylin is an open-source distributed analysis engine, focused on providing real-time OLAP (Online Analytical Processing) capabilities for big data.

big-datakylin
Tutorial Series 9 min read Big Data Engineering

Apache Kylin Cube Practice: From Modeling to Build and Query

Scenario: Using e-commerce sales fact table, pre-compute aggregation queries accelerated by "date" dimension on Kylin.

big-datakylin
Tutorial Series 3 min read Big Data Engineering

From MapReduce to Spark: Big Data Computing Evolution

Systematic overview of big data processing engine evolution from MapReduce to Spark to Flink, analyzing Spark in-memory computing model, unified ecosystem and core compon...

big-datasparkdistributed-system +2
Tutorial Series 12 min read Big Data Engineering

Apache Kylin Comprehensive Guide: MOLAP Architecture, Hive Integration

Background, evolution and engineering practice of Apache Kylin, focusing on MOLAP solution implementation path for massive data analysis.

big-datakylin
Tutorial Series 4 min read Big Data Engineering

Big Data 158 - Apache Kylin 3.1.1 Deployment on Hadoop, Hive and HBase

Complete deployment record of Apache Kylin 3.1.1 on Hadoop 2.9.2, Hive 2.3.9, HBase 1.3.1, Spark 2.4.5 (without-hadoop.

big-datakylin
Tutorial Series 5 min read Big Data Engineering

Kafka Storage Mechanism: Log Segmentation & Retention

This is article 65 in the Big Data series, deeply analyzing Kafka's log storage mechanism.

big-datakafkamessaging +1
Tutorial Series 3 min read Big Data Engineering

Kafka High Performance: Zero-Copy, mmap & Sequential Write

This is article 66 in the Big Data series, deeply analyzing Kafka's underlying I/O optimization technologies achieving extremely high throughput.

big-datakafkamessaging +1
Tutorial Series 4 min read Big Data Engineering

Kafka Replica Mechanism: ISR & Leader Election

Deep dive into Kafka replica mechanism, including ISR sync node set maintenance, Leader election process, and unclean election trade-offs between consistency and availabi...

big-datakafkamessaging +1
Tutorial Series 4 min read Big Data Engineering

Kafka Exactly-Once: Idempotence & Transactions

Systematic explanation of how Kafka achieves Exactly-Once semantics through idempotent producers and transactions, covering PID/sequence number principle...

big-datakafkamessaging +1
Tutorial Series 9 min read Big Data Engineering

Big Data 155 - Apache Druid Storage & Query Architecture: Segment, Chunk, Roll-up & Bitmap Indexes

Apache Druid data storage and high-performance query path: from DataSource/Chunk/Segment layering, to columnar storage, Roll-up pre-aggregation, Bitmap.

big-datadruid
Tutorial Series 5 min read Big Data Engineering

Big Data 156 - Apache Druid + Kafka Real-time Analysis: JSON Flattening, Ingestion & SQL Metrics

Scala Kafka Producer writes order/click data to Kafka Topic (example topic: druid2), continuous ingestion in Druid through Kafka Indexing Service.

big-datadruidkafka
Tutorial Series 2 min read Big Data Engineering

Big Data 153 - Apache Druid Real-time Kafka Ingestion: Complete Practice from Ingestion to Query

Complete practice of Apache Druid real-time Kafka ingestion, using network traffic JSON as example, completing data ingestion through Druid console's Streaming/Kafka wiza...

big-datadruidkafka
Tutorial Series 8 min read Big Data Engineering

Apache Druid Architecture & Component Responsibilities: Coordinator/Overlord

Apache Druid component responsibilities and deployment points from 0.13.0 to current (2025): Coordinator manages Historical node Segment.

big-datadruid
Tutorial Series 6 min read Big Data Engineering

Apache Druid Cluster Deployment [Part 1]: MySQL Metadata Store

Scenario: 2C4G/2C2G three-node mixed deployment, Druid 30.0.0, Kafka/HDFS/MySQL collaboration. Conclusion: Can run on low config, but core is DirectMemory and processing.

big-datadruid
Tutorial Series 2 min read Big Data Engineering

Apache Druid Cluster Mode [Part 2]: Low-Memory Cluster Practice

Low-memory cluster practice for Apache Druid 30.0.0 on three nodes: provides JVM parameters and runtime.

big-datadruid
Tutorial Series 4 min read Big Data Engineering

Kafka Topic, Partition & Consumer: Rebalance Optimization

Deep dive into Kafka Topic, Partition, Consumer Group core mechanisms, covering custom deserialization, offset management and rebalance optimization configuration.

big-datakafkamessaging +1
Tutorial Series 4 min read Big Data Engineering

Kafka Topic Management: Commands & Java API

Comprehensive introduction to Kafka Topic operations, including kafka-topics.sh commands, replica assignment strategy principles, and KafkaAdminClient Java API core usage.

big-datakafkamessaging +2
Tutorial Series 5 min read Big Data Engineering

Apache Druid Real-time OLAP Architecture & Selection Points

Apache Druid real-time OLAP practice: suitable for event detail with time as primary key, sub-second aggregation and high-concurrency self-service analysis.

big-datadruidolap
Tutorial Series 7 min read Big Data Engineering

Big Data 150 - Apache Druid Single-Machine Deployment: Architecture Overview and Startup

Scenario: Quickly experience Apache Druid 30.0.0 locally/single-machine, verify real-time and historical queries and console access.

big-datadruiddeployment
Tutorial Series 6 min read Big Data Engineering

Big Data 148 - Flink Write to Kudu: Custom Sink Full Practice

Complete runnable example for Kudu, based on Flink 1.11.1 (Scala 2.12)/Java 11 and kudu-client 1.17.0 (2025 test).

big-dataflinkkudu