Gleam Lab · Blog Archive
Blog Page 27
Technical exploration and engineering notes, 655 articles in total.
Kafka Producer Interceptor & Interceptor Chain
Introduction to Kafka 0.10 Producer interceptor mechanism, covering onSend and onAcknowledgement interception points, interceptor chain execution order and error isolatio...
Big Data 60 - Kafka Consumer: Consumption Flow, Heartbeat and Parameter Tuning
Detailed explanation of Kafka Consumer Group consumption model, partition assignment strategy, heartbeat keep-alive mechanism, and tuning practices for key parameters lik...
Apache Kudu Docker Quick Deployment: 3 Master/5 TServer Pattern
Apache Kudu Docker Compose quick deployment solution on Ubuntu 22.04 cloud host, covering Kudu Master and Tablet Server components.
Big Data 147 - Java Access Apache Kudu: From Table Creation to CRUD
Java client (kudu-client 1.4.0) connects to Apache Kudu with multiple Masters (example ports 7051/7151/7251), completes full process of table creation.
Apache Kudu: Real-time Write + OLAP Architecture, Performance
Apache Kudu is an open-source storage engine developed by Cloudera and contributed to Apache Software Foundation.
Apache Kudu Architecture & Practice: RowSet, Partition & Raft Consensus
Apache Kudu's Master/TabletServer architecture, RowSet (MemRowSet/DiskRowSet) write/read path, MVCC, and Raft consensus role in replica and failover
ClickHouse MergeTree Partition/TTL, Materialized View, ALTER
ClickHouse is a columnar database for OLAP (Online Analytical Processing), favored in big data analysis for its high-speed data processing.
Kafka Producer Message Sending Flow & Core Parameters
Deep analysis of Kafka Producer initialization, message interception, serialization, partition routing, buffer batch sending, ACK confirmation and complete sending chain.
Kafka Serialization & Partitioning: Custom Implementation
Deep dive into Kafka message serialization and partition routing, including complete code for custom Serializer and Partitioner, mastering precise message routing and eff...
Big Data 141 - ClickHouse Replicas: ReplicatedMergeTree and ZooKeeper
ReplicatedMergeTree ZooKeeper: Implements communication between multiple instances.
ClickHouse Sharding × Replica × Distributed: ReplicatedMergeTree
Replica refers to storing the same data on different physical nodes in a distributed system. Its core idea is to improve system reliability through data redundancy.
Big Data 139 - ClickHouse MergeTree Best Practices: Replacing Deduplication, Summing Aggregation, Partition Design & Materialized View Alternatives
Scenario: Solve two common "quasi-real-time detail table" requirements: deduplication/update and key-based summing.
Big Data 140 - ClickHouse CollapsingMergeTree & External Data Sources
ClickHouse external data source engine guide: DDL templates, key parameters and read/write pipelines for ENGINE=HDFS, ENGINE=MySQL, ENGINE=Kafka, and distributed table co...
Big Data 137 - ClickHouse MergeTree Practical Guide
ClickHouse MergeTree key mechanisms: batch writes form parts, background merge (Compact/Wide two part forms).
Big Data 138 - ClickHouse MergeTree Deep Dive: Partition Pruning × Sparse Primary Index × Marks × Compression
ClickHouse MergeTree storage and query path: column files (*.bin), sparse primary index (primary.idx), marker files (.mrk/.
Kafka Operations: Shell Commands & Java Client Examples
Covers Kafka daily operations: daemon startup, Shell topic management commands, and Java client programming (complete Producer/Consumer code) with key configuration param...
Spring Boot Integration with Kafka
Detailed guide on integrating Kafka in Spring Boot projects, including dependency configuration, KafkaTemplate sync/async message sending, and complete @KafkaListener con...
Spark Distributed Environment Setup
Step-by-step Apache Spark distributed computing environment setup, covering download and extract, environment variable configuration, slaves/spark-env.
Big Data 135 - ClickHouse Cluster Connectivity Self-Check & Data Types Guide | Run ON CLUSTER in 10 Minutes
Using three-node cluster (h121/122/123) as example, first complete cluster connectivity self-check: system.
Big Data 136 - ClickHouse Table Engines: TinyLog/Log/StripeLog/Memory/Merge Selection Guide
Scenario: Need to trade-off among small data/temporary table/log landing/multi-table combined reads, often using MergeTree is "using a cannon to kill a mosquito".