Gleam Lab · Blog Archive
Blog Page 28
Technical exploration and engineering notes, 655 articles in total.
Kafka Components: Producer, Broker, Consumer Full Flow
Deep dive into Kafka's three core components: Producer partitioning strategy and ACK mechanism, Broker Leader/Follower architecture, Consumer Group partition assignment a...
Kafka Installation: From ZooKeeper to KRaft Evolution
Introduction to Kafka 2.x vs 3.x core differences, detailed cluster installation steps, ZooKeeper configuration, Broker parameter settings, and how KRaft mode replaces Zo...
Big Data 133 - ClickHouse Concepts & Basics | Why Fast? Columnar + Vectorized + MergeTree Comparison
Scenario: Want high-concurrency low-latency OLAP, and don't want to use entire Hadoop/lakehouse.
Big Data 134 - ClickHouse Single Machine + Cluster Node Deployment Guide | Installation Configuration | systemd Management / config.d
Official recommended keyring + signed-by installation of ClickHouse on Ubuntu, start with systemd and self-check
Big Data 131 - Flink CEP Practice: 24 Hours ≥5 Transactions & 10 Minutes Unpaid Detection Cases
Flink CEP (Complex Event Processing) is an extension library provided by Apache Flink for real-time complex event processing.
Big Data 132 - Flink SQL Quick Start | Table API + SQL in 3 Minutes with toChangelogStream New Syntax
Engineering perspective to quickly run Flink SQL: Provides modern dependencies (no longer using blink planner), minimum runnable example (MRE).
Flink CEP Deep Dive: Complex Event Processing Complete Guide
Flink CEP (Complex Event Processing) is a core component of Apache Flink, specifically designed for processing complex event streams.
Flink CEP Timeout Event Extraction: Complete Guide with Matched and Timed-out Events
Flink CEP timeout event extraction is a key step in stream processing, used to capture partial matching events that exceed the window time (within) during pattern matchin...
Redis High Availability: Master-Slave Replication & Sentinel
This is article 51 in the Big Data series, covering Redis high availability architecture: master-slave replication, Sentinel mode, and distributed lock design.
Kafka Architecture: High-Throughput Distributed Messaging
Systematic introduction to Kafka core architecture: Topic/Partition/Replica model, ISR mechanism, zero-copy optimization, message format and typical use cases.
Flink StateBackend Deep Dive: Memory, Fs, RocksDB and Operator State
ManagedOperatorState is used to manage non-keyed state, achieving state consistency when operators recover from faults or scale.
Flink Parallelism Deep Dive: From Concepts to Best Practices
Basic Concept of Parallelism In Apache Flink, Parallelism refers to the number of parallel tasks that can run simultaneously for each operator during execution.
Big Data 125 - Flink Broadcast State: Dynamic Logic Updates in Real-Time Streaming
Broadcast State is an important mechanism in Apache Flink that supports dynamic logic updates in streaming applications.
Big Data 126 - Flink State Backend: Memory, Fs, RocksDB and Performance Differences
State Storage Methods: MemoryStateBackend: Stores state in TaskManager's Java memory. Fast but limited (5MB per state default, 10MB per task).
Flink Parallelism Setting Priority: Principles, Configuration and Tuning
A Flink program consists of multiple Operators (Source, Transformation, Sink).
Big Data 124 - Flink State: Keyed State, Operator State and KeyGroups
Based on whether intermediate state is needed, Flink computation can be divided into stateful and stateless: Stateless computation like Map, Filter, FlatMap.
Redis Cache Problems: Penetration, Breakdown, Avalancheand Solutions
Systematic overview of the five most common Redis cache problems in high-concurrency scenarios: cache penetration, cache breakdown, cache avalanche, hot key, and big key.
Big Data 50 - Redis Distributed Lock: Optimistic Lock, WATCH and SETNX
Redis optimistic lock in practice: WATCH/MULTI/EXEC mechanism explained, Lua scripts for atomic operations, SETNX+EXPIRE distributed lock from basics to Redisson...
Big Data 121 - Flink Time Semantics: EventTime, ProcessingTime, IngestionTime & Watermarks
Watermark is a special marker used to tell Flink the progress of events in the data stream.
Big Data 122 - Flink Watermark Guide: Event Time, Out-of-Order Data and Late Events
When using event-time based windows, Flink relies on Watermark to decide when to trigger window computation.