Blog
Technical exploration and thoughts · 655 articles
Spark Streaming Kafka Consumption: Offset Acquisition, Storage and Management
When Spark Streaming integrates with Kafka, Offset management is key to ensuring data processing continuity and consistency.
Big Data 104 - Spark Streaming with Kafka: Offset Management Mechanisms & Best Practices
Offset is used to mark message position in Kafka partition. Proper management can achieve at-least-once or even exactly-once data processing semantics.
Spark Streaming Stateful Transformations: Window Operations and State Tracking
Window operations integrate data from multiple batches over a longer time range by setting window length and slide duration.
Big Data 102 - Spark Streaming with Kafka: Receiver and Direct Approaches
This article introduces two Spark Streaming integration methods with Kafka: Receiver Approach and Direct Approach.
Redis Advanced Data Types: Bitmap, Geo and Stream
Deep dive into Redis three advanced data types: Bitmap, Geo (GeoHash, Z-order curve, Base32 encoding), and Stream message stream, with common commands and practical examp...
Redis Pub/Sub: Mechanism, Weak Transaction and Risks
Detailed explanation of Redis Pub/Sub working mechanism, three weak transaction flaws (no persistence, no acknowledgment, no retry), and alternative solutions in producti...
Redis Single Node and Cluster Installation
Install Redis 6.2.9 from source on Ubuntu, configure redis.conf for daemon mode, start redis-server and verify connection via redis-cli.
Big Data 40 - Redis Five Data Types: Command Reference and Practice
Comprehensive explanation of Redis five data types: String, List, Set, Sorted Set, and Hash. Includes common commands, underlying characteristics, and typical usage scena...
Big Data 37 - HBase Java API: Complete CRUD Code with Table Creation
Using HBase Java Client API to implement table creation, insert, delete, Get query, full table scan, and range scan.
Redis Introduction: Features and Architecture
Introduction to Redis: in-memory data structure store, key-value database, with comparison to traditional databases and typical use cases.
HBase Cluster Deployment and High Availability Configuration
This is article 35 in the Big Data series. Complete HBase distributed cluster deployment on three-node Hadoop + ZooKeeper cluster.
HBase Shell CRUD Operations and Data Model
HBase Shell commands: create table, Put/Get/Scan/Delete operations, explain HBase data model with practical examples.
Big Data 33 - HBase Overall Architecture: HMaster, HRegionServer and Data Model
Comprehensive analysis of HBase distributed database overall architecture, including ZooKeeper coordination, HMaster management node, HRegionServer data node...
HBase Single Node Configuration: hbase-env and hbase-site.xml
Step-by-step configure HBase single node environment, explain hbase-env.sh, hbase-site.xml key parameters, complete integration with Hadoop HDFS and ZooKeeper cluster.
ZooKeeper Leader Election and ZAB Protocol Principles
This is article 31 in the Big Data series. Deep analysis of ZooKeeper Leader election mechanism and ZAB (ZooKeeper Atomic Broadcast) protocol implementation principles.
ZooKeeper Distributed Lock Java Implementation Details
This is article 32 in the Big Data series. Demonstrates how to implement fair distributed lock using ZooKeeper ephemeral sequential nodes, with complete Java code.
ZooKeeper Watcher Principle and Command Line Practice Guide
Complete analysis of Watcher registration-trigger-notification flow from client, WatchManager to ZooKeeper server, and zkCli command line practice demonstrating node CRUD...
ZooKeeper Java API Practice: Node CRUD and Monitoring
Use ZkClient library to operate ZooKeeper via Java code, complete practical examples of session establishment, persistent node CRUD, child node change monitoring...
ZooKeeper Cluster Configuration Details and Startup Verification
Deep dive into zoo.cfg core parameter meanings, explain myid file configuration specifications, demonstrate 3-node cluster startup process and Leader election result veri...
ZooKeeper ZNode Data Structure and Watcher Mechanism Details
Deep dive into ZooKeeper's four ZNode node types, ZXID transaction ID structure, and one-time trigger Watcher monitoring mechanism principles and practice.