Gleam Lab · Blog Archive
Blog Page 17
Technical exploration and engineering notes, 655 articles in total.
Offline Data Warehouse Hive ADS Export MySQL DataX Practice
The landing path for exporting Hive ADS layer tables to MySQL in offline data warehouse. Gives typical DataX solution: hdfsreader -> mysqlwriter.
MyBatis Annotation Development - One-to-Many and Many-to-Many Mapping
The relationship between user table and order table: one user has multiple orders, one order belongs to one user.
Big Data 233 - Offline Data Warehouse Retention Rate: DWS Modeling & ADS Hive Aggregation
Implementation of 'Member Retention' in offline data warehouse: DWS layer uses dwsmemberretention_day table to join new member and startup detail tables to.
MyBatis Annotation Development - Basic Introduction and One-to-One Mapping
Detailed introduction to MyBatis annotation development approach, including usage of common annotations (@Select, @Insert, @Update, @Delete, @Results, etc.
Neo4j CQL Quick Reference: String, Aggregate, Relationship Functions and Paths
Neo4j CQL functions quick reference: string functions, aggregate functions, relationship functions and shortest path queries.
Big Data 232 - Hive New Member & Retention
Offline data warehouse calculates 'new members' daily, and provides consistent definition data foundation for subsequent 'member retention'.
Big Data 90 - Apache Flink Introduction: Unified Stream-Batch Real-Time Computing
Systematic introduction to Apache Flink's origin, core features, and architecture components: JobManager, TaskManager, Dispatcher responsibilities, unified stream-batch p...
MyBatis Deep Dive - Many-to-Many Model with Collection Tags
In database design, a "many-to-many" relationship means records in two tables can be associated with each other, with multiple records associated with multiple records.
Offline Data Warehouse Hive Practice: DWD to DWS Daily/Weekly/Monthly Active Member
This article introduces using Hive to build an offline data warehouse for calculating active members (daily/weekly/monthly).
MyBatis Deep Dive - One-to-Many Model with Collection Tags
In database design, one-to-many (One-to-Many) relationship is one of the most common relationship models.
Neo4j CQL Practice: WHERE, DELETE/DETACH, SET, Sorting and Pagination
Neo4j CQL advanced queries: WHERE clause filtering, DELETE removal, SET updates, ORDER BY sorting and pagination.
Hive ODS Layer JSON Parsing: UDF Array Extraction, explode/json_tuple
JSON data processing in Hive offline data warehouse, covering three most common needs: 1) Extract array fields from JSON strings and explode-expand in SQL
MyBatis Deep Dive - One-to-One Mapping, SqlMapConfig and Mapper XML Configuration
Detailed explanation of one-to-one model mapping in MyBatis, including database table design, Java entity class definitions, Mapper interface and XML configuration.
Hive ODS Layer Practice: External Table Partition Loading and JSON Parsing
Engineering implementation of the ODS (Operational Data Store) layer in an offline data warehouse: the minimal closed loop of Hive external table + daily.
Big Data 89 - Spark Streaming with Kafka: Receiver vs Direct Mode
This is article 89 in the Big Data series, deeply comparing two core modes of Spark Streaming integration with Kafka, focusing on Direct mode production practices.
MyBatis Deep Dive - Dynamic SQL, Parameters, Foreach and SQL Fragments
In-depth explanation of MyBatis dynamic SQL configuration including if conditions, where tags, foreach loops, and sql fragment extraction.
Neo4j CQL Practice: CREATE/MATCH and Relationship Modeling
Neo4j CQL query language guide: CREATE creates nodes and relationships, MATCH queries graph data, with complete code examples for graph database operations and traversal...
MyBatis Mapper Proxy Invoke - Source Code Analysis
Detailed introduction to MyBatis Mapper Proxy dynamic proxy implementation principles, including getMapper source code analysis and MapperMethod execution process.
Big Data 228 - Flume Taildir + Custom Interceptor: Extract JSON Timestamps, Mark Headers & Partition HDFS by Event Time
Apache Flume offline log collection implementation using Taildir Source and a custom Interceptor to extract JSON timestamps, mark headers, and route HDFS partitions by ev...
MyBatis Deep Dive - SqlSessionFactory, SqlSession, DAO and Mapper
In-depth explanation of MyBatis core components: SqlSessionFactory and SqlSession working principles, traditional DAO layer implementation, and MyBatis proxy development...