Gleam Lab · Blog Archive
Blog Page 16
Technical exploration and engineering notes, 655 articles in total.
Big Data 244 - Offline Data Warehouse: Hive ODS Layer
Sync MySQL data to specified HDFS directory via DataX, then create ODS external tables in Hive with unified dt string partitioning.
Big Data 243 - Offline Data Warehouse: E-commerce Core Transaction Incremental Import
Scenario: Three core e-commerce transaction tables do daily incremental to offline data warehouse ODS, partitioned by dt Conclusion: DataX uses MySQLReader + HDFSWriter.
MyBatis Design Patterns - Proxy Pattern and Source Code Analysis
Proxy Pattern: Provides a proxy for an object and controls access to the original object through the proxy. The English name of proxy pattern is Proxy.
Neo4j + Spring Boot Practice: Integration from Driver to Repository
Complete guide to integrating Spring Boot with Neo4j: Java Driver, Neo4jTemplate, Repository pattern with practical examples for graph database CRUD operations and relati...
MyBatis Design Patterns - Builder Pattern, Factory Pattern and Source Code Analysis
Detailed introduction to design patterns used in MyBatis source code including builder pattern, factory method pattern, singleton pattern, proxy pattern...
Big Data 241 - Offline Data Warehouse Practice: E-commerce Core Transaction Data Model & MySQL Source Table Design
Focusing on three main metrics: order count, product count, payment amount, breakdown analysis dimensions by sales region and product type (3-level category).
Big Data 95 - Flink State and Checkpoint: State Management, Fault Tolerance and Savepoints
Flink stateful computation explanation: Keyed State, Operator State, Checkpoint configuration, Savepoint backup and recovery, production environment practices.
Big Data 240 - Offline Data Warehouse Advertising Hive ADS Practice: DataX Export to MySQL
Complete solution for exporting Hive ADS layer data to MySQL using DataX.
Neo4j Access Modes: Embedded vs Server with Java Examples
Neo4j embedded database vs server mode comparison, Java API access examples.
Offline Data Warehouse Advertising Business: Flume Import HDFS + ODS/DWD
Using Flume Agent to collect event logs and write to HDFS, then use Hive scripts to complete ODS and DWD layer data loading by date.
Neo4j Backup/Recovery + Warm-up and Execution Plan Practice
Neo4j database backup and recovery, data warm-up and execution plan analysis.
Offline Data Warehouse Advertising Business Hive Analysis: CTR/CVR/Top100
action: User behavior; 0 impression; 1 click after impression; 2 purchase duration: Stay duration shopid: Merchant id eventtype: "ad" adtype: Format type; 1 JPG; 2 PNG
Big Data 93 - Flink Streaming Introduction: DataStream API and Program Structure
This is article 93 in the Big Data series, introducing Flink DataStream API core concepts and program structure.
Flink Window and Watermark: Time Windows, Tumbling/Sliding/Session
Comprehensive analysis of Flink Window mechanism: tumbling windows, sliding windows, session windows, Watermark principle and generation strategies, late data processing...
Big Data 237 - Offline Data Warehouse Hive Advertising Practice: ODS to DWD Event Parsing
This article introduces completing parsing, cleaning, and detail modeling from ODS to DWD for offline data warehouse based on advertising events in tracking logs.
Big Data 236 - Offline Data Warehouse Member Metrics Verification, DataX Export and Advertising Pipeline
This is a practical verification article for member theme and advertising business pipeline based on Hadoop + Hive + HDFS + DataX + MySQL.
Neo4j Transaction, Index and Constraint Practice: Syntax, Concurrency and Troubleshooting
Neo4j transaction handling, index creation, constraint settings and concurrency issue troubleshooting.
Big Data 235 - Offline Data Warehouse Practice: Flume, HDFS and Hive for ODS, DWD, DWS and ADS
This article demonstrates a complete offline data warehouse pipeline from log collection to member metric analysis.
Big Data 91 - Flink Installation & Deployment: Local, Standalone and YARN Modes
Apache Flink is a distributed stream processing framework widely used for real-time data computing scenarios.
Flink on YARN Deployment: Environment Preparation, Resource Manager
Detailed explanation of three Flink deployment modes on YARN cluster: Session, Application, Per-Job modes, Hadoop dependency configuration, YARN resource application and...