Gleam Lab · Blog Archive
Blog Page 15
Technical exploration and engineering notes, 655 articles in total.
Offline Data Warehouse ADS Layer and Airflow Task Task Scheduling
Apache Airflow is an open-source task scheduling and workflow management platform, primarily used for developing, debugging, and monitoring data pipelines.
Big Data 251 - Airflow Installation
Apache Airflow is an open-source task scheduling and workflow management tool for orchestrating complex data processing tasks.
Big Data 96 - Flink Broadcast State: BroadcastState Practice and Rule Updates
Flink Broadcast State explanation: BroadcastState principle, dynamic rule updates, state partitioning and memory management, demonstrating broadcast stream and non-broadc...
Big Data 97 - Flink State Backend: State Storage and Performance Optimization
Flink State Backend detailed explanation: HashMapStateBackend, EmbeddedRocksDBStateBackend selection, memory configuration and performance tuning.
Implementing Simple IoC-02: Implementing Business Without Spring
Manually implement business logic layering without using Spring, including DAO layer, Service layer, and Controller layer, demonstrating dependency management issues in t...
Implementing Simple IoC-01: Servlet Introduction and Basic Principles
Step-by-step guide to implementing a simple Spring IoC container, Part 1 covering Servlet basics, characteristics, working principles, and writing your first Servlet prog...
Big Data 247 - Offline Data Warehouse - Hive Order Zipper Table: Incremental Refresh Implementation
This article continues the zipper table practice, focusing on order history state incremental refresh.
Big Data 248 - Offline Data Warehouse: Dimension Tables
First determines fact tables vs dimension tables: green indicates fact tables, gray indicates dimension tables.
Big Data 249 - Offline Data Warehouse DWD and DWS Layer
The order table is a periodic fact table; zipper tables can be used to retain order status history. Order detail tables are regular fact tables.
FastDFS Architecture: Storage/Tracker/Group Design
FastDFS core architecture deep dive: Tracker Server, Storage Server, Group mechanism. This article analyzes principles and practices to help readers master key technical...
Spring Overview: History and Core Concepts IoC, AOP
Comprehensive introduction to Spring framework basics, evolution from EJB to Spring, detailed explanation of Spring's core concepts including IoC and AOP with practical i...
Big Data 247 - Offline Data Warehouse - Hive Zipper Table Practice: Order History State Incremental Refresh
This article provides a practical guide to Hive zipper tables for preserving order history states at low cost in offline data warehouses, supporting daily backtracking an...
Handwritten ORM Framework 4: Framework Optimization and DefaultSqlSession Proxy Implementation
Optimize handwritten ORM framework, focusing on getMapper method implementation in DefaultSqlSession, using JDK dynamic proxy to create Mapper interface proxy objects.
Big Data 246 - Offline Data Warehouse - Hive Zipper Table Practice: Initialization, Incremental Update, Rollback Script
userinfo (partitioned table) => userid, mobile, regdate => Daily changed data (modified + new) / Historical data (first day) userhis (zipper table) => Two additional fiel...
FastDFS 6.15.0 + Nginx Dynamic Module: Source Compile to Access Link
Complete guide for FastDFS 6.15.0 source compilation and Nginx fastdfs-nginx-module configuration. This article analyzes principles and practices to help readers master k...
Handwritten ORM Framework 3: SqlSession and Executor Working Principles
In-depth analysis of SqlSession and Executor working principles in handwritten ORM framework, detailing implementation mechanisms of SqlSessionFactoryBuilder...
Big Data 245 - Offline Data Warehouse - Hive Zipper Table Getting Started: SCD Types, Table Creation and Loading
Slowly Changing Dimensions (SCD) refer to dimension attributes that change slowly over time in the real world (slow is relative to fact tables, where data changes faster...
Handwritten ORM Framework 2: Resources, MappedStatement, and XMLBuilder Implementation
In the current project, create under resources: sqlMapConfig.xml mapper.xml
FastDFS: From Standalone to Distributed File Storage
FastDFS distributed file system architecture, evolution from standalone to distributed. This article analyzes principles and practices to help readers master key technica...
Handwritten ORM Framework 1 - JDBC Problems and Solutions
First article in the handwritten ORM framework series, introducing problems and disadvantages of traditional JDBC development and ORM framework design ideas and solutions.