Gleam Lab · Blog Archive
Blog Page 13
Technical exploration and engineering notes, 655 articles in total.
Spark MLlib Decision Tree Pruning: Pre-pruning, Post-Principles and Practice
This article systematically introduces decision tree pre-pruning and post-pruning principles, compares core differences between three mainstream algorithms.
Spark MLlib Decision Tree: Classification Principles, Gini/Entropy and Practice
This article introduces the basic concepts, classification principles, and classification principles of decision trees.
Big Data 272 - Spark MLlib Logistic Regression: Basics, Input Function, Sigmoid & Loss
This article introduces the basic principles, application scenarios, and implementation in Spark MLlib of logistic regression.
Big Data 271 - Spark MLlib Linear Regression: Scenarios, Loss Function & Optimization
Linear regression uses regression equations to model relationships between independent and dependent variables.
Big Data 268 - Real-time Warehouse ODS Layer: Writing Kafka Dimension Tables into DIM
Kafka is a distributed streaming platform for high-throughput message passing. In ETL processes, Kafka serves as a data message queue or stream processing source.
Big Data 269 - Real-time Warehouse DIM, DW and ADS: Scala Pipelines to HBase
Original MySQL area table to HBase: Convert area table to region ID, region name, city ID, city name, province ID, province name, and write to HBase.
Big Data 271 - Spark MLlib Logistic Regression: Sigmoid, Loss Function & Diabetes Prediction Case
Logistic Regression is a classification model in machine learning. Despite having "regression" in its name, it is a classification algorithm.
Big Data #266: Canal Integration with Kafka - Real-time Data Sync
This article introduces Alibaba's open-source Canal tool, which implements Change Data Capture (CDC) by parsing MySQL binlog.
Big Data 267 - Real-Time Warehouse ODS: Lambda and Kappa Architecture
In internet companies, common ODS data includes business log data (Log) and business DB data.
Spark MLlib Linear Regression: Scenarios, Loss Function and Optimization
Linear Regression is an analytical method that uses regression equations to model the relationship between one or more independent variables and a dependent.
Big Data 265 - Canal Deployment
Canal is an open-source data synchronization tool from Alibaba for MySQL database incremental log parsing and synchronization.
Big Data 263 - Canal Working Principle: Workflow and MySQL Binlog Basics
Canal is an open-source tool for MySQL database binlog incremental subscription and consumption.
MySQL Binlog Deep Dive: Storage Directory, Change Records and Format
MySQL's Binary Log (binlog) is a log file type in MySQL that records all change operations performed on the database (excluding SELECT and SHOW queries).
Canal Data Sync: Introduction, Background, Principles and Architecture
Alibaba B2B's cross-region business between domestic sellers and overseas buyers drove the need for data synchronization between Hangzhou and US data centers.
Spring In-Depth: AOP Aspect Enhancement Core Concepts, Terminology and Proxy Configuration
In-depth introduction to Spring AOP aspect enhancement covering core concepts, related terminology, and proxy configuration with practical examples.
Big Data 261 - Real-Time Warehouse Business Table Structure
Realtime data warehouse is a data warehouse system that differs from traditional batch processing data warehouses by emphasizing low latency, high throughput.
Spring In-Depth: IoC Container Circular Dependency, Prototype Scope and Lazy ObjectFactory
In-depth analysis of Spring IoC container system covering circular dependency, prototype beans, prototype scope, and lazy ObjectFactory with solutions.
Big Data 260 - Real-Time Data Warehouse: Background, Architecture and Requirements
Real-time data processing capability has become a key competitive factor for enterprises.
Spring In-Depth: IoC Container System BeanFactory Analysis and Lazy-Init
In-depth analysis of Spring IoC container system covering BeanFactory process analysis and Bean lazy loading mechanism with practical examples.
Big Data 259 - Griffin Configuration
Apache Griffin is an open-source data quality management framework designed to help organizations monitor and improve data quality in big data environments.