Gleam Lab · Blog Archive
Blog Page 22
Technical exploration and engineering notes, 655 articles in total.
Spark Streaming Introduction: From DStream to Structured Streaming
This is article 85 in the Big Data series, introducing the architecture and evolution background of Spark's two generations of streaming frameworks.
Spark Streaming Data Sources: File Stream, Socket, RDD RDD Queue
Comprehensive explanation of three Spark Streaming basic data sources: file stream directory monitoring, Socket TCP ingestion, RDD queue stream for testing simulation.
MyBatis Deep Dive - Level 1 Cache, Code Testing, and Source Code Analysis
Detailed introduction to MyBatis level 1 cache working principles, code testing, invalidation scenarios, and source code analysis.
MyBatis Level 2 Cache - Testing and Source Code Analysis
Detailed introduction to MyBatis level 2 cache working principles, enable configuration, code testing, and source code analysis.
Big Data 219 - Grafana 11.3.0 Installation & Startup: YUM, systemd and Login Setup
For OPs/devs still using CentOS/RHEL (including compatible distributions) in 2026, provides Grafana 11.3.0 (grafana-enterprise-11.3.0-1.x86_64.
Big Data 220 - Data Warehouse Introduction
In 1988, IBM first introduced the concept of "Information Warehouse" when facing increasingly scattered enterprise information systems and growing data silo problems.
Big Data 217 - Prometheus 2.53.2 Installation and Configuration Practice
Scenario: Single-machine deployment of Prometheus 2.53.2, pull node_exporter metrics from multiple hosts and verify Targets status.
Big Data 218 - Prometheus Node Exporter 1.8.2 and Pushgateway 1.10.0
Common Prometheus monitoring deployment: Install node_exporter-1.8.2 on Rocky Linux to expose host metrics, integrate with Prometheus scrape config, and visualize in Graf...
sklearn KMeans Key Attributes & Evaluation: cluster_cluster_centers_、inertia_、metrics
Scenario: Using sklearn for KMeans clustering, want to explain centroids/loss and use metrics for K selection.
Big Data 216 - KMeans n_clusters Selection
KMeans nclusters selection method: calculate silhouettescore and silhouette_samples on candidate cluster numbers (e.g.
SparkSQL Statements: DataFrame Operations, SQL Queries &
Comprehensive guide to SparkSQL core usage including DataFrame API operations, SQL query syntax, lateral view explode, and Hive integration via enableHiveSupport for meta...
Big Data 84 - SparkSQL Internals: Five Join Strategies & Catalyst Optimizer
This is article 84 in the Big Data series, deeply analyzing SparkSQL kernel's Join strategy auto-selection logic and SQL parsing optimization flow.
Big Data 213 - Python Hand-Written K-Means Clustering
Scenario: Hand-write K-Means using NumPy/Pandas, perform 3-class clustering on Iris.txt and output centroids with clustering results.
Big Data 214 - K-Means Clustering Practice: Self-Implemented Algorithm vs sklearn
K-Means clustering provides an engineering workflow that is 'verifiable, reproducible, and debuggable': first use 2D testSet dataset for algorithm verification.
Big Data 211 - Scikit-Learn Logistic Regression Implementation
When using Logistic Regression in Scikit-Learn, max_iter controls maximum iterations affecting model convergence speed and accuracy.
Big Data 212 - K-Means Clustering Guide
K-Means clustering algorithm, comparing supervised vs unsupervised learning (whether labels Y are needed).
Big Data 209 - Deep Understanding of Logistic Regression
Logistic Regression (LR) is an important classification algorithm in machine learning.
Big Data 210 - How to Implement Logistic Regression in Scikit-Learn and Regularization Detailed (L1 and L2)
As C gradually increases, regularization strength gets smaller, model performance on training and test shows upward trend, until around C=0.8.
SparkSQL Core Abstractions: RDD, DataFrame, Dataset & SparkSession
This is article 81 in the Big Data series, comprehensively introducing Spark's three core data abstractions' features, use cases and mutual conversions.
SparkSQL Operators: Transformation & Action Operations
This is article 82 in the Big Data series, systematically introducing SparkSQL Transformation and Action operators with complete test cases.