Gleam Lab · Blog Archive
Blog Page 23
Technical exploration and engineering notes, 655 articles in total.
Big Data 207 - How to Handle Multicollinearity
When using scikit-learn for linear regression, how to handle multicollinearity in least squares method.
Big Data 208 - Ridge Regression and Lasso Regression
Ridge Regression and Lasso Regression are two commonly used linear regression regularization methods for solving overfitting and multicollinearity in machine.
Big Data 205 - Linear Regression Machine Learning Perspective
Linear Regression core chain: unify prediction function y=Xw in matrix form, treat parameter vector w as only unknown
Big Data 206 - NumPy Matrix Multiplication Hand-written Multivariate Linear Regression
pandas DataFrame and NumPy matrix multiplication hand-written multivariate linear regression (linear regression implementation).
Big Data 203 - sklearn Decision Tree Pruning Parameters
Common parameters for decision tree pruning (pre-pruning) in engineering: maxdepth, minsamplesleaf, minsamplessplit, maxfeatures, minimpuritydecrease.
Big Data 204 - Confusion Matrix to ROC: Imbalanced Binary Classification Metrics in sklearn
Confusion matrix (TP, FP, FN, TN) with unified metrics: Accuracy, Precision, Recall (Sensitivity), F1 Measure, ROC curve, AUC value, and practical business interpretation...
Spark Standalone Mode: Architecture & Performance Tuning
Comprehensive explanation of Spark Standalone cluster four core components, application submission flow, SparkContext internal architecture, Shuffle evolution history and...
SparkSQL Introduction: SQL & Distributed Computing Fusion
Systematic introduction to SparkSQL evolution history, core abstractions DataFrame/Dataset, Catalyst optimizer principle, and practical usage of multi-data source integra...
Big Data 201 - Decision Tree from Split to Pruning
Decision tree is a tree-structured supervised learning model, commonly used for classification and regression tasks.
Big Data 202 - sklearn Decision Tree Practice: criterion, Graphviz Visualization & Pruning
Complete flow of DecisionTreeClassifier on load_wine dataset from data splitting, model evaluation to decision tree visualization (2026 version).
Big Data 199 - Decision Tree Model Explained: Node Structure, Conditional Probability & Shannon Entropy
Tree model is a widely used algorithm type in supervised learning, can be applied to both classification and regression problems.
Big Data 200 - Decision Tree Information Gain Detailed
Scenario: Use information entropy/information gain to explain why decision tree selects certain column for splitting, and use Python to reproduce "best split column".
Big Data 197 - K-Fold Cross-Validation Practice
Random train/test split causes evaluation metrics to be unstable, and gives engineering solution: K-Fold Cross Validation.
Big Data 198 - KNN Must Normalize First: Min-Max Scaling, Data Leakage Pitfalls & sklearn Practice
In scikit-learn pipelines, distance-based models like KNN are highly sensitive to inconsistent feature scales. Split first, fit MinMaxScaler only on the training set...
Spark RDD Fault Tolerance: Checkpoint Principle & Best Best Practices
Detailed explanation of Spark Checkpoint execution flow, core differences with persist/cache, partitioner strategies, and best practices for iterative algorithms and long...
Spark Broadcast Variables: Efficient Shared Read-Only Data
Detailed explanation of Spark broadcast variable working principle, configuration parameters and best practices.
Big Data 195 - KNN/K-Nearest Neighbors Algorithm Practice
KNN/K-Nearest Neighbors Algorithm: From Euclidean distance calculation, distance sorting, TopK voting to function encapsulation, giving reproducible Python.
Big Data 196 - scikit-learn KNN Practice: KNeighborsClassifier, kneighbors & Learning Curves
Since being initiated in 2007 by David Cournapeau, scikit-learn (sklearn) has become one of the most important machine learning libraries in the Python ecosystem.
Big Data 193 - Apache Tez Practice: Hive on Tez Installation, DAG Principles & Common Pitfalls
Tez (pronounced "tez") is an efficient data processing framework running in the Hadoop ecosystem, designed to optimize batch processing and interactive queries.
Big Data 194 - Data Mining Overview: From Wine Classification to Supervised, Unsupervised & Reinforcement Learning
In a bar, there are ten almost identical glasses of wine on the counter. The boss says want to play a game, win and drink for free, lose and pay three times for the wine.