Gleam Lab · Blog Archive

Blog Page 29

Technical exploration and engineering notes, 655 articles in total.

Gleam Lab technical blog cover: AI Engineering, Java backend, and long-form writing
All Articles Java243Backend50Microservices10AI Engineering86LLM35Big Data271Data Engineering57Kubernetes / Cloud Native3Real-time Voice1Robotics40Personal Growth29 More Tags →
Tutorial Series 3 min read Big Data Engineering

Flink Window Complete Guide: Tumbling, Sliding, Session

Flink's Window mechanism is the core bridge between stream processing and unified batch processing architecture.

big-dataflinkwindow +1
Tutorial Series 2 min read Big Data Engineering

Flink Sliding Window Deep Dive: Principles, Use Cases and Implementation

Sliding window is a more generalized form of fixed window, achieving dynamic window movement through introducing slide interval. It consists of two key parameters

big-dataflinkwindow +1
Tutorial Series 5 min read Big Data Engineering

Flink JDBC Sink Deep Dive: MySQL Real-time Write, Batch Output and Retry

In Apache Flink, JDBC Sink is an important data output component that allows writing stream or batch processed data to relational databases through JDBC connections.

big-dataflinkjdbc +2
Tutorial Series 5 min read Big Data Engineering

Flink Batch Processing DataSet API: Use Cases, Code Examples and Core Operators

Apache Flink's DataSet API is the core programming interface for Flink batch processing, specifically designed for processing static, bounded datasets.

big-dataflinkdataset +1
Tutorial Series 5 min read Big Data Engineering

Redis Memory Management: Key Expiration and Eviction Policies

Comprehensive analysis of Redis memory control mechanisms, including maxmemory configuration, three key expiration deletion strategies (lazy/active/scheduled).

big-datarediscaching +1
Tutorial Series 4 min read Big Data Engineering

Big Data 48 - Redis Communication Internals: RESP Protocol and Reactor Model

This is article 48 in the Big Data series. This article provides an in-depth analysis of Redis communication protocol RESP and Reactor-based event-driven architecture.

big-dataredisdistributed-system +1
Tutorial Series 3 min read Big Data Engineering

Big Data 115 - Flink DataStream Transformation: Map, FlatMap and Filter

Flink provides rich operators for DataStream to support flexible data stream processing in different scenarios.

big-dataflinkdatastream +1
Tutorial Series 3 min read Big Data Engineering

Big Data 116 - Flink Sink Usage Guide: Types, Fault Tolerance Semantics & Scenarios

Flink's Sink is the final output endpoint for data stream processing, used to write processed results to external systems or storage media.

big-dataflinksink +1
Tutorial Series 1 min read Big Data Engineering

Flink Source Operator Deep Dive: Non-Parallel Source Principles

Non-Parallel Source is a source operation in Flink with fixed parallelism of 1. It can only run in a single instance regardless of cluster scale, ensuring tasks are proce...

big-dataflinksource +1
Tutorial Series 5 min read Big Data Engineering

Flink SourceFunction to RichSourceFunction: Enhanced Source Lifecycle and Resource Management

RichSourceFunction and RichParallelSourceFunction are enhanced source functions suitable for scenarios requiring complex logic and resource management.

big-dataflinkdatastream +1
Tutorial Series 2 min read Big Data Engineering

Big Data 111 - Flink on YARN Deployment: Environment Variables, Configuration & Resource Requests

Deploying Flink in YARN mode requires completing a series of environment configuration and cluster management operations.

big-dataflinkyarn +1
Tutorial Series 2 min read Big Data Engineering

Flink DataStream API: DataSource, Transformation and Sink Components

DataSource, Transformation and Sink. DataSource provides diverse data input methods including file systems, message queues, databases and custom data sources.

big-dataflinkdatastream +1
Tutorial Series 4 min read Big Data Engineering

Redis Persistence: RDB vs AOF Comparison and Production Settings

Systematic comparison of Redis two persistence solutions: RDB snapshot and AOF log — configuration methods, trigger mechanisms, pros and cons, AOF rewrite mechanism.

big-datarediscaching +1
Tutorial Series 4 min read Big Data Engineering

Big Data 46 - Redis RDB Persistence: Snapshot Principles, Configuration and Tradeoffs

In-depth analysis of Redis RDB persistence mechanism, covering trigger methods, BGSAVE execution flow, configuration parameters, file structure, and comparison with AOF.

big-datarediscaching +1
Tutorial Series 9 min read Big Data Engineering

Flink Architecture Deep Dive: JobManager, TaskManager and Client

Flink's runtime architecture adopts typical Master/Slave pattern with clear division of responsibilities among core components.

big-dataflink
Tutorial Series 3 min read Big Data Engineering

Big Data 110 - Flink Installation and Deployment Guide: Local, Standalone and YARN

Flink provides multiple installation modes to suit different scenarios.

big-dataflink
Tutorial Series 8 min read Big Data Engineering

Apache Flink Deep Dive: From Origin to Technical Features

Apache Flink is an open-source big data stream processing framework, supporting efficient computation of unbounded stream and bounded batch data.

big-dataflink
Tutorial Series 4 min read Big Data Engineering

Big Data 108 - Flink Stream-Batch Integration: Concepts & WordCount Practice

Definition: Stream processing means real-time processing of continuously flowing data streams.

big-dataflink
Tutorial Series 4 min read Big Data Engineering

Redis Lua Scripts: EVAL, redis.call and Atomic Operations

Systematic explanation of Redis Lua script EVAL command syntax, differences between redis.call and redis.

big-dataredisjava +1
Tutorial Series 3 min read Big Data Engineering

Redis Slow Query Log and Performance Tuning in Production

Detailed explanation of Redis slow query log configuration parameters (slowlog-log-slower-than, slowlog-max-len), core commands.

big-datarediscaching +1