AI Agent, multimodal interaction, and edge-cloud systems engineer
📖
This article is from CSDN original.
Click to read the original for a better reading experience.
Reading Guide
AI Research & Notes
Research-oriented notes for technical trends, model capabilities, and industry context.
Best For
Readers who need a quick technical or industry background
Engineers turning research notes into engineering judgment
Prerequisites
Basic AI or backend concepts are helpful
Focus on conclusions, boundaries, and implementation implications
Takeaways
Get topic background, key terms, and trend signals
Identify engineering questions that need deeper validation
This is a research or archive-style note. Treat it as background material and verify implementation details against current official documentation when applying it.
I. From Batch Processing to In-Memory Computing
Batch Processing Era (2006-2012)
MapReduce framework dominated
Typical scenarios: overnight ETL, log analysis, data cleaning
Performance bottlenecks: intermediate results require disk writes, high task scheduling overhead
Spark Revolution (2013)
Innovation: RDD in-memory computing, DAG execution plan, multi-language support
Performance improvement:
Iterative algorithms: 100x faster
Interactive queries: 10-100x faster
Batch jobs: 10-30x faster
Ecosystem Evolution
Cloudera Impala (2013): First open-source MPP SQL engine
Facebook Presto (2013): Supports multiple data sources
Apache Drill (2015): Supports semi-structured data
II. From Offline to Real-time Computing
Offline Computing Era
T+1 mode: process previous day’s data on the current day
Applicable: daily reports, historical analysis, ML model training
Rise of Real-time Stream Processing
Apache Storm: Sub-second latency, but only supports “at-most-once” semantics
