Big Data #269: Real-time Warehouse DIM, DW and ADS Layer ...

DIM Layer Processing

Implementation Approach

Original MySQL area table to HBase: Convert area table to region ID, region name, city ID, city name, province ID, province name, and write to HBase.

When analyzing transactions, dimensions like seller, buyer, product, and time describe the transaction context. Dimensions generally serve query constraints, classification summary, and sorting.

HBaseReader

class HBaseReader extends RichSourceFunction[(String, String)] {
  private var conn: Connection = null
  private var table: Table = null
  private var scan: Scan = null
  // ... Configure HBase connection, scan table, read data
}

HBaseWriterSink

class HBaseWriterSink extends RichSinkFunction[String] {
  // Write processed data to HBase
  def insertDimArea(hbTable: Table, value: String): Unit = {
    val infos: Array[String] = value.split(",")
    // Write areaId, aname, cid, city, proId, province to HBase
  }
}

AreaDetailInfo (Main Program)

object AreaDetailInfo {
  def main(args: Array[String]): Unit = {
    // 1. Read raw data from HBase
    // 2. Convert to AreaDetail format
    // 3. Use Flink SQL for three-table join (area -> city -> province)
    // 4. Write results to new HBase table
  }
}

DW Layer Processing

DW (Data Warehouse layer) is built from DWD, DWS, and DIM layer data, primarily completing data architecture and integration, establishing consistent dimensions, building reusable detail fact tables oriented toward analysis and statistics, and aggregating metrics at common granularity.

• DWD (Data Warehouse Detail): Isolation layer between business layer and data warehouse, driven by business process modeling
• DWS (Data WareHouse Service): Based on DWD, integrates and aggregates data for analyzing specific subject domains
• DIM (Common Dimension Layer): Based on dimensional modeling concepts, establishes consistent dimensions
• TMP: Temporary layer for storing intermediate data during computation

ADS Layer Processing

ADS (Application Data Store). Based on DW data, integrates and aggregates subject domain service data for subsequent business queries.

Data Detail Layer

Write analysis results from data detail layer to Redis via ClickHouse, Redis, Druid, etc.

class MySinkToRedis extends RichSinkFunction[(CityOrder, Long)] {
  override def invoke(value: (CityOrder, Long), context: SinkFunction.Context[_]): Unit = {
    // Write to Redis
    jedis.hset(s"${value._1.province}${value._1.city}", map);
  }
}