The concept of data warehouse may be traced back to last century. with the continuous growth of big data and developments of Hadoop ecosystem, offline data warehouse based on Hive/HDFS architect can rise. And recently years, Storm/Spark(Steaming)/Flink etc... real-time frameworks go up and have a rapid development. Every company need a real-time data solution in their system. In this article, we will talk about some typical real-time data architect in Chinese internet companies like Meituan,Netease and OPPO; selections of their storage and computing engines, also with layers division may inspire us in some point.

Four examples:

• Meituan Flink based real-time data warehouse platform
• Yanxuan of Netease Flink based real-time data warehouse in practical
• OPPO real-time data warehouse secrets and offline to real-time smooth migration

Meituan Flink based real-time data warehouse platform

• Lowest layer is design to collect data from different channels like Binlog of mysql, logs printed by server, IoT data... All of these data will be sent into Kafka, they're not only related to real-time computing, but also offline's
• Above the collection is the storage layer. Out of Kafka, there are also HDFS store for status data and Hbase for dimensional data
• And above is the Engines layer. Include Storm/Flink to provide common platform package and component support
• And beyond engine is the platform layer. platform will do the management in data, tasks and resources three dimension
• In the toppest layer is the applications like real-time data warehouse, machine learning, etl and event drive applet

from a functional perspective, Meituan's real-time computing platform contains jobs config\publish\status managements and resource managements. Resource management means multi-tenant resource isolation, delivery and deployment.

traditional data warehouse model

There are always 4 layers from bottom to top. ODS(Operational Data Store), DWD(Data Warehouse Detail), DWS(DWS, Data Warehouse Summary), ADS(ADS，Application Data Store) with Hive or spark for query.

real-time data warehouse

In real-time model, DWD & DWS always based on Kafka. considering of performance, dimensional data are always placed in HBase or Tair KV storage.

Quasi-real-time data warehouse model