TL;DR

  • Scenario: Deploy Apache Kylin 3.1.1 (HBase1.x version) in resource-constrained 3-node (h121/122/123) environment
  • Conclusion: Through environment variables and soft links supplement, correct HADOOP_CONF_DIR, start components in order, successfully start and login to Kylin (7070)
  • Output: Complete command sequence, version matrix (verified), error quick reference card (common symptom→location→fix)

Version Matrix

Component/PackageVersionVerifiedNote
Apache Kylin3.1.1 (apache-kylin-3.1.1-bin-hbase1x)YesFrom archive.apache.org, runs on h122, Web port 7070, default ADMIN/KYLIN (uppercase)
Hadoop2.9.2YesHADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop; start HDFS first, then YARN; HADOOP_CLASSPATH=hadoop classpath
Hive2.3.9YesStart Metastore independently (9083), log /tmp/metastore.log
HBase1.3.1Yeshbase.zookeeper.quorum only fill hostname; three-node collaborative start
Spark2.4.5 (without-hadoop, Scala 2.12)YesConfigure as dependency soft link; didn’t run compute jobs
ZooKeeperVersion not recorded (three-node)YesEach node zkServer.sh start

Cluster Planning

Note: Requires hbase.zookeeper.quorum value to be only host1, host2, not host1:2181, host2:2181.

Configure hbase-site.xml:

cd /opt/servers/hbase-1.3.1/conf
vim hbase-site.xml

Project Download

Download address:

https://archive.apache.org/dist/kylin/

Use version:

https://archive.apache.org/dist/kylin/apache-kylin-3.1.1/apache-kylin-3.1.1-bin-hbase1x.tar.gz

Download command:

cd /opt/software
wget https://archive.apache.org/dist/kylin/apache-kylin-3.1.1/apache-kylin-3.1.1-bin-hbase1x.tar.gz

Extract and Move

cd /opt/software
tar -zxvf apache-kylin-3.1.1-bin-hbase1x.tar.gz

Move to servers directory:

mv apache-kylin-3.1.1-bin-hbase1x /opt/servers/

Environment Variables

vim /etc/profile

Add Kylin environment variables:

export KYLIN_HOME=/opt/servers/apache-kylin-3.1.1-bin-hbase1x
export PATH=$PATH:$KYLIN_HOME/bin

Refresh environment variables:

source /etc/profile

Dependency Components

cd $KYLIN_HOME/conf

# Create soft links
ln -s $HADOOP_HOME/etc/hadoop/hdfs-site.xml hdfs-site.xml
ln -s $HADOOP_HOME/etc/hadoop/core-site.xml core-site.xml
ln -s $HBASE_HOME/conf/hbase-site.xml hbase-site.xml
ln -s $HIVE_HOME/conf/hive-site.xml hive-site.xml
ln -s $SPARK_HOME/conf/spark-defaults.conf spark-defaults.conf

Configure Environment

Modify kylin.sh:

cd $KYLIN_HOME/bin
vim kylin.sh

Need to write these dependencies to prevent subsequent errors:

export HADOOP_HOME=/opt/servers/hadoop-2.9.2
export HIVE_HOME=/opt/servers/apache-hive-2.3.9-bin
export HBASE_HOME=/opt/servers/hbase-1.3.1
export SPARK_HOME=/opt/servers/spark-2.4.5-bin-without-hadoop-scala-2.12

Fix HADOOP_CONF_DIR (common issue):

# Fix Flink YARN config
# export HADOOP_CONF_DIR=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_CLASSPATH=`hadoop classpath`

Check Dependencies

$KYLIN_HOME/bin/check-env.sh

If error, check if HADOOP_CONF_DIR is correct, then re-check.

Start Cluster

ZooKeeper

Start h121 h122 h123 cluster mode, each node runs:

zkServer.sh start

HDFS

Start h121 (check confirm):

start-dfs.sh

YARN

Start h121 (check confirm):

start-yarn.sh

HBase

Start h121 (check confirm):

start-hbase.sh

Metastore

Start on h121:

nohup hive --service metastore > /tmp/metastore.log 2>&1 &

History Server

Start on h121:

mr-jobhistory-daemon.sh start historyserver

Kylin

Start on h122:

kylin.sh start

Node Details

h121

  • Metastore
  • Zookeeper
  • HBase
  • HDFS
  • YARN
  • Hadoop

h122

  • YARN
  • ZooKeeper
  • HBase
  • Kylin
  • HDFS

h123

  • YARN
  • HBase
  • ZooKeeper
  • Hadoop
  • HDFS

Start Result

Access address:

http://h122.wzk.icu:7070/kylin/login

Login info (default uppercase):

  • Username: ADMIN
  • Password: KYLIN

Error Quick Reference

SymptomRoot Cause LocationFix
check-env.sh self-check fails,提示找不到 Hadoop 配置HADOOP_CONF_DIR points to $HADOOP_HOME not …/etc/hadoopRun self-check script output export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop && export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop && export HADOOP_CLASSPATH=\hadoop classpath`thensource /etc/profile` re-check
After Kylin starts, connection to HBase reports ZK exceptionhbase.zookeeper.quorum mistakenly written as host1:2181,host2:2181Change quorum to only hostname (like host1,host2), restart HBase and Kylin
Access :7070 no response/connection refusedKylin not pulled up successfully; or firewall/port not openedps -ef check process
Various “No FileSystem for scheme: hdfs”/config not found$KYLIN_HOME/conf missing soft links to Hadoop/HBase/Hive/Spark config directoriesCheck $KYLIN_HOME/conf, supplement soft links per article and restart
Hive Metastore connection refused/NoSuchObjectExceptionMetastore not started or 9083 occupiedCheck /tmp/metastore.log, netstat -lntp
Login failed/wrong passwordUsername password case error or custom overrideUse default uppercase ADMIN/KYLIN; if changed, confirm account strategy in conf and reset
JDK version incompatible at start (common in old stack)Using non-JDK8Switch to JDK8 and ensure JAVA_HOME and PATH point consistently

Summary

This article records in detail the minimum available deployment process of Apache Kylin 3.1.1 on Hadoop 2.9/Hive 2.3/HBase 1.3 environment:

  1. Environment preparation and version selection
  2. Soft link configuration
  3. Environment variable settings
  4. Cluster components start in order
  5. Common error排查 and fixes

Through steps in this article, can successfully deploy and run Kylin in resource-constrained three-node environment, achieve sub-second OLAP queries.