TL;DR

  • Scenario: Three-node Elasticsearch cluster deployment to achieve high availability and horizontal scaling
  • Conclusion: Cluster mode provides data sharding and replica mechanism to ensure high availability
  • Output: Complete cluster deployment flow, config explanation, error quick reference

Version Matrix and Environment Prep

ComponentVersion
Elasticsearch7.3.0 (linux-x86_64 tar.gz)
Nodesh121 / h122 / h123
Running Useres_server (non-root)

Directory and Permission Settings

Execute on Three Machines

mkdir -p /opt/servers/es/{data,logs}
chown -R es_server /opt/servers/es
chown -R es_server /opt/servers/elasticsearch-7.3.0

User Creation and Permissions

useradd es_server
passwd es_server

# Add in /etc/sudoers: es ALL=(ALL) ALL

Core Config elasticsearch.yml

h121 Node Config

cluster.name: wzkicu-es
node.name: h121.wzk.icu
path.data: /opt/servers/es/data
path.logs: /opt/servers/es/logs
network.host: h121.wzk.icu
network.bind_host: h121.wzk.icu
network.publish_host: 114.115.221.144
http.port: 9200
cluster.initial_master_nodes: ["h121.wzk.icu","h122.wzk.icu","h123.wzk.icu"]
discovery.seed_hosts: ["h121.wzk.icu", "h122.wzk.icu","h123.wzk.icu"]

h122 Node Config

cluster.name: wzkicu-es
node.name: h122.wzk.icu
path.data: /opt/servers/es/data
path.logs: /opt/servers/es/logs
network.host: h122.wzk.icu
network.bind_host: h122.wzk.icu
network.publish_host: 114.115.221.145
http.port: 9200
cluster.initial_master_nodes: ["h121.wzk.icu","h122.wzk.icu","h123.wzk.icu"]
discovery.seed_hosts: ["h121.wzk.icu", "h122.wzk.icu","h123.wzk.icu"]

h123 Node Config

cluster.name: wzkicu-es
node.name: h123.wzk.icu
path.data: /opt/servers/es/data
path.logs: /opt/servers/es/logs
network.host: h123.wzk.icu
network.bind_host: h123.wzk.icu
network.publish_host: 114.115.221.146
http.port: 9200
cluster.initial_master_nodes: ["h121.wzk.icu","h122.wzk.icu","h123.wzk.icu"]
discovery.seed_hosts: ["h121.wzk.icu", "h122.wzk.icu","h123.wzk.icu"]

System Parameter Config

Modify sysctl.conf

vim /etc/sysctl.conf

# Add
vm.max_map_count=655360

Apply:

sysctl -p

Modify limits.conf

vim /etc/security/limits.conf

# Add
* soft nofile 65536
* hard nofile 65536
* soft nproc 4096
* hard nproc 4096

JVM Memory Config

Modify jvm.options

vim /opt/servers/elasticsearch-7.3.0/config/jvm.options

# Set
-Xms2g
-Xmx2g

Distribution and Startup

Distribute Config

# Distribute configured ES to each node
scp -r /opt/servers/elasticsearch-7.3.0 es_server@h122.wzk.icu:/opt/servers/
scp -r /opt/servers/elasticsearch-7.3.0 es_server@h123.wzk.icu:/opt/servers/

Start on Each Node

su es_server
/opt/servers/elasticsearch-7.3.0/bin/elasticsearch -d

Verify Cluster Status

Using Elasticsearch Head Tool

Access http://h121.wzk.icu:9200/_plugin/head/ or use curl:

# Cluster health
curl -XGET 'http://h121.wzk.icu:9200/_cluster/health?pretty'

# Node list
curl -XGET 'http://h121.wzk.icu:9200/_cat/nodes?v'

# Shard status
curl -XGET 'http://h121.wzk.icu:9200/_cat/shards?v'

Error Quick Reference

SymptomRoot CauseFix
vm.max_map_count too lowKernel parameter not setAdd vm.max_map_count=655360 in /etc/sysctl.conf
max file descriptors too lowlimits not appliedSet nofile 65536 in limits.conf
ES refuses root runStarted as rootSwitch to es_server user
Node starts but doesn’t join cluster9300 not accessible/hostname resolution failedCheck discovery.seed_hosts and network
node name already in useMultiple nodes with duplicate node.nameAssign unique name to each node
Cluster RedShard allocation failedCheck disk space, node status
Split-brain issueMaster node election disagreementAdjust minimum_master_nodes