Big Data 189 - Nginx JSON Logs to ELK

1. Overall Architecture

Nginx → Filebeat → Kafka → Logstash → Elasticsearch → Kibana

This practice uses ZK + Kafka for buffering and decoupling.

2. Nginx Config

2.1 Configure JSON Format Logs

log_format json escape=json '{'
    '"@timestamp":"$time_iso8601",'
    '"request_time":"$request_time",'
    '"status": "$status",'
    '"request_uri": "$request_uri",'
    '"remote_addr": "$remote_addr",'
    '"ua": "$http_user_agent"'
'}';

access_log /var/log/nginx/access.log json;

2.2 Listen Port

server {
    listen 8888;
    server_name _;
    location / {
        root /usr/share/nginx/html;
    }
}

3. Zookeeper Cluster

Start on h121/h122/h123 three nodes:

./bin/zookeeper-server-start.sh config/zookeeper.properties

4. Kafka Cluster

Start Kafka service:

./bin/kafka-server-start.sh config/server.properties

Create Topic:

./bin/kafka-topics.sh --create --topic nginx-log --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092

5. Elasticsearch 7.3.0

Start ES cluster (h121/h122/h123):

./bin/elasticsearch -d

6. Kibana 7.3.0

./bin/kibana

Access http://localhost:5601

7. Error Quick Reference

Issue	Possible Cause	Solution
Kafka startup fails	ZK not started	Start ZK first
ES cluster unhealthy	Node network not connected	Check network config
Kibana can’t access	ES not started	Check ES status
Log format error	Nginx config error	Check log_format

8. Summary

• Configure Nginx json format logs
• ZK + Kafka for buffering and decoupling
• ES storage + Kibana visualization
• Complete ELK pipeline practice