Big Data 188 - Logstash Output Plugin Practice

1. Output Plugin Overview

Output is the final stage of Logstash pipeline, responsible for outputting processed data to target system.

2. stdout Output

2.1 rubydebug codec

output {
  stdout {
    codec => rubydebug
  }
}

2.2 Usage

• Debug verification
• Local development debugging

3. file Output

3.1 Basic Config

output {
  file {
    path => "/var/log/logstash/%{type}-%{+YYYY.MM.dd}.log"
  }
}

3.2 Dynamic Path Template

Variable	Description
%{+YYYY.MM.dd}	Time format
%{type}	Event type
%{host}	Hostname

4. Elasticsearch Output

4.1 Basic Config

output {
  elasticsearch {
    hosts => ["es1:9200", "es2:9200"]
    index => "logstash-%{type}-%{+YYYY.MM.dd}"
  }
}

4.2 Batch Submit

elasticsearch {
  hosts => ["es:9200"]
  manage_template => false
  index => "my-index"
  document_type => "_doc"
}

4.3 Retry Mechanism

• Default retry 3 times
• Can configure retry_max_items

5. Advanced Features

5.1 Load Balancing

output {
  elasticsearch {
    hosts => ["es1:9200", "es2:9200", "es3:9200"]
  }
}

5.2 Conditional Routing

output {
  if [type] == "nginx" {
    elasticsearch {
      hosts => ["es:9200"]
      index => "nginx-%{+YYYY.MM.dd}"
    }
  } else if [type] == "app" {
    elasticsearch {
      hosts => ["es:9200"]
      index => "app-%{+YYYY.MM.dd}"
    }
  }
}

5.3 Multi-output Parallel

output {
  stdout { codec => rubydebug }
  file {
    path => "/var/log/archive/%{+YYYY.MM.dd}.log"
  }
}

6. Common Issues

Issue	Cause	Solution
Slow data write	Small bulk size	Increase flush_size
Write failure	ES unreachable	Check network/ES status
Index template conflict	manage_template=true	Set to false

7. Summary

• stdout for debugging
• file for archiving
• Elasticsearch for search and analysis
• Conditional routing implements data shunting