Big Data 185 - Logstash 7 Getting Started: stdin/file Collection, sincedb, start_position & Error Quick Reference

1. Logstash Architecture

Logstash pipeline three stages:

Input → Filter → Output

2. stdin Input

bin/logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'

3. file Input

3.1 Basic Config

input {
  file {
    path => "/var/log/nginx/access.log"
    type => "nginx"
    start_position => "end"
  }
}

output {
  stdout { codec => rubydebug }
}

3.2 Key Parameters

4. sincedb Mechanism

• Records each file’s read offset
• Resume from breakpoint, prevent duplicate collection
• Associated with file path + inode

4.1 start_position Effect Conditions

• Only effective for first-seen files
• If sincedb already has record, start_position is ignored

5. Error Quick Reference

6. Summary

• Logstash pipeline: Input → Filter → Output
• sincedb implements breakpoint resume
• start_position only effective for new files