[ELK] 로그스태시 설치

설치환경

• OS : CentOS 7

---

설치

wget https://artifacts.elastic.co/downloads/logstash/logstash-7.16.2-linux-x86_64.tar.gz
--2021-12-31 08:49:32--  https://artifacts.elastic.co/downloads/logstash/logstash-7.16.2-linux-x86_64.tar.gz
Resolving artifacts.elastic.co (artifacts.elastic.co)... 34.120.127.130, 2600:1901:0:1d7::
Connecting to artifacts.elastic.co (artifacts.elastic.co)|34.120.127.130|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 365618045 (349M) [application/x-gzip]
Saving to: ‘logstash-7.16.2-linux-x86_64.tar.gz’

100%[==============================================================================================================================>] 365,618,045  127MB/s   in 2.7s

2021-12-31 08:49:36 (127 MB/s) - ‘logstash-7.16.2-linux-x86_64.tar.gz’ saved [365618045/365618045]

• wget https://artifacts.elastic.co/downloads/logstash/logstash-7.16.2-linux-x86_64.tar.gz
• tar -xvf logstash-7.16.2-linux-x86_64.tar.gz

---

환경세팅

로그스태시는 그냥 실행하게 되면 파이프라인 과정을 명시를 안해서 실행이 안됨, 해당 설정을 진행해야함

ERROR: Pipelines YAML file is empty. Location: /home/untact/logstash-7.16.2/config/pipelines.yml
usage:
  bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
  bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
  bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
  bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash --help
[2021-12-31T09:05:08,489][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
        at home.untact.logstash_minus_7_dot_16_dot_2.lib.bootstrap.environment.<main>(/home/untact/logstash-7.16.2/lib/bootstrap/environment.rb:94) ~[?:?]

• config 폴더 경로로 이동, logstash-sample.conf 복사 (cp logstash-sample.conf logstash.conf)
• 복사한 conf 파일을 수정함

---

실행

[tgyun615@logstash1 bin]$ ./logstash -f /home/tgyun615/logstash-7.16.2/config/logstash.conf
Using JAVA_HOME defined java: /usr/local/java
WARNING: Using JAVA_HOME while Logstash distribution comes with a bundled JDK.
DEPRECATION: The use of JAVA_HOME is now deprecated and will be removed starting from 8.0. Please configure LS_JAVA_HOME instead.
Sending Logstash logs to /home/untact/logstash-7.16.2/logs which is now configured via log4j2.properties
[2021-12-31T10:40:28,622][INFO ][logstash.runner          ] Log4j configuration path used is: /home/untact/logstash-7.16.2/config/log4j2.properties
[2021-12-31T10:40:28,649][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.16.2", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 Java HotSpot(TM) 64-Bit Server VM 25.291-b10 on 1.8.0_291-b10 +indy +jit [linux-x86_64]"}
[2021-12-31T10:40:29,168][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-12-31T10:40:31,456][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2021-12-31T10:40:32,653][INFO ][org.reflections.Reflections] Reflections took 129 ms to scan 1 urls, producing 119 keys and 417 values
[2021-12-31T10:40:34,237][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://34.64.254.12:9200"]}
[2021-12-31T10:40:34,779][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://34.64.254.12:9200/]}}
[2021-12-31T10:40:35,269][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://34.64.254.12:9200/"}
[2021-12-31T10:40:35,301][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (7.16.2) {:es_version=>7}
[2021-12-31T10:40:35,305][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2021-12-31T10:40:35,424][WARN ][logstash.outputs.elasticsearch][main] Configuration is data stream compliant but due backwards compatibility Logstash 7.x will not assume writing to a data-stream, default behavior will change on Logstash 8.0 (set `data_stream => true/false` to disable this warning)
[2021-12-31T10:40:35,438][WARN ][logstash.outputs.elasticsearch][main] Configuration is data stream compliant but due backwards compatibility Logstash 7.x will not assume writing to a data-stream, default behavior will change on Logstash 8.0 (set `data_stream => true/false` to disable this warning)
[2021-12-31T10:40:35,565][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2021-12-31T10:40:35,595][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/home/untact/logstash-7.16.2/config/logstash.conf"], :thread=>"#<Thread:0x6d0b47c6 run>"}
[2021-12-31T10:40:35,673][INFO ][logstash.outputs.elasticsearch][main] Installing Elasticsearch template {:name=>"logstash"}
[2021-12-31T10:40:36,219][INFO ][logstash.outputs.elasticsearch][main] Created rollover alias {:name=>"<logstash-{now/d}-000001>"}
[2021-12-31T10:40:36,259][INFO ][logstash.outputs.elasticsearch][main] Installing ILM policy {"policy"=>{"phases"=>{"hot"=>{"actions"=>{"rollover"=>{"max_size"=>"50gb", "max_age"=>"30d"}}}}}} {:name=>"logstash-policy"}
[2021-12-31T10:40:36,909][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.31}
[2021-12-31T10:40:37,358][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2021-12-31T10:40:37,392][INFO ][logstash.inputs.tcp      ][main][a999fdc1607e393f8ff5f14f64dfd8b4149a621d115929d27c601b7fb9e48c6a] Starting tcp input listener {:address=>"0.0.0.0:9900", :ssl_enable=>false}
[2021-12-31T10:40:37,547][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}

• 정상구동 확인후 연결된 키바나의 DevTool에서 GET _cat/indices로 구동날짜의 로그스태시를 확인