Bii o ṣe le Fi Elasticsearch sori ẹrọ, Logstash, ati Kibana (ELK Stack) lori CentOS/RHEL 7
Ti o ba jẹ eniyan ti o jẹ, tabi ti wa ni iṣaaju, ni idiyele ti ṣayẹwo ati itupalẹ awọn iwe akọọlẹ eto ni Linux, o mọ kini alaburuku ti iṣẹ-ṣiṣe le di ti o ba n ṣetọju awọn iṣẹ lọpọlọpọ nigbakanna.
Ni awọn ọjọ ti o ti kọja, iṣẹ-ṣiṣe naa ni lati ṣee ṣe ni ọwọ pẹlu ọwọ, pẹlu iru log kọọkan ni a mu lọna lọtọ. Ni akoko, idapọ ti Elasticsearch, Logstash, ati Kibana ni ẹgbẹ olupin, pẹlu Filebeat ni ẹgbẹ alabara, jẹ ki iṣẹ ṣiṣe nira lẹẹkan dabi ẹni pe o rin ni papa itura loni.
Awọn paati mẹta akọkọ dagba ohun ti a pe ni akopọ ELK, eyiti idi akọkọ rẹ ni lati ṣajọ awọn akọọlẹ lati awọn olupin pupọ ni akoko kanna (eyiti a tun mọ ni gedu ti aarin).
Ifilelẹ oju opo wẹẹbu ti o da lori Java ti o jẹ ki o ṣe ayewo awọn àkọọlẹ yarayara ni wiwo kan fun afiwe ti o rọrun ati laasigbotitusita. Awọn akọọlẹ alabara wọnyi ni a firanṣẹ si olupin aringbungbun nipasẹ Filebeat, eyiti o le ṣe apejuwe bi oluṣowo gbigbe ọkọ wọle.
Jẹ ki a wo bi gbogbo awọn ege wọnyi ṣe baamu pọ. Aaye idanwo wa yoo ni awọn ẹrọ wọnyi:
Central Server: CentOS 7 (IP address: 192.168.0.29). 2 GB of RAM. Client #1: CentOS 7 (IP address: 192.168.0.100). 1 GB of RAM. Client #2: Debian 8 (IP address: 192.168.0.101). 1 GB of RAM.
Jọwọ ṣe akiyesi pe awọn iye Ramu ti a pese nihin kii ṣe awọn ibeere to muna, ṣugbọn awọn iye ti a ṣe iṣeduro fun imuse aṣeyọri ti akopọ ELK lori olupin aringbungbun. Ramu ti o kere si lori awọn alabara kii yoo ṣe iyatọ pupọ, ti eyikeyi, ni gbogbo.
Fifi ELK Stack sori Server
Jẹ ki a bẹrẹ nipa fifi ipilẹ ELK sori olupin naa, pẹlu alaye ṣoki lori ohun ti paati kọọkan nṣe:
- Elasticsearch tọju awọn akọọlẹ ti awọn alabara firanṣẹ.
- Awọn ilana Logstash ṣe awọn ilana wọnyẹn.
- Kibana pese wiwo wẹẹbu ti yoo ṣe iranlọwọ fun wa lati ṣayẹwo ati ṣe itupalẹ awọn iwe-akọọlẹ.
Fi awọn idii wọnyi sii sori olupin aringbungbun. Ni akọkọ, a yoo fi Java 8 JDK sori ẹrọ (imudojuiwọn 102, tuntun ni akoko kikọ yi), eyiti o jẹ igbẹkẹle ti awọn paati ELK.
O le fẹ lati ṣayẹwo akọkọ ni oju-iwe awọn igbasilẹ Java nibi lati rii boya imudojuiwọn tuntun wa.
# yum update # cd /opt # wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u102-b14/jre-8u102-linux-x64.rpm" # rpm -Uvh jre-8u102-linux-x64.rpm
Akoko lati ṣayẹwo boya fifi sori ẹrọ pari ni aṣeyọri:
# java -version
Lati fi awọn ẹya tuntun ti Elasticsearch, Logstash, ati Kibana sori ẹrọ, a ni lati ṣẹda awọn ibi ipamọ fun yum pẹlu ọwọ bi atẹle:
1. Ṣe akowọle bọtini GPG ti Elasticsearch ni gbangba si oluṣakoso package rpm:
# rpm --import http://packages.elastic.co/GPG-KEY-elasticsearch
2. Fi awọn ila wọnyi si faili iṣeto ni ibi ipamọ elasticsearch.repo :
[elasticsearch] name=Elasticsearch repository baseurl=http://packages.elastic.co/elasticsearch/2.x/centos gpgcheck=1 gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch enabled=1
3. Fi sori ẹrọ package Elasticsearch.
# yum install elasticsearch
Nigbati fifi sori ba ti pari, iwọ yoo ti ṣetan lati bẹrẹ ati mu ṣiṣẹ elasticsearch:
4. Bẹrẹ ki o mu iṣẹ naa ṣiṣẹ.
# systemctl daemon-reload # systemctl enable elasticsearch # systemctl start elasticsearch
5. Gba ijabọ laaye nipasẹ ibudo TCP 9200 ninu ogiriina rẹ:
# firewall-cmd --add-port=9200/tcp # firewall-cmd --add-port=9200/tcp --permanent
6. Ṣayẹwo ti Elasticsearch ba dahun si awọn ibeere ti o rọrun lori HTTP:
# curl -X GET http://localhost:9200
Ijade ti aṣẹ loke yẹ ki o jẹ iru si:
Rii daju pe o pari awọn igbesẹ ti o wa loke lẹhinna tẹsiwaju pẹlu Logstash. Niwọn igba ti Logstash ati Kibana pin bọtini Elasticsearch GPG, ko si iwulo lati tun gbe wọle ṣaaju fifi awọn idii sii.
7. Fi awọn ila wọnyi si faili iṣeto iṣeto ibi ipamọ logstash.repo :
[logstash] name=Logstash baseurl=http://packages.elasticsearch.org/logstash/2.2/centos gpgcheck=1 gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch enabled=1
8. Fi package Logstash sii:
# yum install logstash
9. Ṣafikun ijẹrisi SSL kan ti o da lori adirẹsi IP ti olupin ELK ni ila atẹle ni isalẹ apakan [v3_ca] ni apakan /etc/pki/tls/openssl.cnf:
[ v3_ca ] subjectAltName = IP: 192.168.0.29
10. Ṣe ijẹrisi ijẹrisi ti ara ẹni ti o wulo fun awọn ọjọ 365:
# cd /etc/pki/tls # openssl req -config /etc/pki/tls/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
11. Ṣe atunto igbewọle Logstash, iṣẹjade, ati awọn faili idanimọ:
Input: Ṣẹda /etc/logstash/conf.d/input.conf ki o fi awọn ila wọnyi sinu. Eyi jẹ pataki fun Logstash si\"kọ ẹkọ" bii o ṣe le ṣe ilana awọn lilu ti n bọ lati ọdọ awọn alabara. Rii daju pe ọna si ijẹrisi ati bọtini baamu awọn ọna ti o tọ bi a ti ṣe ilana ni igbesẹ ti tẹlẹ:
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
O wu ( /etc/logstash/conf.d/output.conf ) faili:
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
Àlẹmọ ( /etc/logstash/conf.d/filter.conf ) faili. A yoo wọle awọn ifiranṣẹ syslog fun ayedero:
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGLINE}" }
}
date {
match => [ "timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
12. Daju awọn faili iṣeto Logstash.
# service logstash configtest
13. Bẹrẹ ati mu ṣiṣẹ logstash:
# systemctl daemon-reload # systemctl start logstash # systemctl enable logstash
14. Ṣe atunto ogiriina lati gba Logstash laaye lati gba awọn iwe lati ọdọ awọn alabara (ibudo TCP 5044):
# firewall-cmd --add-port=5044/tcp # firewall-cmd --add-port=5044/tcp --permanent
14. Fi sii awọn ila wọnyi si faili iṣeto ni ibi ipamọ kibana.repo :
[kibana] name=Kibana repository baseurl=http://packages.elastic.co/kibana/4.4/centos gpgcheck=1 gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch enabled=1
15. Fi package Kibana sii:
# yum install kibana
16. Bẹrẹ ati mu Kibana ṣiṣẹ.
# systemctl daemon-reload # systemctl start kibana # systemctl enable kibana
17. Rii daju pe o le wọle si wiwo wẹẹbu Kibana lati kọmputa miiran (gba laaye ijabọ lori ibudo TCP 5601):
# firewall-cmd --add-port=5601/tcp # firewall-cmd --add-port=5601/tcp --permanent
18. Ifilole Kibana ( http://192.168.0.29:5601 ) lati ṣayẹwo pe o le wọle si wiwo wẹẹbu:
A yoo pada si ibi lẹhin ti a ti fi sii ati tunto Filebeat lori awọn alabara.
Fi Filebeat sori Awọn olupin Onibara
A yoo fihan ọ bi o ṣe le ṣe fun Onibara # 1 (tun ṣe fun Onibara # 2 lẹhinna, awọn ọna iyipada ti o ba wulo fun pinpin rẹ).
1. Daakọ ijẹrisi SSL lati ọdọ olupin si awọn alabara:
# scp /etc/pki/tls/certs/logstash-forwarder.crt [email :/etc/pki/tls/certs/
2. Ṣe agbewọle bọtini GPG ti Elasticsearch ni gbangba si oluṣakoso package rpm:
# rpm --import http://packages.elastic.co/GPG-KEY-elasticsearch
3. Ṣẹda ibi ipamọ kan fun Filebeat ( /etc/yum.repos.d/filebeat.repo ) ni awọn ipinpinpin orisun CentOS:
[filebeat] name=Filebeat for ELK clients baseurl=https://packages.elastic.co/beats/yum/el/$basearch enabled=1 gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch gpgcheck=1
4. Ṣe atunto orisun lati fi Filebeat sori Debian ati awọn itọsẹ rẹ:
# aptitude install apt-transport-https # echo "deb https://packages.elastic.co/beats/apt stable main" > /etc/apt/sources.list.d/filebeat.list # aptitude update
5. Fi package Filebeat sii:
# yum install filebeat [On CentOS and based Distros] # aptitude install filebeat [On Debian and its derivatives]
6. Bẹrẹ ki o mu Filebeat ṣiṣẹ:
# systemctl start filebeat # systemctl enable filebeat
Ọrọ ti iṣọra nibi. Iṣeto faili Faili ti wa ni fipamọ ni faili YAML kan, eyiti o nilo ifunni ti o muna. Ṣọra pẹlu eyi bi o ṣe ṣatunkọ /etc/filebeat/filebeat.yml bi atẹle:
- Labẹ awọn ipa-ọna, tọka iru awọn faili igbasilẹ wo ni o yẹ ki o wa ni "" firanṣẹ "si olupin ELK.
- Labẹ awọn oluyẹwo:
input_type: log document_type: syslog
- Labẹ iṣẹjade:
- Uncomment laini ti o bẹrẹ pẹlu logstash.
- Ṣe afihan adirẹsi IP ti olupin ELK ati ibudo rẹ nibiti Logstash n tẹtisi awọn ọmọ-ogun.
- Rii daju pe ọna si ijẹrisi naa tọka si faili gangan ti o ṣẹda ni Igbese I (apakan Logstash) loke.
Awọn igbesẹ ti o wa loke ni a sapejuwe ninu aworan atẹle:
Ṣafipamọ awọn ayipada, lẹhinna tun bẹrẹ Filebeat lori awọn alabara:
# systemctl restart filebeat
Lọgan ti a ba ti pari awọn igbesẹ ti o wa loke lori awọn alabara, ni ọfẹ lati tẹsiwaju.
Lati rii daju pe awọn akọọlẹ lati ọdọ awọn alabara le firanṣẹ ati gba ni aṣeyọri, ṣiṣe aṣẹ wọnyi lori olupin ELK:
# curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'
Imujade yẹ ki o jọra si (ṣakiyesi bawo ni a ṣe n gba awọn ifiranṣẹ lati/var/log/awọn ifiranṣẹ ati/var/log/aabo lati ọdọ onibara1 ati client2):
Bibẹẹkọ, ṣayẹwo faili iṣeto faili Filebeat fun awọn aṣiṣe.
# journalctl -xe
lẹhin igbidanwo lati tun bẹrẹ Filebeat yoo tọka si ila (s) ti o ṣẹ.
Lẹhin ti a ti ṣayẹwo pe awọn iwe aṣẹ ti wa ni gbigbe nipasẹ awọn alabara ati gba ni aṣeyọri lori olupin naa. Ohun akọkọ ti a ni lati ṣe ni Kibana n ṣatunṣe apẹẹrẹ itọka ati ṣeto bi aiyipada.
O le ṣapejuwe itọka kan bi ibi ipamọ data ni kikun ni ibatan ibi ipamọ data ibatan. A yoo lọ pẹlu filebeat- * (tabi o le lo awọn ilana wiwa ti o daju julọ bi a ti ṣalaye ninu iwe aṣẹ osise).
Tẹ filebeat- * sii ni orukọ Atọka tabi aaye apẹrẹ ati lẹhinna tẹ Ṣẹda:
Jọwọ ṣe akiyesi pe ao gba ọ laaye lati tẹ awọn ilana wiwa didara diẹ sii nigbamii. Nigbamii, tẹ irawọ inu onigun merin alawọ lati tunto rẹ bi apẹẹrẹ itọka aiyipada:
Lakotan, ninu akojọ Awari iwọ yoo wa ọpọlọpọ awọn aaye lati ṣafikun si ijabọ iwoye log. O kan rababa lori wọn ki o tẹ Fikun-un:
Awọn abajade yoo han ni agbegbe aarin ti iboju bi a ti han loke. Ni idaniloju lati ṣere ni ayika (ṣafikun ati yọ awọn aaye kuro ninu ijabọ log) lati faramọ Kibana.
Nipa aiyipada, Kibana yoo ṣe afihan awọn igbasilẹ ti a ti ṣiṣẹ lakoko awọn iṣẹju 15 to ṣẹṣẹ (wo igun apa ọtun ni oke) ṣugbọn o le yi ihuwasi yẹn pada nipa yiyan aaye akoko miiran:
Akopọ
Ninu àpilẹkọ yii a ti ṣalaye bi a ṣe le ṣeto akopọ ELK lati ṣajọ awọn akọọlẹ eto ti a firanṣẹ nipasẹ awọn alabara meji, CentOS 7 ati awọn ẹrọ Debian 8 kan.
Bayi o le tọka si iwe aṣẹ Elasticsearch osise ati rii awọn alaye diẹ sii lori bii o ṣe le lo iṣeto yii lati ṣayẹwo ati ṣe itupalẹ awọn iwe rẹ daradara siwaju sii.
Ti o ba ni ibeere eyikeyi, ma ṣe ṣiyemeji lati beere. A n reti lati gbọ lati ọdọ rẹ.