CentOS7下Elastic Stack 5.0日志分析系统搭建 - 24小时 - 剑客 -「科技引领未来」

CentOS7下Elastic Stack 5.0日志分析系统搭建

一、概述 Elasticsearch是个开源分布式搜索引擎,它的特点有:分布式,零配置,自动发现,索引自动分片,索引副本机制,restful风格接

一、概述

Elasticsearch是个开源分布式搜索引擎,它的特点有:分布式,零配置,自动发现,索引自动分片,索引副本机制,restful风格接口,多数据源,自动搜索负载等。

Logstash是一个开源的用于收集,分析和存储日志的工具。

Kibana 也是一个开源和免费的工具,Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面,可以汇总、分析和搜索重要数据日志。

  Beats是elasticsearch公司开源的一款采集系统监控数据的代理agent,是在被监控服务器上以客户端形式运行的数据收集器的统称,可以直接把数据发送给Elasticsearch或者通过Logstash发送给Elasticsearch,然后进行后续的数据分析活动。Beats由如下组成:

1.Packetbeat:是一个网络数据包分析器,用于监控、收集网络流量信息,

Packetbeat嗅探服务器之间的流量,解析应用层协议,并关联到消息的处理,                                      其支  ICMP (v4 and v6)、DNS、HTTP、Mysql、PostgreSQL、Redis、

MongoDB、Memcache等协议;

2. Filebeat:用于监控、收集服务器日志文件,其已取代 logstash forwarder;

3. Metricbeat:可定期获取外部系统的监控指标信息,其可以监控、收集

Apache、HAProxy、MongoDB、MySQL、Nginx、PostgreSQL、

Redis、System、Zookeeper等服务;

4. Winlogbeat:用于监控、收集Windows系统的日志信息;

5. Create your own Beat:自定义beat ,如果上面的指标不能满足需求,elasticsarch鼓励开发者          使用go语言,扩展实现自定义的beats,只需要按照模板,实现监控的输入,日志,输出等即可。

CentOS7下Elastic Stack 5.0日志分析系统搭建

Beats 将搜集到的数据发送到 Logstash,经 Logstash 解析、过滤后,将其发送到 Elasticsearch 存储,并由 Kibana 呈现给用户。

Beats 作为日志搜集器没有Logstash 作为日志搜集器消耗资源,解决了 Logstash 在各服务器节点上占用系统资源高的问题。

Elastic Stack官方下载地址: https://www.elastic.co/downloads

二、开源实时日志分析系统Elastic Stack 5.0部署:

A.安装依赖包JDK

①关闭防火墙

# cat /etc/selinux/config |grep -v "#"

SELINUX=disabled ##关闭selinux

SELINUXTYPE=targeted

# systemctl stop firewalld ##关闭防火墙

②下载安装JDK

# wget http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm  

# yum install jre ##这里我只安装的是jre环境,一样可用

# java -version

openjdk version "1.8.0_111"

OpenJDK Runtime Environment (build 1.8.0_111-b15)

OpenJDK 64-Bit Server VM (build 25.111-b15, mixed mode

CentOS7下Elastic Stack 5.0日志分析系统搭建

B.安装Elasticsearch

①下载安装elasticsearch

# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

# cat > /etc/yum.repos.d/elasticsearch.repo <<EOF

[elasticsearch-5.x]

name=Elasticsearch repository for 5.x packages

baseurl=https://artifacts.elastic.co/packages/5.x/yum

gpgcheck=1

gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch

enabled=1

autorefresh=1

type=rpm-md

EOF

# yum install elasticsearch

# systemctl start elasticsearch

# systemctl status elasticsearch

# /usr/share/elasticsearch/bin/elasticsearch -V ##查看elasticsearch版本

Version: 5.0.1, Build: 080bb47/2016-11-11T22:08:49.812Z, JVM: 1.8.0_111

注:安装后出现如下问题,致使 elasticsearch 无法启动

# systemctl status elasticsearch

OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should conf...CThreads=N

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000085330000, 2060255232, 0) failed; error='Cannot a ...'(errno=12)

# There is insufficient memory for the Java Runtime Environment to continue.

# Native memory allocation (mmap) failed to map 2060255232 bytes for committing reserved memory.

# An error report file with more information is saved as:

# /tmp/hs_err_pid11084.log

解决:

配置elasticsearch下的jvm.options:

# vi /etc/elasticsearch/jvm.options

-Xms4g ##启用如下两项

-Xmx4g

##-Xms2g ##关闭如下两项

##-Xmx2g

②ElasticSearch默认的对外服务的HTTP端口是9200,节点间交互的TCP端口是9300。

# ss -tlnp |grep -E '9200|9300'

CentOS7下Elastic Stack 5.0日志分析系统搭建

③测试服务

# curl -X GET http://localhost:9200

{

"name" : "XVY0Ovb",

"cluster_name" : "elasticsearch",

"cluster_uuid" : "tR_H9avzT6Kf4hXWTIfWyA",

"version" : {

"number" : "5.0.1",

"build_hash" : "080bb47",

"build_date" : "2016-11-11T22:08:49.812Z",

"build_snapshot" : false,

"lucene_version" : "6.2.1"

},

"tagline" : "You Know, for Search"

}

CentOS7下Elastic Stack 5.0日志分析系统搭建

也可以使用如下命令测试

# curl -i -XGET 'localhost:9200/'

HTTP/1.1 200 OK

content-type: application/json; charset=UTF-8

content-length: 327

{

"name" : "XVY0Ovb",

"cluster_name" : "elasticsearch",

"cluster_uuid" : "tR_H9avzT6Kf4hXWTIfWyA",

"version" : {

"number" : "5.0.1",

"build_hash" : "080bb47",

"build_date" : "2016-11-11T22:08:49.812Z",

"build_snapshot" : false,

"lucene_version" : "6.2.1"

},

"tagline" : "You Know, for Search"

}

CentOS7下Elastic Stack 5.0日志分析系统搭建

C.安装Logstash

# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

# cat > /etc/yum.repos.d/logstash.repo <<EOF

[logstash-5.x]

name=Elastic repository for 5.x packages

baseurl=https://artifacts.elastic.co/packages/5.x/yum

gpgcheck=1

gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch

enabled=1

autorefresh=1

type=rpm-md

EOF

# yum install logstash

# systemctl start logstash

# systemctl status logstash

# /usr/share/logstash/bin/logstash -V ##查看logstash版本

logstash 5.0.1

D.安装Kibana

①安装Kibana

# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

# cat > /etc/yum.repos.d/kibana.repo <<EOF

[kibana-5.x]

name=Kibana repository for 5.x packages

baseurl=https://artifacts.elastic.co/packages/5.x/yum

gpgcheck=1

gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch

enabled=1

autorefresh=1

type=rpm-md

EOF

# yum install kibana

# systemctl start kibana

# systemctl status kibana

# /usr/share/kibana/bin/kibana -V ##查看kibana版本

5.0.1

②只需更改如下配置

# cat /etc/kibana/kibana.yml |grep -v "#"

server.host: "192.168.147.128"

③检测服务

# ss -tlnp|grep 5601                ##Kibana默认进程名:node ,端口5601

CentOS7下Elastic Stack 5.0日志分析系统搭建

浏览器输入 http://localhost:5601

CentOS7下Elastic Stack 5.0日志分析系统搭建

E、Beats安装部署

a. 安装部署Filebeat

CentOS7下Elastic Stack 5.0日志分析系统搭建

①安装Filebeat

# curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.0.1-x86_64.rpm

# rpm -ivh filebeat-5.0.1-x86_64.rpm

# systemctl start filebeat

# systemctl status filebeat

# filebeat.sh -version ##查看版本

filebeat version 5.0.1 (amd64), libbeat 5.0.1

②配置Filebeat

# cp /etc/filebeat

# cp filebeat.yml filebeat.yml.bak

# vi /etc/filebeat/filebeat.yml ##配置filebeat

#============= Filebeat prospectors ===============

filebeat.prospectors:

- input_type: log

paths:

- /var/log/*.log

#==================== Outputs =====================

#------------- Elasticsearch output ---------------

output.elasticsearch:

# Array of hosts to connect to.

hosts: ["localhost:9200"]

#---------------- Logstash output -----------------

output.logstash:

# The Logstash hosts

hosts: ["localhost:5043"] ##只需配置该处,其他默认即可

# filebeat.sh -configtest -e ##验证配置文件

③配置Logstash

# cat /etc/logstash/conf.d/logstash.conf 

input {

beats {

port => "5043"

}

}

filter {

grok {

match => { "message" => "%{COMBINEDAPACHELOG}"}

}

geoip {

source => "clientip"

}

}

output {

elasticsearch {

hosts => [ "localhost:9200" ]

}

}

# systemctl restart logstash ##重启logstash

# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --config.test_and_exit ##验证配置文件

Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties

Configuration OK

# ss -tlnp|grep -E '5043|9600'

CentOS7下Elastic Stack 5.0日志分析系统搭建

④配置kibana

浏览器输入http://localhost:5601,配置filebeat的索引(只需输入filebeat-*即可)。

CentOS7下Elastic Stack 5.0日志分析系统搭建 在第一个框里输入filebeat-*后稍等片刻,kibana会自动识别,OK后下面的按钮会由灰色变为可操控的按钮"Create",如上图所示。点击该按钮后,最后就会呈现如下图所示:

CentOS7下Elastic Stack 5.0日志分析系统搭建 我们再会过头新建logstash的索引,浏览器输入 http://localhost:5601 ,点击左边栏的”Management”===>然后点击“index Patterns”===>

CentOS7下Elastic Stack 5.0日志分析系统搭建 ===>然后点击“Add New”===>

CentOS7下Elastic Stack 5.0日志分析系统搭建

===>点击“Crete”按钮创建logstash索引,创建完成后即会展现如下图所示:

CentOS7下Elastic Stack 5.0日志分析系统搭建

b.安装部署Packetbeat

①安装配置Packetbeat

# yum install libpcap

# curl -L -O https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-5.0.1-x86_64.rpm

# rpm -ivh packetbeat-5.0.1-x86_64.rpm

# cat /etc/packetbeat/packetbeat.yml

==================== Network device ===================

packetbeat.interfaces.device: any ##捕获所有消息发送或接收的网络接口

======================== Flows ========================

packetbeat.flows:

timeout: 30s

period: 10s

================== Transaction protocols ==============

##如下是packetbeat默认支持的主要协议及端口

packetbeat.protocols.icmp:

enabled: true

packetbeat.protocols.amqp:

ports: [5672]

packetbeat.protocols.cassandra:

ports: [9042]

packetbeat.protocols.dns:

ports: [53]

include_authorities: true

include_additionals: true

packetbeat.protocols.http:

ports: [80, 8080, 8000, 5000, 8002]

packetbeat.protocols.memcache:

ports: [11211]

packetbeat.protocols.mysql:

ports: [3306]

packetbeat.protocols.pgsql:

ports: [5432]

packetbeat.protocols.redis:

ports: [6379]

packetbeat.protocols.thrift:

ports: [9090]

packetbeat.protocols.mongodb:

ports: [27017]

packetbeat.protocols.nfs:

ports: [2049]

========================= General =========================

========================= Outputs =========================

------------------- Elasticsearch output ------------------

output.elasticsearch:

hosts: ["localhost:9200"]

--------------------- Logstash output ---------------------

output.logstash:

hosts: ["localhost:5043"] ##只需配置该处,其他默认即可

============================= Logging =====================


②验证配置并启动packetbeat

# packetbeat.sh -version                         ##查看packetbeat版本

packetbeat version 5.0.1 (amd64), libbeat 5.0.1

# packetbeat.sh -configtest -e ##测试配置文件

......

Config OK

# systemctl start packetbeat

# systemctl status packetbeat


③配置Kibana(新建packetbeat索引)

http://localhost:5601 下新建索引页面输入“packetbeat-*”,之后kibana会自动更新,在“Time-field name”下面的三个选项中选择“@timestamp”,最后点击“Create”创建即可。

CentOS7下Elastic Stack 5.0日志分析系统搭建

创建完成后,kibana显示如下:

CentOS7下Elastic Stack 5.0日志分析系统搭建

c.安装部署Metricbeat

①安装配置metricbeat

# curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-5.0.1-x86_64.rpm

# rpm -ivh metricbeat-5.0.1-x86_64.rpm

# cat /etc/metricbeat/metricbeat.yml

================= Modules configuration =================

metricbeat.modules:

---------------------- System Module ---------------------

- module: system

metricsets:

- cpu

- load

- filesystem

- fsstat

- memory

- network

- process

enabled: true

period: 10s

processes: ['.*']

========================= General =======================

========================= Outputs =======================

------------------- Elasticsearch output ----------------

output.elasticsearch:

hosts: ["localhost:9200"]

--------------------- Logstash output -------------------

output.logstash:

hosts: ["localhost:5043"] ##只需配置该项,其他默认即可

======================= Logging =========================


②验证配置并启动metricbeat

# metricbeat.sh -version                           ##查看版本信息

metricbeat version 5.0.1 (amd64), libbeat 5.0.1

# metricbeat.sh -configtest -e ##验证配置文件

# systemctl start metricbeat

# systemctl status metricbeat

③配置kibana(新建metricbeat索引)

在http://localhost:5601下新建索引页面输入“metricbeat-*”,之后kibana会自动更新,在“Time-field name”下面的选项中选择“@timestamp”,最后点击“Create”创建即可。

CentOS7下Elastic Stack 5.0日志分析系统搭建

最后呈现如下图所示:

CentOS7下Elastic Stack 5.0日志分析系统搭建

注1:

关于ELK Stack的一些查询语句:

①查询filebeat

# curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'

②查询packetbeat

# curl -XGET 'http://localhost:9200/packetbeat-*/_search?pretty'

③查询metricbeat

# curl -XGET 'http://localhost:9200/metricbeat-*/_search?pretty'

④查询集群健康度

# curl 'localhost:9200/_cat/health?v'

⑤查看节点列表

# curl 'localhost:9200/_cat/nodes?v'

ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name

127.0.0.1 37 93 3 0.05 0.07 0.41 mdi * XVY0Ovb

⑥列出所有索引

# curl 'localhost:9200/_cat/indices?v'

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size

yellow open filebeat-2016.11.28 Mn4MzxdTRaCj9iseutcmqA 5 1 2 0 12kb 12kb

yellow open filebeat-2016.11.29 iMrr710mT42mApxdV62k-A 5 1 159 0 65.9kb 65.9kb

yellow open packetbeat-2016.11.29 wkTcIwD6RgiiCFwlWBIILA 5 1 5652 0 1.6mb 1.6mb

yellow open customer NvxXLgHoREefJLRhot13Ug 5 1 0 0 800b 800b

yellow open packetbeat-2016.11.28 Beoe07S7QB-dntNV4nxJNQ 5 1 2446 0 676.4kb 676.4kb

yellow open test M7WbkYq2QNmeJ9NOyMfMZA 5 1 0 0 800b 800b

yellow open logstash-2016.11.28 pcb_84ChSBe9A7VRd-SQNw 5 1 161 0 123.2kb 123.2kb

yellow open metricbeat-2016.11.29 AmVeT1xCQGCnxlAFXUxhYw 5 1 94459 0 37.6mb 37.6mb

yellow open logstash-2016.11.29 6PCKMYKCSVmPfdg-Sx2ARA 5 1 85772 0 20.3mb 20.3mb

yellow open .kibana QYTg0I5KS-yc3d7GSey3Zw 1 1 5 0 102kb 102kb

注2:

如果搭建期间有什么不清楚或不了解的,建议去看官方文档,文档地址如下:

https://www.elastic.co/guide/index.html

未登录用户
全部评论0
到底啦