免费看隐私男生网站,美国大片在线观看,2023天天操

一、elk架構簡介

首先 logstash 具有日志采集、過濾、篩選等功能，功能完善但同時體量也會比較大，消耗系統資源自然也多。filebeat作為一個輕量級日志采集工具，雖然沒有過濾篩選功能，但是僅僅部署在應用服務器作為我們采集日志的工具可以是說最好的選擇。但我們有些時候可能又需要logstash的過濾篩選功能，所以我們在采集日志時用filebeat，然后交給logstash過濾篩選。
其次，logstash的吞吐量是有限的，一旦短時間內filebeat傳過來的日志過多會產生堆積和堵塞，對日志的采集也會受到影響，所以在filebeat與logstash中間又加了一層kafka消息隊列來緩存或者說解耦，當然redis也是可以的。這樣當眾多filebeat節點采集大量日志直接放到kafka中，logstash慢慢的進行消費，兩邊互不干擾。
至于zookeeper，分布式服務管理神器，監控管理kafka的節點注冊，topic管理等，同時彌補了kafka集群節點對外界無法感知的問題，kafka實際已經自帶了zookeeper，這里將會使用獨立的zookeeper進行管理，方便后期zookeeper集群的擴展。

二、環境

阿里云ECS：5臺部署ES節點，3臺分別部署logstash、kafka、zookeeper和kibana等服務。
阿里云ECS配置：5臺 4核16G SSD磁盤。3臺 4核16G SSD磁盤。都是 Centos7.8系統
安裝 docker 和 docker-compose
ELK版本7.10.1;zookeeper版本3.6.2;kafka版本2.13-2.6.0;

詳解容器部署ELK7.10，適用于生產

三、系統參數優化

# 最大用戶打開進程數
$ vim /etc/security/limits.d/20-nproc.conf
* soft nproc 65535
* hard nproc 65535
# 優化內核，用于 docker 支持
$ modprobe br_netfilter
$ cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
$ sysctl -p /etc/sysctl.d/k8s.conf
# 優化內核，對 es 支持
$ echo 'vm.max_map_count=262144' >> /etc/sysctl.conf
# 生效配置
$ sysctl -p

四、部署 docker 和 docker-compose

部署 docker

# 安裝必要的一些系統工具
$ yum install -y yum-utils device-mapper-persistent-data lvm2
# 添加軟件源信息
$ yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 更新并安裝 Docker-CE
$ yum makecache fast
$ yum -y install docker-ce
# 配置docker
$ systemctl enable docker
$ systemctl start docker
$ vim /etc/docker/daemon.json
{"data-root": "/var/lib/docker", "bip": "10.50.0.1/16", "default-address-pools": [{"base": "10.51.0.1/16", "size": 24}], "registry-mirrors": ["https://4xr1qpsp.mirror.aliyuncs.com"], "log-opts": {"max-size":"500m", "max-file":"3"}}
$ sed -i '/ExecStart=/i ExecStartPost=\/sbin\/iptables -P FORWARD ACCEPT' /usr/lib/systemd/system/docker.service
$ systemctl enable docker.service
$ systemctl daemon-reload
$ systemctl restart docker

部署 docker-compose

# 安裝 docker-compose
$ sudo curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
$ chmod +x /usr/local/bin/docker-compose

五、部署 ES

es-master1 操作

# 創建 es 目錄
$ mkdir /data/ELKStack
$ mkdir elasticsearch elasticsearch-data elasticsearch-plugins
# 容器es用戶 uid 和 gid 都是 1000
$ chown 1000.1000 elasticsearch-data elasticsearch-plugins
# 臨時啟動一個es
$ docker run --name es-test -it --rm docker.elastic.co/elasticsearch/elasticsearch:7.10.1 bash
# 生成證書，證書有效期10年，證書輸入的密碼這里為空
$ bin/elasticsearch-certutil ca --days 3660
$ bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --days 3660
# 打開新的窗口，拷貝生成的證書
$ cd /data/ELKStack/elasticsearch
$ mkdir es-p12
$ docker cp es-test:/usr/share/elasticsearch/elastic-certificates.p12 ./es-p12
$ docker cp es-test:/usr/share/elasticsearch/elastic-stack-ca.p12 ./es-p12
$ chown -R 1000.1000 ./es-p12
# 創建 docker-compose.yml
$ vim docker-compose.yml
version: '2.2'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.10.1
container_name: es01
environment:
- cluster.name=es-docker-cluster
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms10000m -Xmx10000m"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
mem_limit: 13000m
cap_add:
- IPC_LOCK
restart: always
# 設置 docker host 網絡模式
network_mode: "host"
volumes:
- /data/ELKStack/elasticsearch-data:/usr/share/elasticsearch/data
- /data/ELKStack/elasticsearch-plugins:/usr/share/elasticsearch/plugins
- /data/ELKStack/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
- /data/ELKStack/elasticsearch/es-p12:/usr/share/elasticsearch/config/es-p12
# 創建 elasticsearch.yml 配置文件
$ vim elasticsearch.yml
cluster.name: "es-docker-cluster"
node.name: "es01"
network.host: 0.0.0.0
node.master: true
node.data: true
discovery.zen.minimum_master_nodes: 2
http.port: 9200
transport.tcp.port: 9300
# 如果是多節點es，通過ping來健康檢查
discovery.zen.ping.unicast.hosts: ["172.20.166.25:9300", "172.20.166.24:9300", "172.20.166.22:9300", "172.20.166.23:9300", "172.20.166.26:9300"]
discovery.zen.fd.ping_timeout: 120s
discovery.zen.fd.ping_retries: 6
discovery.zen.fd.ping_interval: 10s
cluster.info.update.interval: 1m
indices.fielddata.cache.size: 20%
indices.breaker.fielddata.limit: 40%
indices.breaker.request.limit: 40%
indices.breaker.total.limit: 70%
indices.memory.index_buffer_size: 20%
script.painless.regex.enabled: true
# 磁盤分片分配
cluster.routing.allocation.disk.watermark.low: 100gb
cluster.routing.allocation.disk.watermark.high: 50gb
cluster.routing.allocation.disk.watermark.flood_stage: 30gb
# 本地數據分片恢復配置
gateway.recover_after_nodes: 3
gateway.recover_after_time: 5m
gateway.expected_nodes: 3
cluster.routing.allocation.node_initial_primaries_recoveries: 8
cluster.routing.allocation.node_concurrent_recoveries: 2
# 允許跨域請求
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type
# 開啟xpack
xpack.security.enabled: true
xpack.monitoring.collection.enabled: true
# 開啟集群中https傳輸
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: es-p12/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: es-p12/elastic-certificates.p12
# 把 es 配置使用 rsync 同步到其它 es 節點
$ rsync -avp -e ssh /data/ELKStack 172.20.166.24:/data/
$ rsync -avp -e ssh /data/ELKStack 172.20.166.22:/data/
$ rsync -avp -e ssh /data/ELKStack 172.20.166.23:/data/
$ rsync -avp -e ssh /data/ELKStack 172.20.166.26:/data/
# 啟動 es
$ docker-compose up -d
# 查看 es
$ docker-compose ps

es-master2 操作

$ cd /data/ELKStack/elasticsearch
# 修改 docker-compose.yml elasticsearch.yml 兩個配置
$ sed -i 's/es01/es02/g' docker-compose.yml elasticsearch.yml
# 啟動 es
$ docker-compose up -d

es-master3 操作

$ cd /data/ELKStack/elasticsearch
# 修改 docker-compose.yml elasticsearch.yml 兩個配置
$ sed -i 's/es01/es03/g' docker-compose.yml elasticsearch.yml
# 啟動 es
$ docker-compose up -d

es-data1 操作

$ cd /data/ELKStack/elasticsearch
# 修改 docker-compose.yml elasticsearch.yml 兩個配置
$ sed -i 's/es01/es04/g' docker-compose.yml elasticsearch.yml
# 不做為 es master 節點，只做數據節點
$ sed -i 's/node.master: true/node.master: false/g' elasticsearch.yml
# 啟動 es
$ docker-compose up -d

es-data2 操作

$ cd /data/ELKStack/elasticsearch
# 修改 docker-compose.yml elasticsearch.yml 兩個配置
$ sed -i 's/es01/es05/g' docker-compose.yml elasticsearch.yml
# 不做為 es master 節點，只做數據節點
$ sed -i 's/node.master: true/node.master: false/g' elasticsearch.yml
# 啟動 es
$ docker-compose up -d

設置 es 訪問賬號

# es-master1 操作
$ docker exec -it es01 bash
# 設置 elastic，apm_system，kibana，kibana_system，logstash_system，beats_system，remote_monitoring_user 等密碼
# 密碼都設置為 elastic123，這里只是舉例，具體根據需求設置
$ ./bin/elasticsearch-setup-passwords interactive

六、部署 Kibana

logstash3 操作

$ mkdir -p /data/ELKStack/kibana
$ cd /data/ELKStack/kibana
# 創建 kibana 相關目錄，用于容器掛載
$ mkdir config data plugins
$ chown 1000.1000 config data plugins
# 創建 docker-compose.yml
$ vim docker-compose.yml
version: '2'
services:
kibana:
image: docker.elastic.co/kibana/kibana:7.10.1
container_name: kibana
restart: always
network_mode: "bridge"
mem_limit: 2000m
environment:
SERVER_NAME: kibana.example.com
ports:
- "5601:5601"
volumes:
- /data/ELKStack/kibana/config:/usr/share/kibana/config
- /data/ELKStack/kibana/data:/usr/share/kibana/data
- /data/ELKStack/kibana/plugins:/usr/share/kibana/plugins
# 創建 kibana.yml
$ vim config/kibana.yml
server.name: kibana
server.host: "0"
elasticsearch.hosts: ["http://172.20.166.25:9200","http://172.20.166.24:9200","http://172.20.166.22:9200"]
elasticsearch.username: "kibana"
elasticsearch.password: "elastic123"
monitoring.ui.container.elasticsearch.enabled: true
xpack.security.enabled: true
xpack.encryptedSavedObjects.encryptionKey: encryptedSavedObjects12345678909876543210
xpack.security.encryptionKey: encryptionKeysecurity12345678909876543210
xpack.reporting.encryptionKey: encryptionKeyreporting12345678909876543210
i18n.locale: "zh-CN"
# 啟動 kibana
$ docker-compose up -d

七、部署 Zookeeper

logstash1 操作

# 創建 zookeeper 目錄
$ mkdir /data/ELKStack/zookeeper
$ cd /data/ELKStack/zookeeper
$ mkdir data datalog
$ chown 1000.1000 data datalog
# 創建 docker-compose.yml
$ vim docker-compose.yml
version: '2'
services:
zoo1:
image: zookeeper:3.6.2
restart: always
hostname: zoo1
container_name: zoo1
network_mode: "bridge"
mem_limit: 2000m
ports:
- 2181:2181
- 3888:3888
- 2888:2888
volumes:
- /data/ELKStack/zookeeper/data:/data
- /data/ELKStack/zookeeper/datalog:/datalog
- /data/ELKStack/zookeeper/zoo.cfg:/conf/zoo.cfg
environment:
ZOO_MY_ID: 1 # 表示 ZK服務的 id, 它是1-255 之間的整數, 必須在集群中唯一
ZOO_SERVERS: server.1=0.0.0.0:2888:3888;2181 server.2=172.20.166.28:2888:3888;2181 server.3=172.20.166.29:2888:3888;2181
# ZOOKEEPER_CLIENT_PORT: 2181
# 創建 zoo.cfg 配置
$ vim zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data
dataLogDir=/datalog
autopurge.snapRetainCount=3
autopurge.purgeInterval=1
maxClientCnxns=60
server.1= 0.0.0.0:2888:3888;2181
server.2= 172.20.166.28:2888:3888;2181
server.3= 172.20.166.29:2888:3888;2181
# 拷貝配置到 logstash2 logstash3 機器上
$ rsync -avp -e ssh /data/ELKStack/zookeeper 172.20.166.28:/data/ELKStack/
$ rsync -avp -e ssh /data/ELKStack/zookeeper 172.20.166.29:/data/ELKStack/
# 啟動 zookeeper
$ docker-compose up -d

logstash2 操作

$ cd /data/ELKStack/zookeeper
# 修改 docker-compose.yml 文件
$ vim docker-compose.yml
version: '2'
services:
zoo2:
image: zookeeper:3.6.2
restart: always
hostname: zoo2
container_name: zoo2
network_mode: "bridge"
mem_limit: 2000m
ports:
- 2181:2181
- 3888:3888
- 2888:2888
volumes:
- /data/ELKStack/zookeeper/data:/data
- /data/ELKStack/zookeeper/datalog:/datalog
- /data/ELKStack/zookeeper/zoo.cfg:/conf/zoo.cfg
environment:
ZOO_MY_ID: 2 # 表示 ZK服務的 id, 它是1-255 之間的整數, 必須在集群中唯一
ZOO_SERVERS: server.1=172.20.166.27:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181 server.3=172.20.166.29:2888:3888;2181
# ZOOKEEPER_CLIENT_PORT: 2181
# 修改 zoo.cfg
$ vim zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data
dataLogDir=/datalog
autopurge.snapRetainCount=3
autopurge.purgeInterval=1
maxClientCnxns=60
server.1= 172.20.166.27:2888:3888;2181
server.2= 0.0.0.0:2888:3888;2181
server.3= 172.20.166.29:2888:3888;2181
# 啟動 zookeeper
$ docker-compose up -d

logstash3 操作

$ cd /data/ELKStack/zookeeper
# 修改 docker-compose.yml 文件
$ vim docker-compose.yml
version: '2'
services:
zoo3:
image: zookeeper:3.6.2
restart: always
hostname: zoo3
container_name: zoo3
network_mode: "bridge"
mem_limit: 2000m
ports:
- 2181:2181
- 3888:3888
- 2888:2888
volumes:
- /data/ELKStack/zookeeper/data:/data
- /data/ELKStack/zookeeper/datalog:/datalog
- /data/ELKStack/zookeeper/zoo.cfg:/conf/zoo.cfg
environment:
ZOO_MY_ID: 3 # 表示 ZK服務的 id, 它是1-255 之間的整數, 必須在集群中唯一
ZOO_SERVERS: server.1=172.20.166.27:2888:3888;2181 server.2=172.20.166.28:2888:3888;2181 server.3=0.0.0.0:2888:3888;2181
# ZOOKEEPER_CLIENT_PORT: 2181
# 修改 zoo.cfg
$ vim zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data
dataLogDir=/datalog
autopurge.snapRetainCount=3
autopurge.purgeInterval=1
maxClientCnxns=60
server.1= 172.20.166.27:2888:3888;2181
server.2= 172.20.166.28:2888:3888;2181
server.3= 0.0.0.0:2888:3888;2181
# 啟動 zookeeper
$ docker-compose up -d
# 操作 zookeeper
$ docker run -it zoo3 bash
$ zkCli.sh -server 172.20.166.27:2181,172.20.166.28:2181,172.20.166.29:2181

八、部署 Kafka

logstash1 操作

# 創建 kafka 目錄
$ mkdir -p /data/ELKStack/kafka
$ cd /data/ELKStack/kafka
# 創建數據目錄，用于存儲kafka容器數據
$ mkdir data
# 把kafka配置拷貝到宿主機上
$ docker run --name kafka-test -it --rm wurstmeister/kafka:2.13-2.6.0 bash
$ cd /opt/kafka
$ tar zcvf /tmp/config.tar.gz config
# 打開一個新的窗口
$ docker cp kafka-test:/tmp/config.tar.gz ./
# 解壓配置文件
$ tar xf config.tar.gz
# 創建 docker-compose.yml
$ vim docker-compose.yml
version: '2'
services:
kafka1:
image: wurstmeister/kafka:2.13-2.6.0
restart: always
hostname: kafka1
container_name: kafka1
network_mode: "bridge"
mem_limit: 5120m
ports:
- 9092:9092
- 9966:9966
environment:
KAFKA_BROKER_ID: 1
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://172.20.166.27:9092 # 宿主機的IP地址而非容器的IP，及暴露出來的端口
KAFKA_ADVERTISED_HOST_NAME: 172.20.166.27 # 外網訪問地址
KAFKA_ADVERTISED_PORT: 9092 # 端口
KAFKA_ZOOKEEPER_CONNECT: 172.20.166.27:2181,172.20.166.28:2181,172.20.166.29:2181 # 連接的zookeeper服務及端口
KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=172.20.166.27 -Dcom.sun.management.jmxremote.rmi.port=9966"
JMX_PORT: 9966 # kafka需要監控broker和topic的數據的時候,是需要開啟jmx_port的
KAFKA_HEAP_OPTS: "-Xmx4096M -Xms4096M"
volumes:
- /data/ELKStack/kafka/data:/kafka # kafka數據文件存儲目錄
- /data/ELKStack/kafka/config:/opt/kafka/config
# 優化 kafka server.properties 配置
$ vim config/server.properties
# 調大socket，防止報錯
socket.send.buffer.bytes=1024000
socket.receive.buffer.bytes=1024000
socket.request.max.bytes=1048576000
# topic 數據保留多久，默認168小時(7day)
log.retention.hours=72
log.cleanup.policy=delete
# 拷貝配置到 logstash2 logstash3 機器上
$ rsync -avp -e ssh /data/ELKStack/kafka 172.20.166.28:/data/ELKStack/
$ rsync -avp -e ssh /data/ELKStack/kafka 172.20.166.29:/data/ELKStack/
# 啟動 kafka
$ docker-compose up -d

logstash2 操作

$ cd /data/ELKStack/kafka
# 修改 docker-compose.yml 文件
$ vim docker-compose.yml
version: '2'
services:
kafka2:
image: wurstmeister/kafka:2.13-2.6.0
restart: always
hostname: kafka2
container_name: kafka2
network_mode: "bridge"
mem_limit: 5120m
ports:
- 9092:9092
- 9966:9966
environment:
KAFKA_BROKER_ID: 2
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://172.20.166.28:9092 # 宿主機的IP地址而非容器的IP，及暴露出來的端口
KAFKA_ADVERTISED_HOST_NAME: 172.20.166.28 # 外網訪問地址
KAFKA_ADVERTISED_PORT: 9092 # 端口
KAFKA_ZOOKEEPER_CONNECT: 172.20.166.27:2181,172.20.166.28:2181,172.20.166.29:2181 # 連接的zookeeper服務及端口
KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=172.20.166.28 -Dcom.sun.management.jmxremote.rmi.port=9966"
JMX_PORT: 9966 # kafka需要監控broker和topic的數據的時候,是需要開啟jmx_port的
KAFKA_HEAP_OPTS: "-Xmx4096M -Xms4096M"
volumes:
- /data/ELKStack/kafka/data:/kafka # kafka數據文件存儲目錄
- /data/ELKStack/kafka/config:/opt/kafka/config
# 啟動 kafka
$ docker-compose up -d

logstash3 操作

$ cd /data/ELKStack/kafka
# 修改 docker-compose.yml 文件
$ vim docker-compose.yml
version: '2'
services:
kafka3:
image: wurstmeister/kafka:2.13-2.6.0
restart: always
hostname: kafka3
container_name: kafka3
network_mode: "bridge"
mem_limit: 5120m
ports:
- 9092:9092
- 9966:9966
environment:
KAFKA_BROKER_ID: 3
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://172.20.166.29:9092 # 宿主機的IP地址而非容器的IP，及暴露出來的端口
KAFKA_ADVERTISED_HOST_NAME: 172.20.166.29 # 外網訪問地址
KAFKA_ADVERTISED_PORT: 9092 # 端口
KAFKA_ZOOKEEPER_CONNECT: 172.20.166.27:2181,172.20.166.28:2181,172.20.166.29:2181 # 連接的zookeeper服務及端口
KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=172.20.166.29 -Dcom.sun.management.jmxremote.rmi.port=9966"
JMX_PORT: 9966 # kafka需要監控broker和topic的數據的時候,是需要開啟jmx_port的
KAFKA_HEAP_OPTS: "-Xmx4096M -Xms4096M"
volumes:
- /data/ELKStack/kafka/data:/kafka # kafka數據文件存儲目錄
- /data/ELKStack/kafka/config:/opt/kafka/config
# 啟動 kafka
$ docker-compose up -d
# 部署 kafka-manager 管理 kafka 平臺
$ mkdir /data/ELKStack/kafka-manager
$ cd /data/ELKStack/kafka-manager
$ vim docker-compose.yml
version: '3.6'
services:
kafka_manager:
restart: always
container_name: kafa-manager
hostname: kafka-manager
network_mode: "bridge"
mem_limit: 1024m
image: hlebalbau/kafka-manager:3.0.0.5-7e7a22e
ports:
- "9000:9000"
environment:
ZK_HOSTS: "172.20.166.27:2181,172.20.166.28:2181,172.20.166.29:2181"
APPLICATION_SECRET: "random-secret"
KAFKA_MANAGER_AUTH_ENABLED: "true"
KAFKA_MANAGER_USERNAME: admin
KAFKA_MANAGER_PASSWORD: elastic123
JMX_PORT: 9966
TZ: "Asia/Shanghai"
# 啟動 kafka-manager
$ docker-compose up -d
# 訪問 http://172.20.166.29:9000 ，把上面創建的三臺 kafka 加入管理，這里不在闡述，網上很多配置教程

九、部署 logstash

logstash1 操作

$ mkdir /data/ELKStack/logstash
$ cd /data/ELKStack/logstash
$ mkdir config data
$ chown 1000.1000 config data
# 創建 docker-compose.yml
$ vim docker-compose.yml
version: '2'
services:
logstash1:
image: docker.elastic.co/logstash/logstash:7.10.1
container_name: logstash1
hostname: logstash1
restart: always
network_mode: "bridge"
mem_limit: 4096m
environment:
TZ: "Asia/Shanghai"
ports:
- 5044:5044
volumes:
- /data/ELKStack/logstash/config:/config-dir
- /data/ELKStack/logstash/logstash.yml:/usr/share/logstash/config/logstash.yml
- /data/ELKStack/logstash/data:/usr/share/logstash/data
- /etc/localtime:/etc/localtime
user: logstash
command: bash -c "logstash -f /config-dir --config.reload.automatic"
# 創建 logstash.yml
$ vim logstash.yml
http.host: "0.0.0.0"
# 指發送到Elasticsearch的批量請求的大小，值越大，處理則通常更高效，但增加了內存開銷
pipeline.batch.size: 3000
# 指調整Logstash管道的延遲，過了該時間則logstash開始執行過濾器和輸出
pipeline.batch.delay: 200
# 創建 logstash 規則配置
$ vim config/01-input.conf
input { # 輸入組件
kafka { # 從kafka消費數據
bootstrap_servers => ["172.20.166.27:9092,172.20.166.28:9092,172.20.166.29:9092"]
#topics => "%{[@metadata][topic]}" # 使用kafka傳過來的topic
topics_pattern => "elk-.*" # 使用正則匹配topic
codec => "json" # 數據格式
consumer_threads => 3 # 消費線程數量
decorate_events => true # 可向事件添加Kafka元數據，比如主題、消息大小的選項，這將向logstash事件中添加一個名為kafka的字段
auto_offset_reset => "latest" # 自動重置偏移量到最新的偏移量
group_id => "logstash-node" # 消費組ID，多個有相同group_id的logstash實例為一個消費組
client_id => "logstash1" # 客戶端ID
fetch_max_wait_ms => "1000" # 指當沒有足夠的數據立即滿足fetch_min_bytes時，服務器在回答fetch請求之前將阻塞的最長時間
}
}
$ vim config/02-output.conf
output { # 輸出組件
elasticsearch {
# Logstash輸出到es
hosts => ["172.20.166.25:9200", "172.20.166.24:9200", "172.20.166.22:9200", "172.20.166.23:9200", "172.20.166.26:9200"]
index => "%{[fields][source]}-%{+YYYY-MM-dd}" # 直接在日志中匹配，索引會去掉elk
# index => "%{[@metadata][topic]}-%{+YYYY-MM-dd}" # 以日期建索引
user => "elastic"
password => "elastic123"
}
#stdout {
# codec => rubydebug
#}
}
$ vim config/03-filter.conf
filter {
# 當非業務字段時，無traceId則移除
if ([message] =~ "traceId=null") { # 過濾組件，這里只是展示，無實際意義，根據自己的業務需求進行過濾
drop {}
}
}
# 拷貝配置到 logstash2 logstash3 機器上
$ rsync -avp -e ssh /data/ELKStack/logstash 172.20.166.28:/data/ELKStack/
$ rsync -avp -e ssh /data/ELKStack/logstash 172.20.166.29:/data/ELKStack/
# 啟動 logstash
$ docker-compose up -d

logstash2 操作

$ cd /data/ELKStack/logstash
$ sed -i 's/logstash1/logstash2/g' docker-compose.yml
$ sed -i 's/logstash1/logstash2/g' config/01-input.conf
# 啟動 logstash
$ docker-compose up -d

logstash3 操作

$ cd /data/ELKStack/logstash
$ sed -i 's/logstash1/logstash3/g' docker-compose.yml
$ sed -i 's/logstash1/logstash3/g' config/01-input.conf
# 啟動 logstash
$ docker-compose up -d

十、部署 filebeat

# 配置 filebeat yum源，這里以 centos7 為例
$ rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
$ vim /etc/yum.repos.d/elastic.repo
[elastic-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
$ yum install -y filebeat-7.10.1
$ systemctl enable filebeat
# 配置
$ cd /etc/filebeat/
$ cp -a filebeat.yml filebeat.yml.old
$ echo > filebeat.yml
# 以收集nginx訪問日志為例
$ vim filebeat.yml
filebeat.inputs: # inputs為復數，表名type可以有多個
- type: log # 輸入類型
access:
enabled: true # 啟用這個type配置
json.keys_under_root: true # 默認這個值是FALSE的，也就是我們的json日志解析后會被放在json鍵上。設為TRUE，所有的keys就會被放到根節點
json.overwrite_keys: true # 是否要覆蓋原有的key，這是關鍵配置，將keys_under_root設為TRUE后，再將overwrite_keys也設為TRUE，就能把filebeat默認的key值給覆蓋
max_bytes: 20480 # 單條日志的大小限制,建議限制(默認為10M,queue.mem.events * max_bytes 將是占有內存的一部分)
paths:
- /var/log/nginx/access.log # 監控nginx 的access日志
fields: # 額外的字段
source: nginx-access-prod # 自定義source字段，用于es建議索引（字段名小寫，我記得大寫好像不行）
# 自定義es的索引需要把ilm設置為false
setup.ilm.enabled: false
output.kafka: # 輸出到kafka
enabled: true # 該output配置是否啟用
hosts: ["172.20.166.27:9092", "172.20.166.28:9092", "172.20.166.29:9092"] # kafka節點列表
topic: "elk-%{[fields.source]}" # kafka會創建該topic，然后logstash(可以過濾修改)會傳給es作為索引名稱
partition.hash:
reachable_only: true # 是否只發往可達分區
compression: gzip # 壓縮
max_message_bytes: 1000000 # Event最大字節數。默認1000000。應小于等于kafka broker message.max.bytes值
required_acks: 1 # kafka ack等級
worker: 1 # kafka output的最大并發數
bulk_max_size: 2048 # 單次發往kafka的最大事件數
logging.to_files: true # 輸出所有日志到file，默認true，達到日志文件大小限制時，日志文件會自動限制替換，詳細配置：https://www.cnblogs.com/qinwengang/p/10982424.html
close_older: 30m # 如果一個文件在某個時間段內沒有發生過更新，則關閉監控的文件handle。默認1h
force_close_files: false # 這個選項關閉一個文件,當文件名稱的變化。只在window建議為true
# 沒有新日志采集后多長時間關閉文件句柄，默認5分鐘，設置成1分鐘，加快文件句柄關閉
close_inactive: 1m
# 傳輸了3h后荏沒有傳輸完成的話就強行關閉文件句柄，這個配置項是解決以上案例問題的key point
close_timeout: 3h
# 這個配置項也應該配置上，默認值是0表示不清理，不清理的意思是采集過的文件描述在registry文件里永不清理，在運行一段時間后，registry會變大，可能會帶來問題
clean_inactive: 72h
# 設置了clean_inactive后就需要設置ignore_older，且要保證ignore_older < clean_inactive
ignore_older: 70h
# 限制 CPU和內存資源
max_procs: 1 # 限制一個CPU核心,避免過多搶占業務資源
queue.mem.events: 256 # 存儲于內存隊列的事件數，排隊發送 (默認4096)
queue.mem.flush.min_events: 128 # 小于 queue.mem.events ,增加此值可提高吞吐量 (默認值2048)
# 啟動 filebeat
$ systemctl start filebeat

十一、部署 curator，定時清理es索引

logstash3 機器操作

# 參考鏈接：https://www.elastic.co/guide/en/elasticsearch/client/curator/current/yum-repository.html
# 安裝 curator 服務，以 centos7 為例
$ rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
$ vim /etc/yum.repos.d/elk-curator-5.repo
[curator-5]
name=CentOS/RHEL 7 repository for Elasticsearch Curator 5.x packages
baseurl=https://packages.elastic.co/curator/5/centos/7
gpgcheck=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
$ yum install elasticsearch-curator -y
# 創建 curator 配置文件目錄與輸出日志目錄
$ mkdir -p /data/ELKStack/curator/logs
$ cd /data/ELKStack/curator
$ vim config.yml
---
# Remember, leave a key empty if there is no value. None will be a string,
# # not a Python "NoneType"
client:
hosts: ["172.20.166.25", "172.20.166.24", "172.20.166.22", "172.20.166.23", "172.20.166.26"]
port: 9200
url_prefix:
use_ssl: False
certificate:
client_cert:
client_key:
ssl_no_validate: False
http_auth: elastic:elastic123
timeout: 150
master_only: False
logging:
loglevel: INFO
logfile: /data/ELKStack/curator/logs/curator.log
logformat: default
blacklist: ['elasticsearch', 'urllib3']
$ vim action.yml
---
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True. If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
1:
action: delete_indices
description: >-
Delete indices older than 30 days. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
options:
ignore_empty_list: True
disable_action: False
filters:
- filtertype: pattern
kind: regex
value: '^((?!(kibana|json|monitoring|metadata|apm|async|transform|siem|security)).)*$'
- filtertype: age
source: creation_date
direction: older
#timestring: '%Yi-%m-%d'
unit: days
unit_count: 30
2:
action: delete_indices
description: >-
Delete indices older than 15 days. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
options:
ignore_empty_list: True
disable_action: False
filters:
- filtertype: pattern
kind: regex
value: '^(nginx-).*$'
- filtertype: age
source: creation_date
direction: older
#timestring: '%Yi-%m-%d'
unit: days
unit_count: 15
# 設置定時任務清理es索引
$ crontab -e
0 0 * * * /usr/bin/curator --config /data/ELKStack/curator/config.yml /data/ELKStack/curator/action.yml

十二、參考鏈接

es證書配置：https://cloud.tencent.com/developer/article/1549834
es忘記密碼找回：https://www.cnblogs.com/woshimrf/p/docker-es7.html
es設置密碼：https://blog.csdn.net/extraordinarylife/article/details/107917764?utm_medium=distribute.pc_relevant.none-task-blog-baidujs_title-2&spm=1001.2101.3001.4242
elk-kafka部署 1：https://www.codenong.com/cs106056710/
elk-kafka部署 2：https://www.cnblogs.com/lz0925/p/12061293.html
elk優化：https://www.clxz.top/2020/06/19/elk-kafka-optimization/
es7索引分片：https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html
filebeat優化：https://www.jianshu.com/p/389702465461
kafka數據與日志清理：https://blog.csdn.net/VIP099/article/details/106257561

原文地址：https://mp.weixin.qq.com/s/Hx2lNG4D6vhgjd1Ftaa8GA