filebeat 使用技巧介绍
一：多行匹配案例
multiline.type: pattern
multiline.pattern: '^\['
multiline.negate: true
multiline.match: after
上面的例子中，Filebeat将所有不以 [ 开始的行与之前的行进行合并
multiline.pattern:'^[[:space:]] '
multiline.negate: false
multiline.match: after
上面表示，如果不以空格开头，则这一行是一条日志的开头行，它与接下来有1或多个空格开头的各行、构成一条完整日志。
日志源文件格式：
#匹配多行实例
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/*.log
fields:
type: sys_log
- type: log
enabled: true
paths:
- /data/game/mhjy_yfb/game_server/logs/app.log.*
fields:
type: game_server_log
#multiline.type: pattern
multiline.pattern: '^\['
multiline.negate: true
multiline.match: after
三：filebeat6.2.3收集多个日志源 多个topic输出
配置filebeat.yml
filebeat.prospectors:
- type: log
enabled: true
paths:
- /var/log/mylog/test1.log
fields:
log_topics: test1
-type:log
enabled:true
paths:
-/var/log/mylog/test2.log
fields:
log_topics:test2
output.kafka:
enabled:true
hosts:["10.2.90.10:9092"]
topic:'%{[fields][log_topics]}'
运行filebeat
./filebeat -e -c filebeat.yml
#filebeat.yml
filebeat.prospectors:
#filebeat.inputs:
- type: log
# Change to true to enable this input configuration.
# enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /logs/*/*/*.log
#==================== elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 3
#index.codec: best_compression
#_source.enabled: false
#----------------------------- Logstash output -------------------
-------------
output.logstash:
# The Logstash hosts
hosts: ["10.2.90.15:8081"]
三：filebeat直接给es传输日志，自定义索引名，自定义多个索引文件
#filebeat.yaml
----------Elasticsearch template setting ----------
setup.ilm.enabled: false
setup.template.name: "filebeat-124"
setup.template.pattern: "filebeat-124-*"
setup.template.settings:
index.number_of_shards: 1
index.number_of_replicas: 0
index.codec: best_compression
----------Elasticsearch Output----------
hosts: ["10.2.90.124:9200"]
index: "filebeat-124-%{ yyyy.MM.dd}"
#filebeat.yaml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/*.log
fields:
type: sys_log
- type: log
enabled: true
paths:
- /data/game/mhjy_yfb/game_server/logs/app.log.*
fields:
type: game_server_log
- type: log
enabled: true
paths:
- /data/game/mhjy_yfb/game_backend/log/*/*.log
fields:
type: game_backend_log
setup.ilm.enabled: false
setup.template.settings:
index.number_of_shards: 1
index.number_of_replicas: 0
index.codec: best_compression
output.elasticsearch:
hosts: ["10.2.90.124:9200"]
indices:
- index: "mhjy_yfb_game_server_log_%{ yyyy.MM.dd}"
when.equals:
fields.type: "game_server_log"
- index: "mhjy_yfb_game_backend_log_%{ yyyy.MM.dd}"
when.equals:
fields.type: "game_backend_log"
- index: "mhjy_yfb_sys_log_%{ yyyy.MM.dd}"
when.equals:
fields.type: "sys_log"
