007.ELK收集Java日志

CoderJed

发布于 2020-04-24 17:59:23

6310

发布于 2020-04-24 17:59:23

文章被收录于专栏：Jed的技术阶梯

1. Java日志的特点

服务器访问日志都是一行一行的：

{"time_local": "16/Apr/2020:17:17:09 +0800", "remote_addr": "10.0.0.101", "referer": "-", "request": "GET / HTTP/1.0", "status": 200, "bytes": 612, "agent": "ApacheBench/2.3", "x_forwarded": "-", "up_addr": "-", "up_host": "-", "upstream_time": "-", "request_time": "0.000"}
{"time_local": "16/Apr/2020:17:17:09 +0800", "remote_addr": "10.0.0.101", "referer": "-", "request": "GET / HTTP/1.0", "status": 200, "bytes": 612, "agent": "ApacheBench/2.3", "x_forwarded": "-", "up_addr": "-", "up_host": "-", "upstream_time": "-", "request_time": "0.000"}

Java日志如果报错的话，一段异常栈信息会很长：

[2020-04-14T18:52:18,889][ERROR][o.e.b.Bootstrap          ] [node-1] Exception
java.lang.IllegalStateException: Failed to create node environment
        at org.elasticsearch.node.Node.<init>(Node.java:298) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.6.0.jar:6.6.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.6.0.jar:6.6.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.6.0.jar:6.6.0]
Caused by: java.nio.file.AccessDeniedException: /data/elasticsearch/node-1/nodes
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:?]
        at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_241]
        at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_241]
        at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_241]
        at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:270) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:203) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:267) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.node.Node.<init>(Node.java:295) ~[elasticsearch-6.6.0.jar:6.6.0]
        ... 11 more
[2020-04-14T18:52:18,896][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]

所以逐行收集Java日志是无意义的

2. filebeat配置

[root@elk-175 ~]# cat /etc/filebeat/filebeat.yml       
filebeat.inputs:
- type: log
  enabled: true 
  paths:
    - /var/log/elasticsearch/elasticsearch.log
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after
setup.kibana:
  host: "192.168.47.175:5601"
output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "elasticsearch-server-%{+yyyy.MM}"
setup.template.name: "elasticsearch"
setup.template.pattern: "elasticsearch-*"
setup.template.enabled: false
setup.template.overwrite: true

multiline.pattern: '^\['：匹配[开头的行
multiline.negate: true：是否锁定pattern，默认false
multiline.match: after：指定Filebeat如何将匹配的行组合到事件中，可选after和before

看一个示例：

negate	match	结果
false	after	将符合正则的行，与前一个不符合正则的行合并为一行
false	before	将符合正则的行，与后面一个不符合正则的行合并为一行
true	after	将不符合正则的行，与前一个符合正则的行合并为一行
true	before	将不符合正则的行，与后一个符合正则的行合并为一行