在Logstash中解析JSON

本文是小编为大家收集整理的关于在Logstash中解析JSON的处理/解决方法，可以参考本文帮助大家快速定位并解决问题，中文翻译不准确的可切换到English标签页查看源文。

中文
English

问题描述

我有一些日志文件，由 log4j2 .我使用log4j2.xml配置文件中的jsonlayout将日志输出到.json文件中.我的jsonlayout是这样的:

<JSONLayout complete="false"></JSONLayout>

当日志输入我的计算机上的日志文件时，它们是附加的，一个遵循的另一个，并在logs.log中看起来像这样.

  {
    "logger":"com.mycompany.myLogger",
    "timestamp":"1396792374326",
    "level":"ERROR",
    "thread":"pool-2-thread-2",
    "message":"System: unable to perform action",
    "throwable":"java.lang.NullPointerException\\n\tat com.myCompany.MyClass $.java:432)\\n\tat java.lang.Thread.run(Thread.java:744)\\n"
  },

我正在尝试构建此JSON，以便可以从Elasticsearch查询它.在此过程中，我正在尝试将自定义字段添加到所有记录中.为此，我正在使用以下内容:

input {
  file {
    type => "json"
    path => "/var/logs/myApp/logs.log"
  }
}
filter {
  json {
    add_tag => [ "HardcodedTagName"]
    source => "message"
  }
}
output {
  elasticsearch {
    protocol => "http"
    codec => "json"
    host => "[myServerAddress]"
    port => "9200"
  }
}

奇怪的是，我的自定义标签似乎从未添加.同时，我真的很想将我的JSON分解为可以在Elasticsearch中查询的领域.我想查询的东西显然可用.他们是:

级别
消息
时间戳

但是，我似乎无法将这些信息删除.当我在Kibana中看到JSON记录时，我只是看到类似的东西:

{
  "_index": "logstash-2014.04.07",
  "_type": "json",
  "_id": "tG-s6-5pSnq5HZwLhM6Dxg",
  "_score": null,
  "_source": {
    "message": "    \"message\":\"System: unable to perform action\"\r",
    "@version": "1",
    "@timestamp": "2014-04-07T18:01:38.739Z",
    "type": "json",
    "host": "ip-MyipAddress",
    "path": "/var/logs/myApp/logs.log"
  },
  "sort": [
    1396893698739,
    1396893698739
  ]
}

最让我感到惊讶的是，日志级别甚至不存在.线程信息也不是.我很惊讶我找不到使用过滤器的log4j示例的博客文章.任何帮助都是真诚的！

推荐答案

您不需要使用JSON过滤器.您可以在输入处指定 codec 格式.

input {
    file {
       type => "json"
       path => "/var/logs/myApp/logs.log"
       codec => json
    }
}

LogStash将自动将您的日志解析为JSON格式，并添加相应的字段.

本文地址：https://www.itbaoku.cn/post/1574875.html

百度翻译此文有道翻译此文

问题描述

I have some log files generated by Log4J2. I am outputting the logs into a .json file using the JSONLayout in the log4j2.xml configuration file. My JSONLayout is defined like this:

<JSONLayout complete="false"></JSONLayout>

As logs get entered into the log file on my machine, they are appended, one-after-another, and look like this in logs.log:

  {
    "logger":"com.mycompany.myLogger",
    "timestamp":"1396792374326",
    "level":"ERROR",
    "thread":"pool-2-thread-2",
    "message":"System: unable to perform action",
    "throwable":"java.lang.NullPointerException\\n\tat com.myCompany.MyClass $.java:432)\\n\tat java.lang.Thread.run(Thread.java:744)\\n"
  },

I am trying to structure this JSON so that I can query it from ElasticSearch. During that process, I'm trying to add a custom field to ALL records. To do that, I'm using the following:

input {
  file {
    type => "json"
    path => "/var/logs/myApp/logs.log"
  }
}
filter {
  json {
    add_tag => [ "HardcodedTagName"]
    source => "message"
  }
}
output {
  elasticsearch {
    protocol => "http"
    codec => "json"
    host => "[myServerAddress]"
    port => "9200"
  }
}

Oddly, my custom tag NEVER seems to get added. At the same time, I'd really like to break out my JSON into fields that I can query in ElasticSearch. The things I'd like to query are clearly available. They are:

level
message
timestamp

Yet, I can't seem to get this information out. When I see the JSON records in Kibana, I just see something like this:

{
  "_index": "logstash-2014.04.07",
  "_type": "json",
  "_id": "tG-s6-5pSnq5HZwLhM6Dxg",
  "_score": null,
  "_source": {
    "message": "    \"message\":\"System: unable to perform action\"\r",
    "@version": "1",
    "@timestamp": "2014-04-07T18:01:38.739Z",
    "type": "json",
    "host": "ip-MyipAddress",
    "path": "/var/logs/myApp/logs.log"
  },
  "sort": [
    1396893698739,
    1396893698739
  ]
}

What surprises me most is that the log level isn't even there. Neither is the thread information. I was surprised I couldn't find a blog post of a Log4J example that uses a filter. Any help is sincerely appreciated!