telegraf日志文件采集并上报kafka

数据采集

2020-01-17

952

0

技术:数据采集,是一个类似flume的文件采集工具

使用input作为输入采集数据,可以是文件,可以是http接口,详见:https://github.com/influxdata/telegraf#input-plugins

使用output作为输出,可以是kafka,可以是时序数据库,可以到es等

运行环境:linux + telegraf

demo功能:提供一个telegraf文件数据上报到kafka的配置demo

下载安装包

github源码:https://github.com/influxdata/telegraf,主要使用和功能详见说明

https://portal.influxdata.com/downloads

安装和配置

## 安装rpm包
rpm -iv telegraf-1.7.4-1.x86_64.rpm

启动

/usr/bin/telegraf -pidfile /var/run/telegraf/telegraf.pid -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d

配置数据收集demo后重启

举例配置搜集rocketmq日志文件, 输出到kafka。可以再消费kafka将数据写入时序数据库就可以展示。

[[inputs.logparser]]
  files = ["/root/logs/rocketmqlogs/store.log"]
  from_beginning = false
  [inputs.logparser.tags]
    cluster='xxmq-ali-cluster-19'
  [inputs.logparser.grok]
    patterns = ["%{GREEDYDATA:log}"]
    timezone = "Local"
    measurement = "xxmq-store"
[[inputs.logparser]]
  files = ["/root/logs/rocketmqlogs/stats.log"]
  from_beginning = false
  [inputs.logparser.tags]
    cluster='xxmq-ali-cluster-19'
  [inputs.logparser.grok]
    patterns = ["%{GREEDYDATA:log}"]
    timezone = "Local"
    measurement = "xxmq-stats"
# [[outputs.file]]
#   files = ["/tmp/metrics-store.out"]
#   data_format = "json"
#   namepass = ["xxmq-store"]
# [[outputs.file]]
#   files = ["/tmp/metrics-stats.out"]
#   data_format = "json"
#   namepass = ["xxmq-stats"]
[[outputs.kafka]]
  brokers = ["127.0.0.1:9092","127.0.0.2:9092","127.0.0.3:9092","127.0.0.4:9092","127.0.0.5:9092"]
  topic = "xxmq-broker-stats-log"
  data_format = "json"
  namepass = ["xxmq-stats"]
  version = "1.1.0"
[[outputs.kafka]]
  brokers = ["127.0.0.1:9092","127.0.0.2:9092","127.0.0.3:9092","127.0.0.4:9092","127.0.0.5:9092"]
  topic = "xxmq-broker-store-log"
  data_format = "json"
  namepass = ["xxmq-store"]
  version = "1.1.0"

支持的inputs

 

支持的output

欢迎添加微信,互相学习↑↑↑ -_-

发表评论

全部评论:0条

白老虎

programming is not only to solve problems, ways to think