Skip to main content

日志数据采集

前置条件

前置条件
  • 安装 light-agent（登录 Lighthouse 平台， 进入「数据采集」模块，获取 light-agent 安装指令和安装步骤。）

日志采集配置

  1. 进入到 light-agent 目录
cd /usr/local/light-agent
  1. 修改 /usr/local/light-agent/conf/agent-config.yaml 配置, 在receivers节点下添加filelog节点
receivers:
  filelog:
    include:
      - /var/log/*/*.log  #指定需要采集的文件
    include_file_path: true   #是否采集文件路径
    start_at: end  # 从文件末尾开始采集, beginning 从文件开头开始采集
    operators:
      - type: regex_parser #使用正则表达式解析日志 
        regex: '^(?P<time>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2})\s+(?P<sev>[A-Z]+)\s+(?P<message>.*)$' #正则表达式
        timestamp:
          parse_from: attributes.time # 时间戳字段
          layout: '%Y-%m-%d %H:%M:%S' # 时间戳格式
        severity:
          parse_from: attributes.sev  # 日志级别字段
      - type: json_parser  # 解析json格式的日志
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%dT%H:%M:%S.%f%z'
        severity:
          parse_from: attributes.level  # 日志级别字段 
        if: 'body matches "^{.*}$"' # 判断是否为json格式
......  #省略其他配置
service:
  pipelines: 
    logs:
      receivers: [otlp,filelog] # otlp用于接收otlp协议数据，filelog用于接收文件日志数据
      processors: [resourcedetection, resource, batch] # resourcedetection, resource 用于采集host信息
      exporters: [lightnsqexporter]
  1. 上述提取的日志样例如下：
  2023-01-01 12:00:00 INFO This is a log message
  2023-01-01 12:00:01 WARN This is another log message
  2023-01-01 12:00:02 ERROR This is an error log message
  
  {"level":"INFO","time":"2025-04-24T04:27:41.587-0400","msg":"This is a log message"}
  {"level":"INFO","time":"2025-04-24T04:27:41.587-0400","msg":"This is another log message"}
  {"level":"ERROR","time":"2025-04-24T04:27:41.588-0400","msg":"This is an error log message"}
  1. 重启 light-agent
// Linux 安装重启
systemctl restart light-agent

## 相关阅读

- [核心功能：日志](/docs/features/log/intro.md)