Logstash 收集 Nginx 错误日志

在 k8s 集群中，微服务日志的查看存在一定难度，尤其是对于开发者而言，如果没有运维权限，将无法直接进入 pod 查看日志。即使对于运维人员，面对大量 pod，逐个查看也是一项繁琐的工作。

通过将日志采集到 Elasticsearch，并使用 Kibana 进行查询，可以使维护工作变得更加方便。

配置文件

配置 Logstash 以收集不同类型的日志，并通过 type 进行区分。

input {     file {        path => ["/data/nginx/log/*error*.log","/data/nginx/log/nginx.log"]         type => "nginx-error"        add_field => {"cluster" => "test-k8s"}        add_field => {"env" => "test"}        add_field => {"app" => "k8s-nginx-error"}        sincedb_path => "/data/nginx/log/logstash_nginx_error_sincedb_path"        codec => multiline {            pattern => "(\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}.*)"            negate => true            what => "previous"        }    }}filter {    if [type] ==  "nginx-error"{        grok {                match => [                "message", ".*\d{2}:\d{2}:\d{2} \[(?<loglevel>(?:\w+))\] .*, client: (?<client>(?:\w+.\w+.\w+.\w+)),.*request: \"(?<method>(?:\w+)) (?<url>(?:.*))\?(?<params>(?:.*)) HTTP.*host: \"(?<domain>(?:\w+.\w+.\w+)|(?:\w+.\w+.\w+.\w+))\".*",                "message", ".*\d{2}:\d{2}:\d{2} \[(?<loglevel>(?:\w+))\] .*request: \"(?<method>(?:\w+)) (?<url>(?:.*))\?(?<params>(?:.*)) HTTP.*host: \"(?<domain>(?:\w+.\w+.\w+)|(?:\w+.\w+.\w+.\w+))\".*",                "message", ".*\d{2}:\d{2}:\d{2} \[(?<loglevel>(?:\w+))\] .*request: \"(?<method>(\w+)) (?<url>(?:.*)) HTTP.*host: \"(?<domain>(?:\w+.\w+.\w+)|(?:\w+.\w+.\w+.\w+))\".*",                "message", ".*\d{2}:\d{2}:\d{2} \[(?<loglevel>(?:\w+))\] .*"                ]        }    }}output {    if [type] ==  "nginx-error"{        elasticsearch {            hosts => ["elsticsearch-http:9200"]            index => "nginx_error-%{+YYYY.MM.dd}"            user => "xxxxx"            password => "xxxxx"            codec => json        }    }}

确保配置文件中的 hosts, user, 和 password 根据你的 Elasticsearch 集群进行相应的替换。

可以创建 ConfigMap 保存配置信息，然后在 Pod 启动时将 ConfigMap 挂载到配置路径。

kubectl create configmap logstash-conf --from-file=nginx-logs.conf

Nginx 错误日志可以通过 hostPath 打印到主机的目录进行持久化，然后 Logstash 通过 DaemonSet 的方式部署到各个 Node 节点。

apiVersion: apps/v1kind: DaemonSetmetadata:  name: logstash-nginx-error  labels:    app: logstash-nginx-errorspec:  selector:    matchLabels:      app: logstash-nginx-error  template:    metadata:      labels:        app: logstash-nginx-error    spec:      tolerations:        - key: node-role.kubernetes.io/master          effect: NoSchedule      containers:        - name: logstash-nginx-error          image: "logstash:6.8.23"          command:            - "/usr/share/logstash/bin/logstash"            - "-f"            - "/etc/logstash/config.d/nginx-logs.conf"          resources:            limits:              memory: 1024Mi          volumeMounts:            - name: nginx-error-dir              mountPath: /data/nginx/log              readOnly: false            - name: logstash-conf              subPath: nginx-logs.conf              mountPath: /etc/logstash/config.d/nginx-logs.conf      terminationGracePeriodSeconds: 10      volumes:        - name: nginx-error-dir          hostPath:            path: /data/nginx-proxy/log            type: DirectoryOrCreate        - name: logstash-conf          configMap:            name: logstash-conf            items:              - key: nginx-logs.conf                path: nginx-logs.conf

根据 Elasticsearch 的版本选择合适的 Logstash 版本，以确保兼容性。

kubectl apply -f daemonset.yaml

在 Kibana 中创建对应的 Index Patterns 后，就可以进行日志的查看和分析了。