文档建议反馈控制台
首页
学习
活动
专区
工具
TVP
最新优惠活动
文章/答案/技术大牛
发布
精选内容/技术社群/优惠产品,尽在小程序
立即前往

Filebeat配置部署指南

文章来源：infoqRancherLabs
举报

在本文中，我们将了解如何配置Filebeat作为DaemonSet在我们的Kubernetes集群中运行，以便将日志运送到Elasticsearch后端。我们使用Filebeat而不是FluentD或FluentBit，因为它是一个非常轻量级的实用程序，并且对Kubernetes有一流的支持，因此这是十分适合生产的配置。

部署架构

Filebeat将在我们的Kubernetes集群中作为DaemonSet运行。它将会：

  • 部署在一个名为logging的单独的命名空间内
  • Pod将会在Master节点和Worker节点被调度
  • master节点pods将转发api-server日志，用于审计和集群管理。
  • 客户端节点Pods将转发工作负载相关的日志，用于应用程序可观察性

创建Filebeat 服务账户和ClusterRole

部署以下manifest以创建Filebeat pod所需的权限：

代码语言：javascript
复制
apiVersion: v1
kind: Namespace
metadata:
  name: logging
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: logging
  labels:
    k8s-app: filebeat
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: filebeat
  namespace: logging
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: filebeat
  namespace: logging
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io

我们应该从安全的角度出发，确保ClusterRole的权限尽可能地受到限制。如果与该服务账户相关联的任何一个pod被泄露，那么攻击者将无法获得对整个集群或其中运行的应用程序的访问权限。

创建Filebeat ConfigMap

使用以下 manifest 来创建一个 ConfigMap，它将由 Filebeat pod 使用：

代码语言：javascript
复制
apiVersion: v1
kind: Namespace
metadata:
  name: logging
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: logging
  labels:
    k8s-app: filebeat
    kubernetes.io/cluster-service: "true"
data:
  filebeat.yml: |-
    filebeat.config:
    #  inputs:
    #    path: ${path.config}/inputs.d/*.yml
    #    reload.enabled: true
      modules:
        path: ${path.config}/modules.d/*.yml
        reload.enabled: true

    filebeat.autodiscover:
      providers:
        - type: kubernetes
          hints.enabled: true
          include_annotations: ["artifact.spinnaker.io/name","ad.datadoghq.com/tags"]
          include_labels: ["app.kubernetes.io/name"]
          labels.dedot: true
          annotations.dedot: true
          templates:
            - condition:
                equals:
                  kubernetes.namespace: myapp   #Set the namespace in which your app is running, can add multiple conditions in case of more than 1 namespace.
              config:
                - type: docker
                  containers.ids:
                    - "${data.kubernetes.container.id}"
                  multiline:
                    pattern: '^[A-Za-z ]+[0-9]{2} (?:[01]\d|2[0123]):(?:[012345]\d):(?:[012345]\d)'.   #Timestamp regex for the app logs. Change it as per format. 
                    negate: true
                    match: after
            - condition:
                equals:
                  kubernetes.namespace: elasticsearch
              config:
                - type: docker
                  containers.ids:
                    - "${data.kubernetes.container.id}"
                  multiline:
                    pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}|^[0-9]{4}-[0-9]{2}-[0-9]{2}T'
                    negate: true
                    match: after
                    
    processors:
      - add_cloud_metadata: ~
      - drop_fields:
          when:
            has_fields: ['kubernetes.labels.app']
          fields:
            - 'kubernetes.labels.app'

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']

关于 Filebeat ConfigMap 需要了解以下几个重要概念：

  • hints.enabled： 这将激活 Filebeat 的 Kubernetes 的 hint 模块。通过使用这个，我们可以使用 pod 注释直接将 config 传递给 Filebeat pod。我们可以指定不同的多行模式和其他各种类型的配置。更多关于这方面的内容可以访问以下链接： https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover-hints.html
  • include_annotations： 将此设置为 “true”，可以让 Filebeat 保留特定日志条目的任何 pod 注释。这些注释可以在以后用于在 Kibana 控制台中过滤日志。
  • include_labels: 将此设置为 “true”，可以让 Filebeat 保留特定日志条目的任何 pod 标签，这些标签以后可以用于在 Kibana 控制台中过滤日志。
  • 我们还可以针对特定的命名空间过滤日志，然后可以对日志条目进行相应的处理。这里使用的是 docker 日志处理器。我们也可以针对不同的命名空间使用不同的多行模式。
  • 因为我们使用 Elasticsearch 作为存储后端，所以输出设置为 Elasticsearch。另外，这也可以指向 Redis、Logstash、Kafka 甚至是一个 File。更多信息可以查看以下链接： https://www.elastic.co/guide/en/beats/filebeat/current/configuring-output.html
  • 云元数据处理器在日志条目中包含一些特定主机的字段。当我们试图过滤特定 worker 节点的日志时，这很有帮助。

部署 Filebeat DaemonSet

使用以下 manifest 来部署 Filebeat DaemonSet：

代码语言：javascript
复制
apiVersion: v1
kind: Namespace
metadata:
  name: logging
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: logging
  labels:
    k8s-app: filebeat
spec:
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
      containers:
      - name: filebeat
        image: elastic/filebeat:6.5.4
        args: [
          "-c", "/usr/share/filebeat/filebeat.yml",
          "-e",
        ]
        env:
        - name: ELASTICSEARCH_HOST
          value: elasticsearch.elasticsearch
        - name: ELASTICSEARCH_PORT
          value: "9200"
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /usr/share/filebeat/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: inputs
          mountPath: /usr/share/filebeat/inputs.d
          readOnly: true
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: inputs
        configMap:
          defaultMode: 0600
          name: filebeat-inputs
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
---

让我们来看看这里发生了什么：

  • 每个 pod 的日志都被写入 /var/log/docker/containers 。我们将这个目录从主机挂载到 Filebeat pod 上，然后 Filebeat 根据提供的配置处理日志。
  • 我们将环境变量 ELASTICSEARCH_HOST 设置为 elasticsearch.elasticsearch，以引用本教程第一部分创建的 Elasticsearch 客户端服务。如果你已经有一个 Elasticsearch 集群在运行，环境变量应该设置为指向它。

请注意 manifest 中的以下设置：

代码语言：javascript
复制
...
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
...

这确保了我们的 Filebeat DaemonSet 也会在 master 节点上调度一个 pod。一旦部署了 Filebeat DaemonSet，我们就可以检查我们的 pod 是否被正确调度。

代码语言：javascript
复制
root$ kubectl -n logging get pods  -o wide
NAME             READY   STATUS    RESTARTS   AGE   IP            NODE                                         NOMINATED NODE   READINESS GATES
filebeat-4kchs   1/1     Running   0          6d    100.96.8.2    ip-10-10-30-206.us-east-2.compute.internal   <none>           <none>
filebeat-6nrpc   1/1     Running   0          6d    100.96.7.6    ip-10-10-29-252.us-east-2.compute.internal   <none>           <none>
filebeat-7qs2s   1/1     Running   0          6d    100.96.1.6    ip-10-10-30-161.us-east-2.compute.internal   <none>           <none>
filebeat-j5xz6   1/1     Running   0          6d    100.96.5.3    ip-10-10-28-186.us-east-2.compute.internal   <none>           <none>
filebeat-pskg5   1/1     Running   0          6d    100.96.64.4   ip-10-10-29-142.us-east-2.compute.internal   <none>           <none>
filebeat-vjdtg   1/1     Running   0          6d    100.96.65.3   ip-10-10-30-118.us-east-2.compute.internal   <none>           <none>
filebeat-wm24j   1/1     Running   0          6d    100.96.0.4    ip-10-10-28-162.us-east-2.compute.internal   <none>           <none>

root$ kubectl -get nodes -o wide
NAME                                         STATUS   ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION   CONTAINER-RUNTIME
ip-10-10-28-162.us-east-2.compute.internal   Ready    master   6d    v1.14.8   10.10.28.162   <none>        Debian GNU/Linux 9 (stretch)   4.9.0-9-amd64    docker://18.6.3
ip-10-10-28-186.us-east-2.compute.internal   Ready    node     6d    v1.14.8   10.10.28.186   <none>        Debian GNU/Linux 9 (stretch)   4.9.0-9-amd64    docker://18.6.3
ip-10-10-29-142.us-east-2.compute.internal   Ready    master   6d    v1.14.8   10.10.29.142   <none>        Debian GNU/Linux 9 (stretch)   4.9.0-9-amd64    docker://18.6.3
ip-10-10-29-252.us-east-2.compute.internal   Ready    node     6d    v1.14.8   10.10.29.252   <none>        Debian GNU/Linux 9 (stretch)   4.9.0-9-amd64    docker://18.6.3
ip-10-10-30-118.us-east-2.compute.internal   Ready    master   6d    v1.14.8   10.10.30.118   <none>        Debian GNU/Linux 9 (stretch)   4.9.0-9-amd64    docker://18.6.3
ip-10-10-30-161.us-east-2.compute.internal   Ready    node     6d    v1.14.8   10.10.30.161   <none>        Debian GNU/Linux 9 (stretch)   4.9.0-9-amd64    docker://18.6.3
ip-10-10-30-206.us-east-2.compute.internal   Ready    node     6d    v1.14.8   10.10.30.206

如果我们跟踪其中一个 pod 的日志，我们可以清楚地看到它连接到 Elasticsearch，并且已经启动了文件的harvester。以下代码段可以看到这一点：

代码语言：javascript
复制
2019-11-19T06:22:03.435Z  INFO  log/input.go:138  Configured paths: [/var/lib/docker/containers/c2b29f5e06eb8affb2cce7cf2501f6f824a2fd83418d09823faf4e74a5a51eb7/*.log]
2019-11-19T06:22:03.435Z  INFO  input/input.go:114  Starting input of type: docker; ID: 4134444498769889169 
2019-11-19T06:22:04.786Z  INFO  input/input.go:149  input ticker stopped
2019-11-19T06:22:04.786Z  INFO  input/input.go:167  Stopping Input: 4134444498769889169
2019-11-19T06:22:19.295Z  INFO  [monitoring]  log/log.go:144  Non-zero metrics in the last 30s  {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":641680,"time":{"ms":16}},"total":{"ticks":2471920,"time":{"ms":180},"value":2471920},"user":{"ticks":1830240,"time":{"ms":164}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":20},"info":{"ephemeral_id":"007e8090-7c62-4b44-97fb-e74e8177dc54","uptime":{"ms":549390018}},"memstats":{"gc_next":47281968,"memory_alloc":29021760,"memory_total":156062982472}},"filebeat":{"events":{"added":111,"done":111},"harvester":{"closed":2,"open_files":15,"running":13}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":108,"batches":15,"total":108},"read":{"bytes":69},"write":{"bytes":123536}},"pipeline":{"clients":1847,"events":{"active":0,"filtered":3,"published":108,"total":111},"queue":{"acked":108}}},"registrar":{"states":{"current":87,"update":111},"writes":{"success":18,"total":18}},"system":{"load":{"1":0.98,"15":1.71,"5":1.59,"norm":{"1":0.0613,"15":0.1069,"5":0.0994}}}}}}

2019-11-19T06:22:49.295Z  INFO  [monitoring]  log/log.go:144  Non-zero metrics in the last 30s  {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":641720,"time":{"ms":44}},"total":{"ticks":2472030,"time":{"ms":116},"value":2472030},"user":{"ticks":1830310,"time":{"ms":72}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":20},"info":{"ephemeral_id":"007e8090-7c62-4b44-97fb-e74e8177dc54","uptime":{"ms":549420018}},"memstats":{"gc_next":47281968,"memory_alloc":38715472,"memory_total":156072676184}},"filebeat":{"events":{"active":12,"added":218,"done":206},"harvester":{"open_files":15,"running":13}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":206,"batches":24,"total":206},"read":{"bytes":102},"write":{"bytes":269666}},"pipeline":{"clients":1847,"events":{"active":12,"published":218,"total":218},"queue":{"acked":206}}},"registrar":{"states":{"current":87,"update":206},"writes":{"success":24,"total":24}},"system":{"load":{"1":1.22,"15":1.7,"5":1.58,"norm":{"1":0.0763,"15":0.1063,"5":0.0988}}}}}}

2019-11-19T06:23:19.295Z  INFO  [monitoring]  log/log.go:144  Non-zero metrics in the last 30s  {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":641750,"time":{"ms":28}},"total":{"ticks":2472110,"time":{"ms":72},"value":2472110},"user":{"ticks":1830360,"time":{"ms":44}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":20},"info":{"ephemeral_id":"007e8090-7c62-4b44-97fb-e74e8177dc54","uptime":{"ms":549450017}},"memstats":{"gc_next":47281968,"memory_alloc":43140256,"memory_total":156077100968}},"filebeat":{"events":{"active":-12,"added":43,"done":55},"harvester":{"open_files":15,"running":13}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":55,"batches":12,"total":55},"read":{"bytes":51},"write":{"bytes":70798}},"pipeline":{"clients":1847,"events":{"active":0,"published":43,"total":43},"queue":{"acked":55}}},"registrar":{"states":{"current":87,"update":55},"writes":{"success":12,"total":12}},"system":{"load":{"1":0.99,"15":1.67,"5":1.49,"norm":{"1":0.0619,"15":0.1044,"5":0.0931}}}}}}

2019-11-19T06:23:25.261Z  INFO  log/harvester.go:255  Harvester started for file: /var/lib/docker/containers/ccb7dc75ecc755734f6befc4965b9fdae74d59810914101eadf63daa69eb62e2/ccb7dc75ecc755734f6befc4965b9fdae74d59810914101eadf63daa69eb62e2-json.log

2019-11-19T06:23:49.295Z  INFO  [monitoring]  log/log.go:144  Non-zero metrics in the last 30s  {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":641780,"time":{"ms":28}},"total":{"ticks":2472310,"time":{"ms":196},"value":2472310},"user":{"ticks":1830530,"time":{"ms":168}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":21},"info":{"ephemeral_id":"007e8090-7c62-4b44-97fb-e74e8177dc54","uptime":{"ms":549480018}},"memstats":{"gc_next":47789200,"memory_alloc":31372376,"memory_total":156086697176,"rss":-1064960}},"filebeat":{"events":{"active":16,"added":170,"done":154},"harvester":{"open_files":16,"running":14,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":153,"batches":24,"total":153},"read":{"bytes":115},"write":{"bytes":207569}},"pipeline":{"clients":1847,"events":{"active":16,"filtered":1,"published":169,"total":170},"queue":{"acked":153}}},"registrar":{"states":{"current":87,"update":154},"writes":{"success":25,"total":25}},"system":{"load":{"1":0.87,"15":1.63,"5":1.41,"norm":{"1":0.0544,"15":0.1019,"5":0.0881}}}}}}

一旦我们运行了所有的 pods，那么我们就可以在 Kibana 中创建一个 filebeat-* 类型的索引模式。Filebeat 索引一般都是有时间戳的。只要我们创建了索引模式，就可以看到所有可搜索的可用字段，并导入。最后，我们可以搜索我们的应用程序日志，并在需要时创建 dashboard。强烈建议在我们的应用程序中使用 JSON logger，因为它使日志处理变得非常容易，并且可以轻松地解析消息。

总结

我们的日志堆栈方案配置到此结束。所有提供的配置文件都已经在生产环境中进行了测试，并且是可以随时部署的。欢迎实践，也十分欢迎你向我们分享关于 Kubernetes 的各种实践。

原文链接： https://appfleet.com/blog/part-2-efk-stack-on-kubernetes/

  • 发表于:
  • 本文为 InfoQ 中文站特供稿件
  • 首发地址：https://www.infoq.cn/article/0ATtUVqlxtV4sPUJWE4W
  • 如有侵权，请联系 cloudcommunity@tencent.com 删除。

相关快讯

扫码

添加站长 进交流群

领取专属 10元无门槛券

私享最新 技术干货

扫码加入开发者社群

腾讯云开发者

扫码关注腾讯云开发者

扫码关注腾讯云开发者

领取腾讯云代金券

热门产品

热门推荐

更多推荐

Copyright © 2013 - 2024 Tencent Cloud. All Rights Reserved. 腾讯云 版权所有

深圳市腾讯计算机系统有限公司 ICP备案/许可证号：粤B2-20090059 深公网安备号 44030502008569

腾讯云计算（北京）有限责任公司 京ICP证150476号 | 京ICP备11018762号 | 京公网安备号11010802020287

领券