K8s Ingress的日志切割

原创

保持热爱奔赴山海

修改于 2021-02-14 07:15:52

2.3K00

代码可运行

文章被收录于专栏：DevOpsDevOps 数据库相关

运行总次数：0

代码可运行

提醒：本文是在minikube上实验的，不同环境的ingress请参考官方文档

ingress的安装

ingress的安装部署，不是这里的重点，忽略。我实验用的是minikube，和云上环境可能差异较大。

在宿主机上创建日志路径并授权

mkdir /data/logs/nginx
chmod 777  /data/logs/nginx -R

修改ingress-nginx-controller的deployment的文件

1、增加initContainer配置

2、增加hostpath配置

3、增加localtime配置

$ kubectl get deployments.apps -n ingress-nginx                                                                                                    
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
ingress-nginx-controller   1/1     1            1           24h


具体内容如下：
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 0.44.0
    helm.sh/chart: ingress-nginx-3.23.0
  name: ingress-nginx-controller
  namespace: ingress-nginx
  resourceVersion: "16309799"
  selfLink: /apis/apps/v1/namespaces/ingress-nginx/deployments/ingress-nginx-controller
  uid: 6b3045e1-b8a6-47bb-b9b6-ca6e0d2fb964
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/name: ingress-nginx
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --election-id=ingress-controller-leader
        - --ingress-class=nginx
        - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
        - --validating-webhook=:8443
        - --validating-webhook-certificate=/usr/local/certificates/cert
        - --validating-webhook-key=/usr/local/certificates/key
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: LD_PRELOAD
          value: /usr/local/lib/libmimalloc.so
        image: k8s.gcr.io/ingress-nginx/controller:v0.44.0@sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a
        imagePullPolicy: IfNotPresent
        lifecycle:
          preStop:
            exec:
              command:
              - /wait-shutdown
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: controller
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          name: https
          protocol: TCP
        - containerPort: 8443
          name: webhook
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          requests:
            cpu: 100m
            memory: 90Mi
        securityContext:
          allowPrivilegeEscalation: true
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - ALL
          runAsUser: 101
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/local/certificates/
          name: webhook-cert
          readOnly: true
        - mountPath: /etc/localtime
          name: localtime
          readOnly: true
        - mountPath: /var/log/nginx/
          name: logdir
      dnsPolicy: ClusterFirst
      initContainers:
      - command:
        - /bin/sh
        - -c
        - |
          mount -o remount rw /proc/sys
          sysctl -w net.core.somaxconn=65535
          sysctl -w net.ipv4.ip_local_port_range="1024 65535"
          sysctl -w fs.file-max=1048576
          sysctl -w fs.inotify.max_user_instances=16384
          sysctl -w fs.inotify.max_user_watches=524288
          sysctl -w fs.inotify.max_queued_events=16384
        image: busybox
        imagePullPolicy: IfNotPresent
        name: init-sysctl
        resources: {}
        securityContext:
          capabilities:
            add:
            - SYS_ADMIN
            drop:
            - ALL
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: ingress-nginx
      serviceAccountName: ingress-nginx
      terminationGracePeriodSeconds: 300
      volumes:
      - name: webhook-cert
        secret:
          defaultMode: 420
          secretName: ingress-nginx-admission
      - hostPath:
          path: /etc/localtime
          type: File
        name: localtime
      - hostPath:
          path: /data/logs/nginx/
          type: ""
        name: logdir

优化下ingress-nginx的configmap配置

1、增加了些常用的nginx配置，具体如下

apiVersion: v1
data:
  allow-backend-server-header: "true"
  enable-underscores-in-headers: "true"
  generate-request-id: "true"
  ignore-invalid-headers: "true"
  log-format-upstream: $remote_addr - [$remote_addr] - $remote_user [$time_local]
    "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length
    $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length
    $upstream_response_time $upstream_status $req_id $host
  max-worker-connections: "65536"
  proxy-body-size: 20m
  proxy-connect-timeout: "10"
  reuse-port: "true"
  server-tokens: "false"
  ssl-redirect: "false"
  worker-cpu-affinity: auto
kind: ConfigMap
metadata:
  annotations:
  creationTimestamp: "2021-02-12T07:08:25Z"
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 0.44.0
    helm.sh/chart: ingress-nginx-3.23.0
  name: ingress-nginx-controller
  namespace: ingress-nginx
  resourceVersion: "16310580"
  selfLink: /api/v1/namespaces/ingress-nginx/configmaps/ingress-nginx-controller
  uid: 12b1b526-fa3c-4680-b4fb-e7f0d9ebbb94

ingress-nginx的日志切割脚本

#!/bin/bash
# 最多保留日志文件个数，可根据需求进行调整。
keep_log_num=5

# 阿里云提供的ingress是这个写法，其它的云厂商的可能也不太一样
# ingress_nginx_container_ids=$(docker ps | grep nginx-ingress-controller  | grep -v pause | awk '{print $1}')

# 我在minikube里面用的nginx-ingress是用的下面这个写法
ingress_nginx_container_ids=$(docker ps | grep  ingress-nginx-controller | grep -v pause | awk '{print $1}')

if [[ -z "$ingress_nginx_container_ids" ]]; then
    echo "error: failed to get ingress nginx container ids"
    exit 1
fi
#随机睡眠5~10秒
sleep $(( RANDOM % (10 - 5 + 1 ) + 5 ))
for id in $ingress_nginx_container_ids; do
    docker exec $id  bash -c "cd /var/log/nginx; if [[ \$(ls access.log-* | wc -l) -gt $keep_log_num ]]; then rm -f \$(ls -t access.log-* | tail -1); fi ; mv access.log  access.log-\$(date +%F_%H_%M_%S) ; kill -USR1 \$(cat /tmp/nginx.pid)"
done

# 这里的历史文件，没有压缩。大家可以根据自己情况看是否要改改。

脚本定时执行

# 在每台宿主机上都跑一个cronjob定时任务（或者各种调度平台也行，只要能到宿主机即可），负责执行这个切割日志的脚本

附：一个ingress的demo

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/client-body-buffer-size: 1M
    nginx.ingress.kubernetes.io/configuration-snippet: |
      access_log /var/log/nginx/access.log upstreaminfo if=$loggable;
      error_log  /var/log/nginx/error.log info;
    nginx.ingress.kubernetes.io/connection-proxy-header: keep-alive
    nginx.ingress.kubernetes.io/enable-rewrite-log: "true"
    nginx.ingress.kubernetes.io/limit-connections: "10000"
    nginx.ingress.kubernetes.io/proxy-buffer-size: 16k
    nginx.ingress.kubernetes.io/proxy-buffering: "on"
    nginx.ingress.kubernetes.io/proxy-max-temp-file-size: 1024m
    nginx.ingress.kubernetes.io/server-alias: shanghai.example.com
    nginx.ingress.kubernetes.io/server-snippet: |
      set $agentflag 0;

      if ($http_user_agent ~* "(Mobile)" ){
        set $agentflag 1;
      }

      if ( $agentflag = 1 ) {
        return 301 https://m.example.com;
      }
    nginx.ingress.kubernetes.io/ssl-ciphers: ALL:!aNULL:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP
    nginx.ingress.kubernetes.io/ssl-prefer-server-ciphers: "true"
    nginx.ingress.kubernetes.io/upstream-hash-by: $request_uri
  creationTimestamp: "2021-02-12T03:58:02Z"
  generation: 5
  name: gray-release
  namespace: default
  resourceVersion: "16130870"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/gray-release
  uid: 8b732b00-a1a9-4d74-bfe4-9210a0094b9f
spec:
  rules:
  - host: www.example.com
    http:
      paths:
      - backend:
          serviceName: new-nginx
          servicePort: 80
        path: /
        pathType: ImplementationSpecific

ingrss的日志采集

流程：
1、这里是把全部vhost的日志都打到了 access.log 中
2、部署在k8s的ds模式的fluentd会将日志都采集发送到kafka中
3、logstash消费kafka的数据，并写到ES中
4、在kibana中，根据域名来查不同的域名的访问记录

这样处理起来比较简单，但是在日志量大的情况下，会造成es的index过大，每次查询都需要加载很多不相关的vhost的日志。
这种情况下，有2个解决方法供参考：
1、在logstash层做处理，对访问量大的某些vhost，单独写到独立的es index里，
其余体量小的vhost还是共同写到一个es index中
2、在ingress层处理，在配置access_log的时候，就单独打到独立的文件中。
然后再fluentd中也做个判断，写到不同kafka topic中，logstash单独写个pipeline去处理。