《Prometheus监控实战》第9章 日志监控

docker run -d --rm -p 9256:9256 --privileged -v /proc:/host/proc -v `pwd`:/config ncabatoff/process-exporter --procfs /host/proc -config.path /config/filename.yml

wget https://github.com/google/mtail/releases/download/v3.0.0-rc33/mtail_v3.0.0-rc33_linux_amd64 -0 mtail
chmod 0755 mtail
sudo cp mtail /usr/local/bin

mtail --version

sudo mkdir /etc/mtail

sudo touch /etc/mtail/line_count.mtail

counter line_count

/$/ {
  line_count++
}

/foo/ {
  ACTION1
} else {
  ACTION2
}

sudo mtail --progs /etc/mtail --logs '/var/log/*.log'

func Match
func Match(pattern, name string) (matched bool, err error)
Match reports whether name matches the shell file name pattern. The pattern syntax is:

pattern:
    { term }
term:
    '*'         matches any sequence of non-Separator characters
    '?'         matches any single non-Separator character
    '[' [ '^' ] { character-range } ']'
                character class (must be non-empty)
    c           matches character c (c != '*', '?', '\\', '[')
    '\\' c      matches character c

character-range:
    c           matches character c (c != '\\', '-', ']')
    '\\' c      matches character c
    lo '-' hi   matches character c for lo <= c <= hi
Match requires pattern to match all of name, not just a substring. The only possible returned error is ErrBadPattern, when pattern is malformed.

On Windows, escaping is disabled. Instead, '\\' is treated as path separator.

# Parser for the common apache "NCSA extended/combined" log format
# LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"
counter apache_http_requests_total by request_method, http_version, request_status
counter apache_http_bytes_total by request_method, http_version, request_status

/^/ +
/(?P<hostname>[0-9A-Za-z\.:-]+) / + # %h
/(?P<remote_logname>[0-9A-Za-z-]+) / + # %l
/(?P<remote_username>[0-9A-Za-z-]+) / + # %u
/\[(?P<timestamp>\d{2}\/\w{3}\/\d{4}:\d{2}:\d{2}:\d{2} (\+|-)\d{4})\] / + # %u
/"(?P<request_method>[A-Z]+) (?P<URI>\S+) (?P<http_version>HTTP\/[0-9\.]+)" / + # \"%r\"
/(?P<request_status>\d{3}) / + # %>s
/((?P<response_size>\d+)|-) / + # %b
/"(?P<referer>\S+)" / + # \"%{Referer}i\"
/"(?P<user_agent>[[:print:]]+)"/ + # \"%{User-agent}i\"
/$/ {
  strptime($timestamp, "02/Jan/2006:15:04:05 -0700") # for tests

  apache_http_requests_total[$request_method][$http_version][$request_status]++
  $response_size > 0 {
      apache_http_bytes_total[$request_method][$http_version][$request_status] += $response_size
  }
}

counter apache_http_requests_total by request_method, http_version, request_status
counter apache_http_bytes_total by request_method, http_version, request_status

Constant pattern fragments
To re-use parts of regular expressions, you can assign them to a const identifier:

const PREFIX /^\w+\W+\d+ /

PREFIX {
  ACTION1
}

PREFIX + /foo/ {
  ACTION2
}
In this example, ACTION1 is done for every line that starts with the prefix regex, and ACTION2 is done for the subset of those lines that also contain 'foo'.

Pattern fragments like this don't need to be prefixes, they can be anywhere in the expression.

counter maybe_ipv4

const IPv4 /(?P<ip>\d+\.\d+\.\d+\.\d+)/

/something with an / + IPv4 + / address/ {
  maybe_ipv4++
}

(?P<request_status>\d{3})

Capture Groups
Regular expressions in patterns can contain capture groups -- subexpressions wrapped in parentheses. These can be referred to in the action block to extract data from the line being matched.

For example, part of a program that can extract from rsyncd logs may want to break down transfers by operation and module.

counter transfers_total by operation, module

/(?P<operation>\S+) (\S+) \[\S+\] (\S+) \(\S*\) \S+ (?P<bytes>\d+)/ {
  transfers_total[$operation][$3]++
}
Or, the value of the counter can be increased by the value of a capture group:

counter bytes_total by operation, module

/(?P<operation>\S+) (\S+) \[\S+\] (\S+) \(\S*\) \S+ (?P<bytes>\d+)/ {
  bytes_total[$operation][$3] += $bytes
}
Numeric capture groups address subexpressions in the match result as you might expect from regular expression groups in other languages, like awk and perl -- e.g. the expression $3 refers to the third capture group in the regular expression.

Named capture groups can be referred to by their name as indicated in the regular expression using the ?P<name> notation, as popularised by the Python regular expression library -- e.g. $bytes refers to (?P<bytes>\d+) in the examples above.

Capture groups can be used in the same expression that defines them, for example in this expression that matches and produces $x, then compares against that value.

/(?P<x>\d+)/ && $x > 1 {
  nonzero_positives++
}

{
  apache_http_requests_total[$request_method][$http_version][$request_status]++
  apache_http_bytes_total[$request_method][$http_version][$request_status] += $response_size
}

sudo mtail --progs /etc/mtail --logs '/var/log/apache/*.access'

counter rails_requests_started_total
counter rails_requests_started by verb

counter rails_requests_completed_total
counter rails_requests_completed by status

histogram rails_requests_completed_seconds by status buckets 0.005, 0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 15.0

/^Started (?P<verb>[A-Z]+) .*/ {
  ###
  # Started HTTP requests by verb (GET, POST, etc.)
  #
  rails_requests_started_total++
  rails_requests_started[$verb]++
}

/^Completed (?P<status>\d{3}) .+ in (?P<request_seconds>\d+)ms .*$/ {
  ###
  # Total numer of completed requests by status
  #
  rails_requests_completed_total++
  rails_requests_completed[$status]++

  ###
  # Completed requests by status with histogram buckets
  #
  # These statements "fall through", so the histogram is cumulative.  The
  # collecting system can compute the percentile bands by taking the ratio of
  # each bucket value over the final bucket.

  rails_requests_completed_seconds[$status] = $request_seconds / 1000.0
}

scrape_configs:
- job_name: 'mtail'
  file_sd_configs:
    - files:
      - targets/mtail/*.json
      refresh_interval: 5m

[{
  "targets": [
    "web:3903",
    "rails:3903"
  ]
}]