背景
linux 中为了防止进程恶意使用资源,系统使用 ulimit 来限制进程的资源使用情况(包括文件描述符,线程数,内存大小等)。同样地在容器化场景中,需要限制其系统资源的使用量。
用户级别资源限制,分为 soft 限制与 hard 限制
修改方式:ulimit 命令,临时修改;/etc/security/limits.conf,永久修改
工作原理:根据 PAM ( Pluggable Authentication Modules 简称 PAM)机制,应用程序启动时,按 /etc/pam.d 配置加载 pam_xxxx.so 模块。/etc/pam.d 下包含了 login 、sshd 、su 、sudo 等程序的 PAM 配置文件, 因此用户重新登录时,将调用 pam_limits.so 加载 limits.conf 配置文件
RLIMIT_NOFILE
This specifies a value one greater than the maximum file
descriptor number that can be opened by this process.
Attempts (open(2), pipe(2), dup(2), etc.) to exceed this
limit yield the error EMFILE. (Historically, this limit was
named RLIMIT_OFILE on BSD.)
Since Linux 4.5, this limit also defines the maximum number of
file descriptors that an unprivileged process (one without the
CAP_SYS_RESOURCE capability) may have "in flight" to other
processes, by being passed across UNIX domain sockets. This
limit applies to the sendmsg(2) system call. For further
details, see unix(7).
根据定义,nofile 限制进程所能最多打开的文件数量,作用范围进程。
$ docker run -d --ulimit nofile=100:200 cr.d.xiaomi.net/containercloud/alpine:webtool top
/ # ulimit -a
-f: file size (blocks) unlimited
-t: cpu time (seconds) unlimited
-d: data seg size (kb) unlimited
-s: stack size (kb) 8192
-c: core file size (blocks) unlimited
-m: resident set size (kb) unlimited
-l: locked memory (kb) 64
-p: processes unlimited
-n: file descriptors 100
-v: address space (kb) unlimited
-w: locks unlimited
-e: scheduling priority 0
-r: real-time priority 0
/ # ab -n 1000000 -c 90 http://61.135.169.125:80/ &
/ # lsof | wc -l
108
/ # lsof | grep -c ab
94
/ # ab -n 1000000 -c 100 http://61.135.169.125:80/
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 61.135.169.125 (be patient)
socket: No file descriptors available (24)
RLIMIT_NPROC
This is a limit on the number of extant process (or, more pre‐
cisely on Linux, threads) for the real user ID of the calling
process. So long as the current number of processes belonging
to this process's real user ID is greater than or equal to
this limit, fork(2) fails with the error EAGAIN.
The RLIMIT_NPROC limit is not enforced for processes that have
either the CAP_SYS_ADMIN or the CAP_SYS_RESOURCE capability.
由定义可知,nproc 进程限制的范围是对于每个 uid,并且对于 root 用户无效。
同一主机上运行的所有容器共享同一个内核(主机的内核),docker 通过 namspace 对 pid/utc/network 等进行了隔离,虽然 docker 中已经实现了 user namespace,但由于各种原因,默认没有开启,见 docker user namespace
$ docker run -d cr.d.xiaomi.net/containercloud/alpine:webtool top
宿主机中查看 top 进程,显示 root 用户
$ ps -ef |grep top
root 4096 4080 0 15:01 ? 00:00:01 top
容器中查看 id,uid 为 0 对应宿主机的 root 用户,虽然同为 root 用户,但 Linux Capabilities 不同,实际权限与宿主机 root 要少很多
在容器中切换用户到 operator(uid 为 11),执行 sleep 命令,主机中查看对应进程用户为 app,对应 uid 同样为 11
/ # id
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)
/ # su operator
/ $ id
uid=11(operator) gid=0(root) groups=0(root)
/ $ sleep 100
$ ps -ef |grep 'sleep 100'
app 19302 19297 0 16:39 pts/0 00:00:00 sleep 100
$ cat /etc/passwd | grep app
app❌11:0::/home/app:
设置 ulimit nproc 限制 soft 10/hard 20,默认启动为 root 用户
$ docker run -d --ulimit nproc=10:20 cr.d.xiaomi.net/containercloud/alpine:webtool top
进入容器查看, fd soft 限制为 100 个
/ # ulimit -a
-f: file size (blocks) unlimited
-t: cpu time (seconds) unlimited
-d: data seg size (kb) unlimited
-s: stack size (kb) 8192
-c: core file size (blocks) unlimited
-m: resident set size (kb) unlimited
-l: locked memory (kb) 64
-p: processes 10
-n: file descriptors 1048576
-v: address space (kb) unlimited
-w: locks unlimited
-e: scheduling priority 0
-r: real-time priority 0
启动 30 个进程
/ # for i in `seq 30`;do sleep 100 &; done
/ # ps | wc -l
36
切换到 operator 用户
/ # su operator
# 启动多个进程,到第11个进程无法进行fork
/ $ for i in `seq 8`; do
> sleep 100 &
> done
/ $ sleep 100 &
/ $ sleep 100 &
sh: can't fork: Resource temporarily unavailable
root 下查看
/ # ps -ef | grep operator
79 operator 0:00 sh
99 operator 0:00 sleep 100
100 operator 0:00 sleep 100
101 operator 0:00 sleep 100
102 operator 0:00 sleep 100
103 operator 0:00 sleep 100
104 operator 0:00 sleep 100
105 operator 0:00 sleep 100
106 operator 0:00 sleep 100
107 operator 0:00 sleep 100
109 root 0:00 grep operator
/ # ps -ef | grep operator| wc -l
10
设置 ulimit nproc 限制 soft 3/hard 3,默认启动为 operator 用户,起 4 个容器,第四个启动失败
$ docker run -d --ulimit nproc=3:3 --name nproc1 -u operator cr.d.xiaomi.net/containercloud/alpine:webtool top
eeb1551bf757ad4f112c61cc48d7cbe959185f65109e4b44f28085f246043e65
$ docker run -d --ulimit nproc=3:3 --name nproc2 -u operator cr.d.xiaomi.net/containercloud/alpine:webtool top
42ff29844565a9cb3af2c8dd560308b1f31306041d3dbd929011d65f1848a262
$ docker run -d --ulimit nproc=3:3 --name nproc3 -u operator cr.d.xiaomi.net/containercloud/alpine:webtool top
b7c9b469e73f969d922841dd77265467959eda28ed06301af8bf83bcf18e8c23
$ docker run -d --ulimit nproc=3:3 --name nproc4 -u operator cr.d.xiaomi.net/containercloud/alpine:webtool top
b49d8bb58757c88f69903059af2ee7e2a6cc2fa5774bc531941194c52edfd763
$
$ docker ps -a |grep nproc
b49d8bb58757 cr.d.xiaomi.net/containercloud/alpine:webtool "top" 16 seconds ago Exited (1) 15 seconds ago nproc4
b7c9b469e73f cr.d.xiaomi.net/containercloud/alpine:webtool "top" 23 seconds ago Up 22 seconds nproc3
42ff29844565 cr.d.xiaomi.net/containercloud/alpine:webtool "top" 31 seconds ago Up 29 seconds nproc2
eeb1551bf757 cr.d.xiaomi.net/containercloud/alpine:webtool "top" 38 seconds ago Up 36 seconds nproc1
cgroup 中对 pid 进行了隔离,通过更改 docker/kubelet 配置,可以限制 pid 总数,从而达到限制线程总数的目的。线程数限制与系统中多处配置有关,取最小值,参考 stackoverflow 上线程数的设置
[root@node01 ~]# ps -ef |grep kubelet
root 18735 1 14 11:19 ? 00:53:28 ./kubelet --v=1 --address=0.0.0.0 --feature-gates=SupportPodPidsLimit=true --pod-max-pids=150 --allow-privileged=true --pod-infra-container-image=cr.d.xiaomi.net/kubernetes/pause-amd64:3.1 --root-dir=/home/kubelet --node-status-update-frequency=5s --kubeconfig=/home/xbox/kubelet/conf/kubelet-kubeconfig --fail-swap-on=false --max-pods=254 --runtime-cgroups=/systemd/system.slice/frigga.service --kubelet-cgroups=/systemd/system.slice/frigga.service --make-iptables-util-chains=false
/ # for i in `seq 100`; do
> sleep 1000 &
> done
/ # ps | wc -l
106
/ # su operator
/ $
/ $ for i in `seq 100`; do
> sleep 1000 &
> done
sh: can't fork: Resource temporarily unavailable
/ $ ps | wc -l
150
[root@node01 ~]# cat /sys/fs/cgroup/pids/kubepods/besteffort/pod8b61d4de-a7ad-11e9-b5b9-246e96ad0900/pids.current
150
[root@node01 ~]# cat /sys/fs/cgroup/pids/kubepods/besteffort/pod8b61d4de-a7ad-11e9-b5b9-246e96ad0900/pids.max
150
limits.conf 是 ulimit 的具体配置,目录项/etc/security/limit.d/
中的配置会覆盖 limits.conf。
sysctl.conf 为机器级别的资源限制,root 用户可修改,目录项/etc/security/sysctl.d/
中的配置会覆盖 sysctl.conf
,在/etc/sysctl.conf
中添加对应配置(fd: fs.file-max = {}; pid: kernel.pid_max = {})
$ docker run -d --ulimit nofile=100:200 cr.d.xiaomi.net/containercloud/alpine:webtool top
cb1250c8fd217258da51c6818fa2ce2e2f6e35bf1d52648f1f432e6ce579cf0d
$ docker exec -it cb1250c sh
/ # ulimit -a
-f: file size (blocks) unlimited
-t: cpu time (seconds) unlimited
-d: data seg size (kb) unlimited
-s: stack size (kb) 8192
-c: core file size (blocks) unlimited
-m: resident set size (kb) unlimited
-l: locked memory (kb) 64
-p: processes unlimited
-n: file descriptors 100
-v: address space (kb) unlimited
-w: locks unlimited
-e: scheduling priority 0
-r: real-time priority 0
/ #
/ # echo 10 > /proc/sys/kernel/pid_max
sh: can't create /proc/sys/kernel/pid_max: Read-only file system
/ # echo 10 > /proc/sys/kernel/pid_max
sh: can't create /proc/sys/kernel/pid_max: Read-only file system
/ # echo "fs.file-max=5" >> /etc/sysctl.conf
/ # sysctl -p
sysctl: error setting key 'fs.file-max': Read-only file system
$ cat /proc/sys/kernel/pid_max
32768
$ docker run -d -- --ulimit nofile=100:200 cr.d.xiaomi.net/containercloud/alpine:webtool top
$ docker exec -it pedantic_vaughan sh
/ # cat /proc/sys/kernel/pid_max
32768
/ # echo 50000 > /proc/sys/kernel/pid_max
/ # cat /proc/sys/kernel/pid_max
50000
/ # exit
$ cat /proc/sys/kernel/pid_max
50000 # 宿主机的文件也变成50000
推荐方案如下:
本公众号【云原生生态圈】(以下均称为本公众号)分享的软件服务以及技术方案均来源于互联网,主要是对互联网上出现的开源产品、技术解决方案以及部分编程语言的实践使用进行分享和整理。本公众号不对任何人进行相关技术的方案的推荐,如果您使用文章中涉及到软件或拷贝了相关代码比如说造成了相关生产事故、甚至导致数据丢失,请您自行承担相应的后果!本公众号维护者概不负责! 对于本公众号的所有原创文章,均是受益于互联网学习后,个人总结整理而来,欢迎大家在技术实践上能够多相互交流与学习,您可以在文章底部进行留言回复,也可以在公众号内添加作者有素质、有文化的、礼貌的进行微信交流。若您觉得公众号发布的内容若侵犯到您的权益,请联系即时管理员沟通!
云原生生态圈 · 往期推荐