k8s集群中namespace状态一直显示Terminating如何解决
k8s集群中namespace状态一直显示Terminating如何解决
linus_lin
发布于 2024-09-06 19:31:15
发布于 2024-09-06 19:31:15
文章被收录于专栏：SRE云原生实践之路
1、问题的现象
今天在清理测试环境的时候，发现一个ns的状态一直显示Terminating
kubectl get ns输出结果：
NAME STATUS AGE
argocd Active 327d
default Active 327d
ingress-apisix Active 27d
ingress-nginx Terminating 234d
istio-ingress Active 103d
istio-system Active 103d
kube-node-lease Active 327d
kube-public Active 327d
kube-system Active 327d
kubernetes-dashboard Active 327d
observability Active 172d
prometheus Active 255d
redis Terminating 291d使用–force 删除也是一直卡着的状态
kubectl delete ns redis --force 输出结果：
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
namespace "redis" force deleted2、查看ns下的资源
根据以上现象，怀疑是该ns下有未释放的资源，使用如下命令查看
kubectl get all -n redis输出结果
No resources found in redis namespace.3、问题处理方法
以json格式导出ns的详细信息
kubectl get ns redis -o json > redis.json编辑test.json文件，确保spec中内容为空，如下：
"spec": {
"finalizers": [ #########
"kubernetes" ######### 删除这三行内容，告知k8s要删除的ns中内容为空
] #########
},将空ns通过调用k8s的api接口覆盖掉原来的ns
curl -k \
> -H "Content-Type: application/json" \
> -X PUT \
> --data-binary @redis.json \
>http://127.0.0.1:8081/api/v1/namespaces/redis/finalize输出结果：
curl: (7) Failed connect to 127.0.0.1:8081; Connection refused因为k8s主节点使用了认证，如果直接使用命令会拒绝连接,需要使用kube-proxy进行代理8081端口
使用kube-proxy开启端口
kubectl proxy --port=8081 输出结果：
Starting to serve on 127.0.0.1:8081打开新的终端，再次执行上述命令，返回如下内容
curl -k -H "Content-Type: application/json" -X PUT --data-binary @redis.json http://127.0.0.1:8081/api/v1/namespaces/redis/finalize输出结果：
{
"kind": "Namespace",
"apiVersion": "v1",
"metadata": {
"name": "redis",
"uid": "acce79d4-f8a4-4e99-8085-bdf7ef481901",
"resourceVersion": "77013481",
"creationTimestamp": "2023-04-06T08:20:21Z",
"deletionTimestamp": "2024-01-22T09:37:32Z",
"labels": {
"kubernetes.io/metadata.name": "redis"
},
"managedFields": [
{
"manager": "kubectl-create",
"operation": "Update",
"apiVersion": "v1",
"time": "2023-04-06T08:20:21Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:labels": {
".": {},
"f:kubernetes.io/metadata.name": {}
}
}
}
},
{
"manager": "kube-controller-manager",
"operation": "Update",
"apiVersion": "v1",
"time": "2024-01-22T09:37:38Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:status": {
"f:conditions": {
".": {},
"k:{\"type\":\"NamespaceContentRemaining\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
},
"k:{\"type\":\"NamespaceDeletionContentFailure\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
},
"k:{\"type\":\"NamespaceDeletionDiscoveryFailure\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
},
"k:{\"type\":\"NamespaceDeletionGroupVersionParsingFailure\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
},
"k:{\"type\":\"NamespaceFinalizersRemaining\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
}
}
}
},
"subresource": "status"
}
]
},
"spec": {},
"status": {
"phase": "Terminating",
"conditions": [
{
"type": "NamespaceDeletionDiscoveryFailure",
"status": "True",
"lastTransitionTime": "2024-01-22T09:37:37Z",
"reason": "DiscoveryFailed",
"message": "Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"
},
{
"type": "NamespaceDeletionGroupVersionParsingFailure",
"status": "False",
"lastTransitionTime": "2024-01-22T09:37:38Z",
"reason": "ParsedGroupVersions",
"message": "All legacy kube types successfully parsed"
},
{
"type": "NamespaceDeletionContentFailure",
"status": "False",
"lastTransitionTime": "2024-01-22T09:37:38Z",
"reason": "ContentDeleted",
"message": "All content successfully deleted, may be waiting on finalization"
},
{
"type": "NamespaceContentRemaining",
"status": "False",
"lastTransitionTime": "2024-01-22T09:37:38Z",
"reason": "ContentRemoved",
"message": "All content successfully removed"
},
{
"type": "NamespaceFinalizersRemaining",
"status": "False",
"lastTransitionTime": "2024-01-22T09:37:38Z",
"reason": "ContentHasNoFinalizers",
"message": "All content-preserving finalizers finished"
}
]
}
}查看ns是否被删除
kubectl get ns输出结果
NAME STATUS AGE
argocd Active 327d
default Active 327d
ingress-apisix Active 27d
ingress-nginx Terminating 235d
istio-ingress Active 103d
istio-system Active 103d
kube-node-lease Active 327d
kube-public Active 327d
kube-system Active 327d
kubernetes-dashboard Active 327d
observability Active 173d
prometheus Active 255d其他Terminating状态的ns也可以使用上述方法删除
本文参与 腾讯云自媒体同步曝光计划，分享自微信公众号。
原始发表：2024-01-23，如有侵权请联系 cloudcommunity@tencent.com 删除
评论
登录后参与评论
推荐阅读
目录
相关产品与服务
云数据库 Redis
腾讯云数据库 Redis（TencentDB for Redis）是腾讯云打造的兼容 Redis 协议的缓存和存储服务。丰富的数据结构能帮助您完成不同类型的业务场景开发。支持主从热备，提供自动容灾切换、数据备份、故障迁移、实例监控、在线扩容、数据回档等全套的数据库服务。
腾讯云开发者
