跳到主要内容

kubernetes 命名空间删不掉的问题

· 阅读需 1 分钟

kubernetes里命名空间删不掉的问题​

如果某个命名空间(此例里是ingress-nginx)迟迟删除不掉，状态一直是Terminating，然后在此命名空间里重新创建资源时报如下错误：

Error from server (Forbidden): error when creating "nginx-controller.yaml": roles.rbac.authorization.k8s.io "ingress-nginx-admission" is forbidden: unable to create new content in namespace ingress-nginx because it is being terminated

解决办法

# 在第一个终端里运行
kubectl proxy

# 在第二个终端里执行
kubectl get namespace ingress-nginx -o json > xx.json
# 用 vim 编辑 xx.json 
"spec": {
    "finalizers": [
        "kubernetes"  # 这一行删掉
    ]
},

# 或者直接命令行删除
sed -i '/"finalizers"/{n;d}' xx.json

# 然后执行命令即可删除
curl -k -H "Content-Type: application/json" -X PUT --data-binary @xx.json \
http://127.0.0.1:8001/api/v1/namespaces/ingress-nginx/finalize

上面方法可能会残留一些其他资源，可以试一下这个​

kubectl api-resources --verbs=list --namespaced -o name \
  | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace>

Shadowsocket5 服务搭建

· 阅读需 2 分钟

Shadowsocket5 服务搭建​

容器安装

# 开放 iptables 端口
iptables -P FORWARD ACCEPT

# 关闭 swap
swapoff -a
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

# 关闭 selinux
sed -ri 's#(SELINUX=).*#\1disabled#' /etc/selinux/config
setenforce 0

# 关闭防火墙
systemctl disable firewalld && systemctl stop firewalld

# 安装 docker 源
curl -o /etc/yum.repos.d/docker-ce.repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

# 下载docker并设置开机启动
yum install -y docker-ce-cli-19.03.9-3.el7  docker-ce-19.03.9-3.el7
systemctl enable docker && systemctl start docker

# 拉取镜像并启动容器
docker pull  shadowsocks/shadowsocks-libev:latest

# -k 设置密码， -m 设置加密方式
docker run -d --restart=always -p 8388:8388 -p 8388:8388/udp shadowsocks/shadowsocks-libev ss-server -p 8388 -k Kk9nxyGweDXzOM1WhJ9MftjWbSg -m aes-256-gcm

pip 直接安装

# 安装 epel 源
yum update && yum -y install epel-release

# 安装python-pip
yum install python-pip

# 升级pip到最新版本
pip install --upgrade pip

# 安装shadowsocks
pip install shadowsocks

# 创建shadowsocks配置文件，内容如下
# - 配置文件路径：/etc/shadowsocks.json。

{
  "server":"0.0.0.0", 
  "server_port":13579,
  "local_address": "127.0.0.1",
  "local_port":1080,
  "password":"gogen123",
  "timeout":300,
  "method":"aes-256-cfb",
  "fast_open": false
}

# 启动服务
ssserver -c /etc/shadowsocks.json -d start

Prometheus 监控 ingress-nginx-controller

· 阅读需 2 分钟

Prometheus 监控 ingress-nginx-controller​

官方文档

检查ingress是否暴露出端口​

# 查看是否有内容
http://xx.xx.xx.xx:10254/metrics

# 如果没有内容，添加部分内容
vim mandatory.yaml
apiVersion: v1
kind: Deployment
metadata:
 annotations:
   prometheus.io/scrape: "true"
   prometheus.io/port: "10254"
..
spec:
  ports:
    - name: prometheus
      containerPort: 10254
      ..

配置 Prometheus 配置文件​

vim prometheus.yml 
- job_name: 'ingress-nginx-controller exporter'
    static_configs:
    - targets: ['xx.xx.xx.xx:10254']

配置 Grafana​

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/grafana/dashboards/nginx.json

# 倒入 JSON 文件

![grafana 展示](/img/Prometheus 监控 ingress-nginx-controller/ingress-exporter.png)

编写告警信息​

cd rules && vim ingress_rules.yaml 

groups:
- name: Ingress_monitor
  rules:
  - alert: 4xx (> 5%) 的 HTTP 请求过多
    expr: sum(rate(nginx_ingress_controller_requests{status=~"^4.."}[1m])) /  sum(rate(nginx_ingress_controller_requests[1m])) * 100 >= 5
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: Nginx high HTTP 4xx error rate (instance {{ $labels.instance }})
      description: "Too many HTTP requests with status 4xx (> 5%)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

  - alert: 5xx (> 5%) 的 HTTP 请求过多
    expr: sum(rate(nginx_ingress_controller_requests{status=~"^5.."}[1m])) /  sum(rate(nginx_ingress_controller_requests[1m])) * 100 >= 5
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: Nginx high HTTP 4xx error rate (instance {{ $labels.instance }})
      description: "Too many HTTP requests with status 5xx (> 5%)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
      
  - alert: ingress-nginx 延迟高于3秒
    expr: histogram_quantile(0.99, sum(rate(nginx_http_request_duration_seconds_bucket[2m])) by (host, node)) > 3
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Nginx latency high (instance {{ $labels.instance }})
      description: "Nginx p99 latency is higher than 3 seconds\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"