本文分享自华为云社区《K8s 镜像缓存管理 kube-fledged 认知-云社区-华为云》，作者： 山河已无恙。

我们知道 k8s 上的容器调度需要在调度的节点行拉取当前容器的镜像，在一些特殊场景中，

需要快速启动和/或扩展的应用程序。例如，由于数据量激增，执行实时数据处理的应用程序需要快速扩展。镜像比较庞大，涉及多个版本，节点存储有限，需要动态清理不需要的镜像无服务器函数通常需要在几分之一秒内立即对传入事件和启动容器做出反应。在边缘设备上运行的 IoT 应用程序，需要容忍边缘设备和镜像镜像仓库之间的间歇性网络连接。如果需要从专用仓库中拉取镜像，并且无法授予每个人从此镜像仓库拉取镜像的访问权限，则可以在群集的节点上提供镜像。如果集群管理员或操作员需要对应用程序进行升级，并希望事先验证是否可以成功拉取新镜像。

kube-fledged 是一个 kubernetes operator，用于直接在 Kubernetes 集群的 worker 节点上创建和管理容器镜像缓存。它允许用户定义镜像列表以及这些镜像应缓存到哪些工作节点上（即拉取）。因此，应用程序 Pod 几乎可以立即启动，因为不需要从镜像仓库中提取镜像。

kube-fledged 提供了 CRUD API 来管理镜像缓存的生命周期，并支持多个可配置的参数，可以根据自己的需要自定义功能。

Kubernetes 具有内置的镜像垃圾回收机制。节点中的 kubelet 会定期检查磁盘使用率是否达到特定阈值（可通过标志进行配置）。一旦达到这个阈值，kubelet 会自动删除节点中所有未使用的镜像。

需要在建议的解决方案中实现自动和定期刷新机制。如果镜像缓存中的镜像被 kubelet 的 gc 删除，下一个刷新周期会将已删除的镜像拉入镜像缓存中。这可确保镜像缓存是最新的。

设计流程

部署 kube-fledgedHelm 方式部署

──[root@vms100.liruilongs.github.io]-[~/ansible]└─$mkdir  kube-fledged┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$cd kube-fledged┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]└─$export KUBEFLEDGED_NAMESPACE=kube-fledged┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]└─$kubectl create namespace ${KUBEFLEDGED_NAMESPACE}namespace/kube-fledged created┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]└─$helm repo add kubefledged-charts ;kubefledged-charts" has been added to your repositories┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]└─$helm repo updateHang tight while we grab the latest from your chart repositories......Successfully got an update from the "kubefledged-charts" chart repository...Successfully got an update from the "kubescape" chart repository...Successfully got an update from the "rancher-stable" chart repository...Successfully got an update from the "skm" chart repository...Successfully got an update from the "openkruise" chart repository...Successfully got an update from the "awx-operator" chart repository...Successfully got an update from the "botkube" chart repositoryUpdate Complete. ⎈Happy Helming!⎈

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]└─$helm install --verify kube-fledged kubefledged-charts/kube-fledged -n ${KUBEFLEDGED_NAMESPACE} --wait

实际部署中发现，由于网络问题，chart 无法下载，所以通过 make deploy-using-yaml 使用 yaml 方式部署

Yaml 文件部署

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]└─$git clone 正克隆到 'kube-fledged'...remote: Enumerating objects: 10613, done.remote: Counting objects: 100% (1501/1501), done.remote: Compressing objects: 100% (629/629), done.remote: Total 10613 (delta 845), reused 1357 (delta 766), pack-reused 9112接收对象中: 100% (10613/10613), 34.58 MiB | 7.33 MiB/s, done.处理 delta 中: 100% (4431/4431), done.┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]└─$lskube-fledged┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]└─$cd kube-fledged/┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]└─$make deploy-using-yamlkubectl apply -f deploy/kubefledged-namespace.yaml

第一次部署，发现镜像拉不下来

┌──[root@vms100.liruilongs.github.io]-[~]└─$kubectl get all -n kube-fledgedNAME                                               READY   STATUS                  RESTARTS         AGEpod/kube-fledged-controller-df69f6565-drrqg        0/1     CrashLoopBackOff        35 (5h59m ago)   21hpod/kube-fledged-webhook-server-7bcd589bc4-b7kg2   0/1     Init:CrashLoopBackOff   35 (5h58m ago)   21hpod/kubefledged-controller-55f848cc67-7f4rl        1/1     Running                 0                21hpod/kubefledged-webhook-server-597dbf4ff5-l8fbh    0/1     Init:CrashLoopBackOff   34 (6h ago)      21hNAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGEservice/kube-fledged-webhook-server   ClusterIP   10.100.194.199   <none>        3443/TCP   21hservice/kubefledged-webhook-server    ClusterIP   10.101.191.206   <none>        3443/TCP   21hNAME                                          READY   UP-TO-DATE   AVAILABLE   AGEdeployment.apps/kube-fledged-controller       0/1     1            0           21hdeployment.apps/kube-fledged-webhook-server   0/1     1            0           21hdeployment.apps/kubefledged-controller        0/1     1            0           21hdeployment.apps/kubefledged-webhook-server    0/1     1            0           21hNAME                                                     DESIRED   CURRENT   READY   AGEreplicaset.apps/kube-fledged-controller-df69f6565        1         1         0       21hreplicaset.apps/kube-fledged-webhook-server-7bcd589bc4   1         1         0       21hreplicaset.apps/kubefledged-controller-55f848cc67        1         1         0       21hreplicaset.apps/kubefledged-webhook-server-597dbf4ff5    1         1         0       21h┌──[root@vms100.liruilongs.github.io]-[~]└─$

这里我们找一下要拉取的镜像

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]└─$cat *.yaml | grep image:      - image: senthilrch/kubefledged-controller:v0.10.0      - image: senthilrch/kubefledged-webhook-server:v0.10.0      - image: senthilrch/kubefledged-webhook-server:v0.10.0

单独拉取一些，当前使用 ansible 在所有工作节点批量操作

┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$ansible k8s_node -m shell -a "docker pull docker.io/senthilrch/kubefledged-cri-client:v0.10.0" -i host.yaml

其他相关的镜像都拉取一下

操作完成之后容器状态全部正常

┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$kubectl -n kube-fledged get allNAME                                               READY   STATUS    RESTARTS   AGEpod/kube-fledged-controller-df69f6565-wdb4g        1/1     Running   0          13hpod/kube-fledged-webhook-server-7bcd589bc4-j8xxp   1/1     Running   0          13hpod/kubefledged-controller-55f848cc67-klxlm        1/1     Running   0          13hpod/kubefledged-webhook-server-597dbf4ff5-ktbsh    1/1     Running   0          13hNAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGEservice/kube-fledged-webhook-server   ClusterIP   10.100.194.199   <none>        3443/TCP   36hservice/kubefledged-webhook-server    ClusterIP   10.101.191.206   <none>        3443/TCP   36hNAME                                          READY   UP-TO-DATE   AVAILABLE   AGEdeployment.apps/kube-fledged-controller       1/1     1            1           36hdeployment.apps/kube-fledged-webhook-server   1/1     1            1           36hdeployment.apps/kubefledged-controller        1/1     1            1           36hdeployment.apps/kubefledged-webhook-server    1/1     1            1           36hNAME                                                     DESIRED   CURRENT   READY   AGEreplicaset.apps/kube-fledged-controller-df69f6565        1         1         1       36hreplicaset.apps/kube-fledged-webhook-server-7bcd589bc4   1         1         1       36hreplicaset.apps/kubefledged-controller-55f848cc67        1         1         1       36hreplicaset.apps/kubefledged-webhook-server-597dbf4ff5    1         1         1       36h

验证是否安装成功

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]└─$kubectl get pods -n kube-fledged -l app=kubefledgedNAME                                          READY   STATUS    RESTARTS   AGEkubefledged-controller-55f848cc67-klxlm       1/1     Running   0          16hkubefledged-webhook-server-597dbf4ff5-ktbsh   1/1     Running   0          16h┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]└─$kubectl get imagecaches -n kube-fledgedNo resources found in kube-fledged namespace.

根据 Demo 文件，创建镜像缓存对象

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]└─$cd deploy/┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]└─$cat kubefledged-imagecache.yaml---apiVersion: kubefledged.io/v1alpha2kind: ImageCachemetadata:  # Name of the image cache. A cluster can have multiple image cache objects  name: imagecache1  namespace: kube-fledged  # The kubernetes namespace to be used for this image cache. You can choose a different namepace as per your preference  labels:    app: kubefledged    kubefledged: imagecachespec:  # The "cacheSpec" field allows a user to define a list of images and onto which worker nodes those images should be cached (i.e. pre-pulled).  cacheSpec:  # Specifies a list of images (nginx:1.23.1) with no node selector, hence these images will be cached in all the nodes in the cluster  - images:    - ghcr.io/jitesoft/nginx:1.23.1  # Specifies a list of images (cassandra:v7 and etcd:3.5.4-0) with a node selector, hence these images will be cached only on the nodes selected by the node selector  - images:    - us.gcr.io/k8s-artifacts-prod/cassandra:v7    - us.gcr.io/k8s-artifacts-prod/etcd:3.5.4-0    nodeSelector:      tier: backend  # Specifies a list of image pull secrets to pull images from private repositories into the cache  imagePullSecrets:  - name: myregistrykey

官方的 Demo 中对应的 镜像拉取不下来，所以换一下

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]└─$docker pull us.gcr.io/k8s-artifacts-prod/cassandra:v7Error response from daemon: Get ";: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]└─$

为了测试选择器标签的使用,我们找一个节点的标签单独做镜像缓存

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]└─$kubectl get nodes  --show-labels

同时我们直接从公有仓库拉取镜像，所以不需要 imagePullSecrets 对象

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]└─$vim kubefledged-imagecache.yaml

修改后的 yaml 文件

添加了一个所有节点的 liruilong/my-busybox:latest 镜像缓存添加了一个 kubernetes.io/hostname: vms105.liruilongs.github.io 对应标签选择器的 liruilong/hikvision-sdk-config-ftp:latest 镜像缓存

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]└─$cat kubefledged-imagecache.yaml---apiVersion: kubefledged.io/v1alpha2kind: ImageCachemetadata:  # Name of the image cache. A cluster can have multiple image cache objects  name: imagecache1  namespace: kube-fledged  # The kubernetes namespace to be used for this image cache. You can choose a different namepace as per your preference  labels:    app: kubefledged    kubefledged: imagecachespec:  # The "cacheSpec" field allows a user to define a list of images and onto which worker nodes those images should be cached (i.e. pre-pulled).  cacheSpec:  # Specifies a list of images (nginx:1.23.1) with no node selector, hence these images will be cached in all the nodes in the cluster  - images:    - liruilong/my-busybox:latest  # Specifies a list of images (cassandra:v7 and etcd:3.5.4-0) with a node selector, hence these images will be cached only on the nodes selected by the node selector  - images:    - liruilong/hikvision-sdk-config-ftp:latest    nodeSelector:      kubernetes.io/hostname: vms105.liruilongs.github.io  # Specifies a list of image pull secrets to pull images from private repositories into the cache  #imagePullSecrets:  #- name: myregistrykey┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]└─$

直接创建报错了

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]└─$kubectl create -f kubefledged-imagecache.yamlError from server (InternalError): error when creating "kubefledged-imagecache.yaml": Internal error occurred: failed calling webhook "validate-image-cache.kubefledged.io": failed to call webhook: Post ";: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubefledged.io")┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]└─$kubectl get imagecaches -n kube-fledgedNo resources found in kube-fledged namespace.┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]└─$

解决办法，删除对应的对象，重新创建

我在当前项目的一个 issues 下面找到了解决办法

看起来这是因为 Webhook CA 是硬编码的，但是当 webhook 服务器启动时，会生成一个新的 CA 捆绑包并更新 webhook 配置。当发生另一个部署时，将重新应用原始 CA 捆绑包，并且 Webhook 请求开始失败，直到再次重新启动 Webhook 组件以修补捆绑包init-server

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]└─$make remove-kubefledged-and-operator# Remove kubefledgedkubectl delete -f deploy/kubefledged-operator/deploy/crds/charts.helm.kubefledged.io_v1alpha2_kubefledged_cr.yamlerror: resource mapping not found for name: "kube-fledged" namespace: "kube-fledged" from "deploy/kubefledged-operator/deploy/crds/charts.helm.kubefledged.io_v1alpha2_kubefledged_cr.yaml": no matches for kind "KubeFledged" in version "charts.helm.kubefledged.io/v1alpha2"ensure CRDs are installed first

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]└─$make deploy-using-yamlkubectl apply -f deploy/kubefledged-namespace.yamlnamespace/kube-fledged createdkubectl apply -f deploy/kubefledged-crd.yamlcustomresourcedefinition.apiextensions.k8s.io/imagecaches.kubefledged.io unchanged....................kubectl rollout status deployment kubefledged-webhook-server -n kube-fledged --watchWaiting for deployment "kubefledged-webhook-server" rollout to finish: 0 of 1 updated replicas are available...deployment "kubefledged-webhook-server" successfully rolled outkubectl get pods -n kube-fledgedNAME                                          READY   STATUS    RESTARTS   AGEkubefledged-controller-55f848cc67-76c4v       1/1     Running   0          112skubefledged-webhook-server-597dbf4ff5-56h6z   1/1     Running   0          66s

重新创建缓存对象，创建成功

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]└─$kubectl create -f kubefledged-imagecache.yamlimagecache.kubefledged.io/imagecache1 created┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]└─$kubectl get imagecaches -n kube-fledgedNAME          AGEimagecache1   10s┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]└─$

查看当前被纳管的镜像缓存

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]└─$kubectl get imagecaches imagecache1 -n kube-fledged -o json{    "apiVersion": "kubefledged.io/v1alpha2",    "kind": "ImageCache",    "metadata": {        "creationTimestamp": "2024-03-01T15:08:42Z",        "generation": 83,        "labels": {            "app": "kubefledged",            "kubefledged": "imagecache"        },        "name": "imagecache1",        "namespace": "kube-fledged",        "resourceVersion": "20169836",        "uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72"    },    "spec": {        "cacheSpec": [            {                "images": [                    "liruilong/my-busybox:latest"                ]            },            {                "images": [                    "liruilong/hikvision-sdk-config-ftp:latest"                ],                "nodeSelector": {                    "kubernetes.io/hostname": "vms105.liruilongs.github.io"                }            }        ]    },    "status": {        "completionTime": "2024-03-02T01:06:47Z",        "message": "All requested images pulled succesfully to respective nodes",        "reason": "ImageCacheRefresh",        "startTime": "2024-03-02T01:05:33Z",        "status": "Succeeded"    }}┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]└─$

通过 ansible 来验证

┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$ansible all -m shell -a "docker images | grep liruilong/my-busybox" -i host.yaml192.168.26.102 | CHANGED | rc=0 >>liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB192.168.26.101 | CHANGED | rc=0 >>liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB192.168.26.103 | CHANGED | rc=0 >>liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB192.168.26.105 | CHANGED | rc=0 >>liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB192.168.26.100 | CHANGED | rc=0 >>liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB192.168.26.106 | CHANGED | rc=0 >>liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$

┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$ansible all -m shell -a "docker images | grep liruilong/hikvision-sdk-config-ftp" -i host.yaml192.168.26.102 | FAILED | rc=1 >>non-zero return code192.168.26.100 | FAILED | rc=1 >>non-zero return code192.168.26.103 | FAILED | rc=1 >>non-zero return code192.168.26.105 | CHANGED | rc=0 >>liruilong/hikvision-sdk-config-ftp                                          latest            a02cd03b4342   4 months ago    830MB192.168.26.101 | FAILED | rc=1 >>non-zero return code192.168.26.106 | FAILED | rc=1 >>non-zero return code┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$

开启自动刷新

┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$kubectl annotate imagecaches imagecache1 -n kube-fledged kubefledged.io/refresh-imagecache=imagecache.kubefledged.io/imagecache1 annotated┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$

添加一个新的镜像缓存

┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$kubectl get imagecaches.kubefledged.io  -n kube-fledged  imagecache1 -o json{    "apiVersion": "kubefledged.io/v1alpha2",    "kind": "ImageCache",    "metadata": {        "creationTimestamp": "2024-03-01T15:08:42Z",        "generation": 92,        "labels": {            "app": "kubefledged",            "kubefledged": "imagecache"        },        "name": "imagecache1",        "namespace": "kube-fledged",        "resourceVersion": "20175233",        "uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72"    },    "spec": {        "cacheSpec": [            {                "images": [                    "liruilong/my-busybox:latest",                    "liruilong/jdk1.8_191:latest"                ]            },            {                "images": [                    "liruilong/hikvision-sdk-config-ftp:latest"                ],                "nodeSelector": {                    "kubernetes.io/hostname": "vms105.liruilongs.github.io"                }            }        ]    },    "status": {        "completionTime": "2024-03-02T01:43:32Z",        "message": "All requested images pulled succesfully to respective nodes",        "reason": "ImageCacheUpdate",        "startTime": "2024-03-02T01:40:34Z",        "status": "Succeeded"    }}┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$

通过 ansible 确认

┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$ansible all -m shell -a "docker images | grep liruilong/jdk1.8_191" -i host.yaml192.168.26.101 | FAILED | rc=1 >>non-zero return code192.168.26.100 | FAILED | rc=1 >>non-zero return code192.168.26.102 | FAILED | rc=1 >>non-zero return code192.168.26.103 | FAILED | rc=1 >>non-zero return code192.168.26.105 | FAILED | rc=1 >>non-zero return code192.168.26.106 | FAILED | rc=1 >>non-zero return code┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$ansible all -m shell -a "docker images | grep liruilong/jdk1.8_191" -i host.yaml192.168.26.101 | CHANGED | rc=0 >>liruilong/jdk1.8_191                                                        latest    17dbd4002a8c   5 years ago     170MB192.168.26.102 | CHANGED | rc=0 >>liruilong/jdk1.8_191                                                        latest    17dbd4002a8c   5 years ago     170MB192.168.26.100 | CHANGED | rc=0 >>liruilong/jdk1.8_191                                                        latest    17dbd4002a8c   5 years ago     170MB192.168.26.103 | CHANGED | rc=0 >>liruilong/jdk1.8_191                                                        latest                                      17dbd4002a8c   5 years ago     170MB192.168.26.105 | CHANGED | rc=0 >>liruilong/jdk1.8_191                                                        latest            17dbd4002a8c   5 years ago     170MB192.168.26.106 | CHANGED | rc=0 >>liruilong/jdk1.8_191                                                        latest            17dbd4002a8c   5 years ago     170MB┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$

┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$kubectl edit imagecaches imagecache1 -n kube-fledgedimagecache.kubefledged.io/imagecache1 edited┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$kubectl get imagecaches.kubefledged.io  -n kube-fledged  imagecache1 -o json{    "apiVersion": "kubefledged.io/v1alpha2",    "kind": "ImageCache",    "metadata": {        "creationTimestamp": "2024-03-01T15:08:42Z",        "generation": 94,        "labels": {            "app": "kubefledged",            "kubefledged": "imagecache"        },        "name": "imagecache1",        "namespace": "kube-fledged",        "resourceVersion": "20175766",        "uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72"    },    "spec": {        "cacheSpec": [            {                "images": [                    "liruilong/jdk1.8_191:latest"                ]            },            {                "images": [                    "liruilong/hikvision-sdk-config-ftp:latest"                ],                "nodeSelector": {                    "kubernetes.io/hostname": "vms105.liruilongs.github.io"                }            }        ]    },    "status": {        "message": "Image cache is being updated. Please view the status after some time",        "reason": "ImageCacheUpdate",        "startTime": "2024-03-02T01:48:03Z",        "status": "Processing"    }}

通过 Ansible 确认，可以看到无论是 mastere 上的节点还是 work 的节点，对应的镜像缓存都被清理

┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$ansible all -m shell -a "docker images | grep liruilong/my-busybox" -i host.yaml192.168.26.102 | CHANGED | rc=0 >>liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB192.168.26.101 | CHANGED | rc=0 >>liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB192.168.26.105 | FAILED | rc=1 >>non-zero return code192.168.26.100 | CHANGED | rc=0 >>liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB192.168.26.103 | FAILED | rc=1 >>non-zero return code192.168.26.106 | FAILED | rc=1 >>non-zero return code┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$ansible all -m shell -a "docker images | grep liruilong/my-busybox" -i host.yaml192.168.26.105 | FAILED | rc=1 >>non-zero return code192.168.26.102 | FAILED | rc=1 >>non-zero return code192.168.26.103 | FAILED | rc=1 >>non-zero return code192.168.26.101 | FAILED | rc=1 >>non-zero return code192.168.26.100 | FAILED | rc=1 >>non-zero return code192.168.26.106 | FAILED | rc=1 >>non-zero return code┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$

这里需要注意如果清除所有的镜像缓存，那么需要把 images 下的数组 写成 "".

┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$kubectl edit imagecaches imagecache1 -n kube-fledgedimagecache.kubefledged.io/imagecache1 edited┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$ansible all -m shell -a "docker images | grep liruilong/jdk1.8_191" -i host.yaml192.168.26.102 | FAILED | rc=1 >>non-zero return code192.168.26.101 | FAILED | rc=1 >>non-zero return code192.168.26.100 | FAILED | rc=1 >>non-zero return code192.168.26.105 | FAILED | rc=1 >>non-zero return code192.168.26.103 | FAILED | rc=1 >>non-zero return code192.168.26.106 | FAILED | rc=1 >>non-zero return code┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$kubectl get imagecaches.kubefledged.io  -n kube-fledged  imagecache1 -o json{    "apiVersion": "kubefledged.io/v1alpha2",    "kind": "ImageCache",    "metadata": {        "creationTimestamp": "2024-03-01T15:08:42Z",        "generation": 98,        "labels": {            "app": "kubefledged",            "kubefledged": "imagecache"        },        "name": "imagecache1",        "namespace": "kube-fledged",        "resourceVersion": "20176849",        "uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72"    },    "spec": {        "cacheSpec": [            {                "images": [                    ""                ]            },            {                "images": [                    "liruilong/hikvision-sdk-config-ftp:latest"                ],                "nodeSelector": {                    "kubernetes.io/hostname": "vms105.liruilongs.github.io"                }            }        ]    },    "status": {        "completionTime": "2024-03-02T01:52:16Z",        "message": "All cached images succesfully deleted from respective nodes",        "reason": "ImageCacheUpdate",        "startTime": "2024-03-02T01:51:47Z",        "status": "Succeeded"    }}┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$

如果通过下面的方式删除，直接注释调对应的标签

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]└─$cat kubefledged-imagecache.yaml---apiVersion: kubefledged.io/v1alpha2kind: ImageCachemetadata:  # Name of the image cache. A cluster can have multiple image cache objects  name: imagecache1  namespace: kube-fledged  # The kubernetes namespace to be used for this image cache. You can choose a different namepace as per your preference  labels:    app: kubefledged    kubefledged: imagecachespec:  # The "cacheSpec" field allows a user to define a list of images and onto which worker nodes those images should be cached (i.e. pre-pulled).  cacheSpec:  # Specifies a list of images (nginx:1.23.1) with no node selector, hence these images will be cached in all the nodes in the cluster  #- images:    #- liruilong/my-busybox:latest  # Specifies a list of images (cassandra:v7 and etcd:3.5.4-0) with a node selector, hence these images will be cached only on the nodes selected by the node selector  - images:    - liruilong/hikvision-sdk-config-ftp:latest    nodeSelector:      kubernetes.io/hostname: vms105.liruilongs.github.io  # Specifies a list of image pull secrets to pull images from private repositories into the cache  #imagePullSecrets:  #- name: myregistrykey┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]└─$

那么会报下面的错

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]└─$kubectl edit imagecaches imagecache1 -n kube-fledgederror: imagecaches.kubefledged.io "imagecache1" could not be patched: admission webhook "validate-image-cache.kubefledged.io" denied the request: Mismatch in no. of image listsYou can run `kubectl replace -f /tmp/kubectl-edit-4113815075.yaml` to try this update again.

© 文中涉及参考链接内容版权归原作者所有，如有侵权请告知,如果你认可它不要吝啬星星哦 :)

关注#华为云开发者联盟# 点击下方，第一时间了解华为云新鲜技术~

华为云博客_大数据博客_AI博客_云计算博客_开发者中心-华为云