Hike News
Hike News

使用ephemeral-storage管理容器的临时存储

容器 本文总阅读量次

ephemeral-storage介绍

Kubernetes在1.8的版本中引入了一种类似于CPU，内存的新的资源模式：ephemeral-storage，并且在1.10的版本在kubelet中默认打开了这个特性。

  • Alpha release target (x.y): 1.7/1.8
  • Beta release target (x.y): 1.10
  • Stable release target (x.y): 1.11

ephemeral-storage是为了管理和调度Kubernetes中运行的应用的短暂存储。

在每个Kubernetes的节点上，kubelet的根目录(默认是/var/lib/kubelet)和日志目录(/var/log)保存在节点的主分区上，这个分区同时也会被Pod的EmptyDir类型的volume、容器日志、镜像的层、容器的可写层所占用。ephemeral-storage便是对这块主分区进行管理，通过应用定义的需求(requests)和约束(limits)来调度和管理节点上的应用对主分区的消耗。

ephemeral-storage的eviction逻辑

在节点上的kubelet启动的时候，kubelet会统计当前节点的主分区的可分配的磁盘资源，或者你可以覆盖节点上kubelet的配置来自定义可分配的资源。在创建Pod时会根据存储需求调度到满足存储的节点，在Pod使用超过限制的存储时会对其做驱逐的处理来保证不会耗尽节点上的磁盘空间。

如果运行时指定了别的独立的分区，比如修改了docker的镜像层和容器可写层的存储位置(默认是/var/lib/docker)所在的分区，将不再将其计入ephemeral-storage的消耗。

  • EmptyDir 的使用量超过了他的 SizeLimit，那么这个 pod 将会被驱逐
  • Container 的使用量（log，如果没有 overlay 分区，则包括 imagefs）超过了他的 limit，则这个 pod 会被驱逐
  • Pod 对本地临时存储总的使用量（所有 emptydir 和 container）超过了 pod 中所有container 的 limit 之和，则 pod 被驱逐

ephemeral-storage使用

和内存和CPU的限制类似，存储的限制也是定义在Pod的container中

spec.containers[].resources.limits.ephemeral-storage

spec.containers[].resources.requests.ephemeral-storage

示例：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
apiVersion: v1
kind: Pod
metadata:
  name: teststorage
  labels:
    app: teststorage
spec:
  containers:
  - name: teststorage
    image: nginx:1.14
    command: ["bash", "-c", "while true; do dd if=/dev/zero of=$(date '+%s').out count=1 bs=10MB; sleep 1; done"] # 持续写入文件到容器的rootfs中
    resources:
      limits:
        ephemeral-storage: 100Mi #定义存储的限制为100M
      requests:
        ephemeral-storage: 100Mi
1
2
3
4
5
[root@master1 ~]# kubectl get pods -o wide
NAME                                READY     STATUS    RESTARTS   AGE       IP               NODE            NOMINATED NODE
teststorage                         1/1       Running   0          7s        172.20.189.69    10.208.204.35   <none>
-------------------------------------------------------------------------------------------------
teststorage                         0/1       Evicted   0          1m        <none>           10.208.204.35   <none>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
[root@master1 ~]# kubectl describe pod teststorage 
Name:         teststorage
Namespace:    default
Node:         10.208.204.35/
Start Time:   Wed, 28 Nov 2018 13:48:37 +0800
Labels:       app=teststorage
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"app":"teststorage"},"name":"teststorage","namespace":"default"},"spec":{"contai...
Status:       Failed
Reason:       Evicted
Message:      Pod ephemeral local storage usage exceeds the total limit of containers 100Mi. 
IP:           
Containers:
  teststorage:
    Image:      nginx:1.14
    Port:       <none>
    Host Port:  <none>
    Command:
      bash
      -c
      while true; do dd if=/dev/zero of=$(date '+%s').out count=1 bs=10MB; sleep 1; done
    Limits:
      ephemeral-storage:  100Mi
    Requests:
      ephemeral-storage:  100Mi
    Environment:          <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-mqzrh (ro)
Volumes:
  default-token-mqzrh:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-mqzrh
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     <none>
Events:
  Type     Reason     Age   From                    Message
  ----     ------     ----  ----                    -------
  Normal   Scheduled  1m    default-scheduler       Successfully assigned default/teststorage to 10.208.204.35
  Normal   Pulled     1m    kubelet, 10.208.204.35  Container image "nginx:1.14" already present on machine
  Normal   Created    1m    kubelet, 10.208.204.35  Created container
  Normal   Started    1m    kubelet, 10.208.204.35  Started container
  Warning  Evicted    8s    kubelet, 10.208.204.35  Pod ephemeral local storage usage exceeds the total limit of containers 100Mi.
  Normal   Killing    8s    kubelet, 10.208.204.35  Killing container with id docker://teststorage:Need to kill Pod

参考文档：

https://v1-11.docs.kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/

https://github.com/kubernetes/enhancements/issues/361

https://yq.aliyun.com/articles/594066

http://www.k8smeetup.com/article/VyEncpgA7

容器