32

How can I tell with kubectl how much ephemeral storage a pod is currently using?

In a Kubernetes pod spec, I can specify resource requests and limits for CPU, memory, and ephemeral storage:

resources:
  requests:
    memory: "60Mi"
    cpu: "70m"
    ephemeral-storage: "2Gi"
  limits:
    memory: "65Mi"
    cpu: "75m"
    ephemeral-storage: "4Gi"

However, to set good requests and limits on ephemeral storage, I need to know what this value actually is for a running pod, which I can't figure out. I can get CPU and memory usage using kubectl top pod, but, from what I can tell, ephemeral storage usage is only actually calculated when making an actual eviction decision.

bskaggs
  • 1,374
  • 2
  • 12
  • 24

3 Answers3

15

You can do this through the raw command.

kubectl get --raw "/api/v1/nodes/(your-node-name)/proxy/stats/summary"

There is also this

kubectl get --raw "/api/v1/nodes/(your-node-name)/proxy/metrics/cadvisor""

EDIT:

I've created a Prometheus exporter for this now.

https://github.com/jmcgrath207/k8s-ephemeral-storage-metrics

jmcgrath207
  • 1,317
  • 2
  • 19
  • 31
  • This should be the accepted answer. It's common to disable anonymous authentication to the kubelet API, so using the node proxy is a good way around that. Note that access to "nodes/proxy" should ONLY be granted to admins since it can be used to escalate privileges. – Mac Chaffee Aug 31 '22 at 15:12
  • 2
    To select a single pod by name, you can pipe the output of that command to `jq '.pods[] | select(.podRef.name == "mypodname")'` – Mac Chaffee Aug 31 '22 at 15:16
  • Does not work for k8s v1.26.5 and/or microk8s though... – mirekphd Jul 07 '23 at 09:00
9

The pure raw approach for this is to use the disk usage (du) Unix command line.

Shell into your pod:

$ kubectl exec -it <pod-id> sh

Change dirs to the mount point of your ephemeral-storage (if you are using volume mounts):

$ mount # check mount points if you'd like
$ cd /mnt/of/ephemeral
$ du .

If you are not using volume mounts:

$ du .

There are other tools that you can use to get metrics:

  • cAdvisor also embedded into the kubelet code, exposed under the /stats/summary or /metrics endpoint. More info here. An example output:

    $ curl -k -H 'Authorization: Bearer <Redacted>' \
https://node-hostname:10250/stats/summary

{
 "node": {
   "nodeName": "node-hostname",
   "systemContainers": [
    {
     "name": "kubelet",
    ...
    "volume": [
     {
      "time": "2018-11-08T23:52:03Z",
      "availableBytes": 1969168384,
      "capacityBytes": 1969180672,
      "usedBytes": 12288,
      "inodesFree": 480748,
      "inodes": 480757,
      "inodesUsed": 9,
      "name": "kube-proxy-token-pprwb"
     }
    ],
    "ephemeral-storage": {
     "time": "2018-11-09T00:05:10Z",
     "availableBytes": 31057477632,
     "capacityBytes": 41567858688,
     "inodesFree": 4873887,
     "inodes": 5120000
    }
...
}

    Similarly:

    $ curl -k -H 'Authorization: Bearer <Redacted>' \
https://node-hostname:10250/stats/summary

# HELP apiserver_audit_event_total Counter of audit events generated and sent to the audit backend.
# TYPE apiserver_audit_event_total counter
apiserver_audit_event_total 0
# HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request.
# TYPE apiserver_client_certificate_expiration_seconds histogram
apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="21600"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="43200"} 0
...

    More info on kubelet authentication/authorization.

  • Prometheus

More info on K8s metrics here.

Rico
  • 58,485
  • 12
  • 111
  • 141
  • "Change dirs to the mount point of your ephemeral-storage" – which of them one should look at? `overlay`? – kivagant Apr 25 '19 at 12:02
  • It depends on the mount point that you define in your pod spec – Rico Apr 25 '19 at 15:08
  • 7
    But I don't have any mount points. Ephemeral storage includes storage used by all containers logs etc. So the answer is not fully correct in that part with "mount" command. – kivagant Apr 26 '19 at 09:14
  • 3
    ephemeral-storage also includes container logs and container images. – Sam Jan 06 '21 at 13:39
0

Assuming you know the node name, just run:

$ kubectl describe $NODE_NAME

And at the very end the output of that command will include section about Allocated resources, including for ephemeral-storage:

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests        Limits
  --------           --------        ------
  cpu                510m (12%)      300m (7%)
  memory             334164096 (2%)  383500800 (2%)
  ephemeral-storage  0 (0%)          0 (0%)
  hugepages-1Gi      0 (0%)          0 (0%)
  hugepages-2Mi      0 (0%)          0 (0%)

Note: this is collective information for the entire node, not individual pods though.

Note: node name can be obtained by e.g.:

NODE_NAME=$(kubectl get nodes -o name | head -1)

mirekphd
  • 4,799
  • 3
  • 38
  • 59