HPA
HPA는 Deployment, StatefulSet과 같은 workload resource를 자동으로 업데이트하며 workload의 크기를 자동으로 수요에 맞게 하는 것을 목표로 한다.
a HorizontalPodAutoscaler automatically updates a workload resource (such as a Deployment or StatefulSet), with the aim of automatically scaling the workload to match demand.
HPA를 사용하기 위해서는 Metric Server를 별도로 배포하여야 한다. Metric Server Install, Metric Server Configuration
- Metrics server
- Cluster addon component that collects and aggregates resource metrics pulled from each kubelet. The API server serves Metrics API for use by HPA, VPA, and by the kubectl top command. Metrics Server is a reference implementation of the Metrics API. Metrics Server collects resource metrics from Kubelets and exposes them in Kubernetes apiserver through Metrics API for use by Horizontal Pod Autoscaler and Vertical Pod Autoscaler.
- 각 Kubelet에서 메트릭을 수집하고 집계하는 Add-on 컴포넌트다. HPA, VPA, kubectl top 명령을 위해서 Metrics API를 제공한다.
- Metric Server는 Add-ons로 제공된다.
- HPA를 사용하는 일반적인 방법은 aggregated API(metrics.k8s.io, custom.metrics.k8s.io, 또는 external.metrics.k8s.io)로 부터 메트릭을 가져오도록 설정하는 것이다. (aggregated API는 Kubernetes를 확장하기 위해 클러스터에 추가적인 API를 배포할 수 있도록 한 것이다.)
- Metric Server를 배포하면 아래와 같이 ‘metrics.k8s.io’ API를 제공하도록 설정된다.
apiVersion: apiregistration.k8s.io/v1 kind: APIService metadata: labels: k8s-app: metrics-server name: v1beta1.metrics.k8s.io spec: group: metrics.k8s.io groupPriorityMinimum: 100 insecureSkipTLSVerify: true service: name: metrics-server namespace: kube-system version: v1beta1 versionPriority: 100 - Metrics API: Kubernetes API supporting access to CPU and memory used for workload autoscaling. To make this work in your cluster, you need an API extension server that provides the Metrics API.
- Metrics API는 HPA, VPA또는 사용자가 kubectl을 통해 metric정보를 사용할 수 있도록 CPU나 메모리에 접근할 수 있게 하는 Kubernetes API다. 즉, API 서버는 HPA, VPA 및 kubectl top 명령에서 사용할 Metrics API를 제공한다. 그리고 Metrics Server는 Metrics API의 참조 구현입니다.
- Cluster addon component that collects and aggregates resource metrics pulled from each kubelet. The API server serves Metrics API for use by HPA, VPA, and by the kubectl top command. Metrics Server is a reference implementation of the Metrics API. Metrics Server collects resource metrics from Kubelets and exposes them in Kubernetes apiserver through Metrics API for use by Horizontal Pod Autoscaler and Vertical Pod Autoscaler.
1. 동작방법
cAdvisor가 Pod내 Container들의 Metrics을 수집함.
kubelet이 cAdvisor가 노출한 메트릭을 수집함.
metric-server가 kubelet으로 부터 metric 리소스를 수집함. 이 주기는 –metric-resolution=15s 옵션에 따라 수집 됨
$ kubectl logs -f metrics-server-58fd485c7-nnltl -n kube-system # Kubelet으로부터 메트릭 정보를 수집하고 있다. I1222 15:01:24.847741 1 scraper.go:115] "Scraping metrics from nodes" nodeCount=2 I1222 15:01:24.855038 1 scraper.go:137] "Scraping node" node="ip-10-20-10-235.ap-northeast-2.compute.internal" I1222 15:01:24.857187 1 scraper.go:137] "Scraping node" node="ip-10-20-11-91.ap-northeast-2.compute.internal" I1222 15:01:24.877046 1 scraper.go:172] "Scrape finished" duration="29.273444ms" nodeCount=2 podCount=17API 서버는 Metrics API를 제공하므로써 사용자, HPA등에게 API를 제공할 수 있다.
따라서, 아래 kubectl 커맨드를 통해 메트릭을 조회하면 실제로는 메트릭 서버에서 아래와 같이 로그가 보인다. (configured by - –v=5)$ kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes/ip-10-20-10-235.ap-northeast-2.compute.internal" | jq '.' $ kubectl logs -f metrics-server-58fd485c7-nnltl -n kube-system # srcIP=x-ENI의 주소 중 하나다. 내 로컬 -> API server -> metrics-server로 왔기 때문에. I1222 15:01:10.534483 1 handler.go:143] metrics-server: GET "/apis/metrics.k8s.io/v1beta1/nodes/ip-10-20-10-235.ap-northeast-2.compute.internal" satisfied by gorestful with webservice /apis/metrics.k8s.io/v1beta1 I1222 15:01:10.534721 1 httplog.go:129] "HTTP" verb="GET" URI="/apis/metrics.k8s.io/v1beta1/nodes/ip-10-20-10-235.ap-northeast-2.compute.internal" latency="432.784µs" userAgent="kubectl/v1.26.0 (linux/amd64) kubernetes/b46a3f8" audit-ID="ae15fa19-9c33-4ebf-8f7d-15931ba355e0" srcIP="10.20.10.23:40266" resp=200아래는 HPA를 배포한 리소스에 대한 Metrics Server로그이다.
# 아래도 주기적으로 관찰되는 로그이며 이는 php-apche hpa에의해 주기적으로 15초마다 조회되는 결과다. I1222 15:01:37.041375 1 handler.go:143] metrics-server: GET "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods" satisfied by gorestful with webservice /apis/metrics.k8s.io/v1beta1 I1222 15:01:37.041560 1 httplog.go:129] "HTTP" verb="LIST" URI="/apis/metrics.k8s.io/v1beta1/namespaces/default/pods?labelSelector=run%3Dphp-apache" latency="4.32772ms" userAgent="kube-controller-manager/v1.21.14 (linux/amd64) kubernetes/b07006b/system:serviceaccount:kube-system:horizontal-pod-autoscaler" audit-ID="de1229ce-5f8f-4636-8698-0ff3003aca05" srcIP="10.20.11.208:32858" resp=200 I1222 15:01:52.104167 1 handler.go:143] metrics-server: GET "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods" satisfied by gorestful with webservice /apis/metrics.k8s.io/v1beta1 I1222 15:01:52.104352 1 httplog.go:129] "HTTP" verb="LIST" URI="/apis/metrics.k8s.io/v1beta1/namespaces/default/pods?labelSelector=run%3Dphp-apache" latency="3.059605ms" userAgent="kube-controller-manager/v1.21.14 (linux/amd64) kubernetes/b07006b/system:serviceaccount:kube-system:horizontal-pod-autoscaler" audit-ID="b57ab53f-0194-41e2-99b1-50c7dc429e03" srcIP="10.20.11.208:32858" resp=200 I1222 15:02:07.127963 1 handler.go:143] metrics-server: GET "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods" satisfied by gorestful with webservice /apis/metrics.k8s.io/v1beta1 I1222 15:02:07.128194 1 httplog.go:129] "HTTP" verb="LIST" URI="/apis/metrics.k8s.io/v1beta1/namespaces/default/pods?labelSelector=run%3Dphp-apache" latency="4.949269ms" userAgent="kube-controller-manager/v1.21.14 (linux/amd64) kubernetes/b07006b/system:serviceaccount:kube-system:horizontal-pod-autoscaler" audit-ID="5f545f72-bb1b-4cce-a7e6-a5681235f1bc" srcIP="10.20.11.208:32858" resp=200 I1222 15:02:22.205786 1 handler.go:143] metrics-server: GET "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods" satisfied by gorestful with webservice /apis/metrics.k8s.io/v1beta1 I1222 15:02:22.206026 1 httplog.go:129] "HTTP" verb="LIST" URI="/apis/metrics.k8s.io/v1beta1/namespaces/default/pods?labelSelector=run%3Dphp-apache" latency="3.820725ms" userAgent="kube-controller-manager/v1.21.14 (linux/amd64) kubernetes/b07006b/system:serviceaccount:kube-system:horizontal-pod-autoscaler" audit-ID="b591499b-e8f3-4313-9f7a-ab2395eaac70" srcIP="10.20.11.208:32858" resp=200kube-controller(HPA controller)가 metrics API의 데이터를 보고 RS replica를 조정함 (15초)
2. HPA 테스트
아래와 같이 php-apache에 대한 Deployment, Service, HPA 리소스를 배포한다.
apiVersion: apps/v1 kind: Deployment metadata: name: php-apache1 spec: selector: matchLabels: run: php-apache1 replicas: 2 template: metadata: labels: run: php-apache1 spec: containers: - name: php-apache1 image: registry.k8s.io/hpa-example ports: - containerPort: 80 resources: limits: cpu: 500m requests: cpu: 200m --- apiVersion: v1 kind: Service metadata: name: php-apache1 labels: run: php-apache1 spec: ports: - port: 80 selector: run: php-apache1 --- apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: name: php-apache1 namespace: default spec: maxReplicas: 10 minReplicas: 2 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: php-apache1 targetCPUUtilizationPercentage: 50아래와 같이 curl을 요청할 수 있는 Deployment를 배포한다.
apiVersion: apps/v1 kind: Deployment metadata: name: netutil labels: app: netutil spec: replicas: 1 selector: matchLabels: app: netutil template: metadata: labels: app: netutil spec: containers: - name: netutil image: praqma/network-multitool
HPA 로직
HPA는 desired수를 아래와 같이 계산한다.desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )]
만약 HPA에 CPU기반 scale 설정이 되었으나 파드 Manifest에 request.cpu가 정의되지 않은 경우에는 HPA가 동작하지 않는다.
부하 & 결과 확인
Test1
부하 발생
$ kubectl exec -it netutil-669f67cb94-2hdns -- /bin/bash
$ while true; do curl http://php-apache1; sleep 0.3; done
결과 확인
$ while true; do date; k get hpa php-apache1; sleep 10; done
#################### Result ####################
Thu Jan 5 09:57:28 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 0%/50% 2 10 2 39h
Thu Jan 5 09:57:38 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 26%/50% 2 10 2 39h
Thu Jan 5 09:57:48 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 26%/50% 2 10 2 39h
Thu Jan 5 09:57:58 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 52%/50% 2 10 2 39h
Thu Jan 5 09:58:08 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 51%/50% 2 10 2 39h
Thu Jan 5 09:58:18 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 51%/50% 2 10 2 39h
Thu Jan 5 09:58:29 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 52%/50% 2 10 2 39h
Thu Jan 5 09:58:39 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 52%/50% 2 10 2 39h
Thu Jan 5 09:58:49 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 52%/50% 2 10 2 39h
Thu Jan 5 09:58:59 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 50%/50% 2 10 2 39h
Thu Jan 5 09:59:09 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 51%/50% 2 10 2 39h
Thu Jan 5 09:59:19 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 51%/50% 2 10 2 39h
Thu Jan 5 09:59:29 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 52%/50% 2 10 2 39h
Thu Jan 5 09:59:40 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 56%/50% 2 10 2 39h
Thu Jan 5 09:59:50 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 56%/50% 2 10 2 39h
Thu Jan 5 10:00:00 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 49%/50% 2 10 3 39h
Thu Jan 5 10:00:10 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 33%/50% 2 10 3 39h
Thu Jan 5 10:00:20 KST 2023
$ while true; do date; k get po -l run=php-apache1; sleep 5; done
#################### Result ####################
Thu Jan 5 09:59:33 KST 2023
NAME READY STATUS RESTARTS AGE
php-apache1-8445df799-bdlln 1/1 Running 0 39h
php-apache1-8445df799-jnmtx 1/1 Running 0 39h
Thu Jan 5 09:59:36 KST 2023
NAME READY STATUS RESTARTS AGE
php-apache1-8445df799-67jjv 0/1 ContainerCreating 0 0s
php-apache1-8445df799-bdlln 1/1 Running 0 39h
php-apache1-8445df799-jnmtx 1/1 Running 0 39h
Thu Jan 5 09:59:39 KST 2023
NAME READY STATUS RESTARTS AGE
php-apache1-8445df799-67jjv 1/1 Running 0 3s
php-apache1-8445df799-bdlln 1/1 Running 0 39h
php-apache1-8445df799-jnmtx 1/1 Running 0 39h
51%, 52% 구간
결과를 확인해보면 중간 51%, 52%구간에서는 Target 임계치인 50%를 넘었지만 파드가 Autoscaling되지 않았다. ceil((52/50) * 2) = ceil(2.08) = 3 이 나오는 것임에도 불구하고 증가하지 않은 이유는 무엇일까?
공식문서에서는 다음과 같이 이야기하고있다. 비율이 1에 충분히 가깝다면 스케일링 건너뛴다. 허용오차는 0.1의 default를 가지고 있다. (
--horizontal-pod-autoscaler-tolerance)The control plane skips any scaling action if the ratio is sufficiently close to 1.0 (within a globally-configurable tolerance, 0.1 by default).
따라서, 2.08은 0.1만큼의 허용오차에서 무시가 된다.
56% 구간
파드는 정확하게 2.1을 넘긴 56% 구간이 되자마자 파드가 하나 증가하였다.
Test2
부하 발생
$ kubectl exec -it netutil-669f67cb94-2hdns -- /bin/bash
$ while true; do curl http://php-apache1; sleep 0.1; done
결과 확인
$ while true; do date; k get hpa php-apache1; sleep 10; done
#################### Result ####################
Thu Jan 5 10:28:49 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 0%/50% 2 10 2 40h
Thu Jan 5 10:28:59 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 0%/50% 2 10 2 40h
Thu Jan 5 10:29:09 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 16%/50% 2 10 2 40h
Thu Jan 5 10:29:19 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 16%/50% 2 10 2 40h
Thu Jan 5 10:29:30 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 91%/50% 2 10 2 40h
Thu Jan 5 10:29:40 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 93%/50% 2 10 4 40h
Thu Jan 5 10:29:50 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 93%/50% 2 10 4 40h
Thu Jan 5 10:30:00 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 45%/50% 2 10 4 40h
Thu Jan 5 10:30:10 KST 2023
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache1 Deployment/php-apache1 48%/50% 2 10 4 40h
$ while true; do date; k get po -l run=php-apache1; sleep 5; done
#################### Result ####################
NAME READY STATUS RESTARTS AGE
php-apache1-8445df799-67jjv 1/1 Running 0 29m
php-apache1-8445df799-bdlln 1/1 Running 0 39h
Thu Jan 5 10:29:23 KST 2023
NAME READY STATUS RESTARTS AGE
php-apache1-8445df799-67jjv 1/1 Running 0 29m
php-apache1-8445df799-bdlln 1/1 Running 0 39h
php-apache1-8445df799-ffff6 0/1 ContainerCreating 0 1s
php-apache1-8445df799-r5z8p 0/1 ContainerCreating 0 1s
Thu Jan 5 10:29:26 KST 2023
NAME READY STATUS RESTARTS AGE
php-apache1-8445df799-67jjv 1/1 Running 0 29m
php-apache1-8445df799-bdlln 1/1 Running 0 39h
php-apache1-8445df799-ffff6 1/1 Running 0 4s
php-apache1-8445df799-r5z8p 1/1 Running 0 4s
- 91~93% 구간
결과를 확인해보면 91~93% 구간이 확인되자 바로 Pod는 4개로 증가하였다.
ceil(2 * (93/50)) = ceil(2* 1.86) = ceil(3.72) = 4
- 48% 구간
파드가 4개로 증가하고 48%를 유지한다. 이 때는 아래와 같이 desiredReplica수가 currentReplica 수와 같기 때문에 유지한다.
ceil(4 * (48/50)) = ceil(4* 0.96) = ceil(3.84) = 4
📚 References
[1] kubernetes-sigs/metrics-server
[2] High Availability
[3] metric-server 가용성
[4] Resource metrics pipeline