EKS metric-servers를 통한 파드 Autoscaling

By Bys on January 2, 2023

HPA

HPA는 Deployment, StatefulSet과 같은 workload resource를 자동으로 업데이트하며 workload의 크기를 자동으로 수요에 맞게 하는 것을 목표로 한다.

a HorizontalPodAutoscaler automatically updates a workload resource (such as a Deployment or StatefulSet), with the aim of automatically scaling the workload to match demand.

HPA를 사용하기 위해서는 Metric Server를 별도로 배포하여야 한다. Metric Server Install, Metric Server Configuration

  • Metrics server
    • Cluster addon component that collects and aggregates resource metrics pulled from each kubelet. The API server serves Metrics API for use by HPA, VPA, and by the kubectl top command. Metrics Server is a reference implementation of the Metrics API. Metrics Server collects resource metrics from Kubelets and exposes them in Kubernetes apiserver through Metrics API for use by Horizontal Pod Autoscaler and Vertical Pod Autoscaler.
      • 각 Kubelet에서 메트릭을 수집하고 집계하는 Add-on 컴포넌트다. HPA, VPA, kubectl top 명령을 위해서 Metrics API를 제공한다.
    • Metric Server는 Add-ons로 제공된다.
    • HPA를 사용하는 일반적인 방법은 aggregated API(metrics.k8s.io, custom.metrics.k8s.io, 또는 external.metrics.k8s.io)로 부터 메트릭을 가져오도록 설정하는 것이다. (aggregated API는 Kubernetes를 확장하기 위해 클러스터에 추가적인 API를 배포할 수 있도록 한 것이다.)
    • Metric Server를 배포하면 아래와 같이 ‘metrics.k8s.io’ API를 제공하도록 설정된다. 
        apiVersion: apiregistration.k8s.io/v1
  kind: APIService
  metadata:
    labels:
      k8s-app: metrics-server
    name: v1beta1.metrics.k8s.io
  spec:
    group: metrics.k8s.io
    groupPriorityMinimum: 100
    insecureSkipTLSVerify: true
    service:
      name: metrics-server
      namespace: kube-system
    version: v1beta1
    versionPriority: 100
    • Metrics API: Kubernetes API supporting access to CPU and memory used for workload autoscaling. To make this work in your cluster, you need an API extension server that provides the Metrics API.
      • Metrics API는 HPA, VPA또는 사용자가 kubectl을 통해 metric정보를 사용할 수 있도록 CPU나 메모리에 접근할 수 있게 하는 Kubernetes API다. 즉, API 서버는 HPA, VPA 및 kubectl top 명령에서 사용할 Metrics API를 제공한다. 그리고 Metrics Server는 Metrics API의 참조 구현입니다.


1. 동작방법

metric-server-architecture

hpa001

  1. cAdvisor가 Pod내 Container들의 Metrics을 수집함.
  2. kubelet이 cAdvisor가 노출한 메트릭을 수집함.
  3. metric-server가 kubelet으로 부터 metric 리소스를 수집함. 이 주기는 –metric-resolution=15s 옵션에 따라 수집 됨 
     $ kubectl logs -f metrics-server-58fd485c7-nnltl -n kube-system
 # Kubelet으로부터 메트릭 정보를 수집하고 있다. 
 I1222 15:01:24.847741       1 scraper.go:115] "Scraping metrics from nodes" nodeCount=2
 I1222 15:01:24.855038       1 scraper.go:137] "Scraping node" node="ip-10-20-10-235.ap-northeast-2.compute.internal"
 I1222 15:01:24.857187       1 scraper.go:137] "Scraping node" node="ip-10-20-11-91.ap-northeast-2.compute.internal"
 I1222 15:01:24.877046       1 scraper.go:172] "Scrape finished" duration="29.273444ms" nodeCount=2 podCount=17
  4. API 서버는 Metrics API를 제공하므로써 사용자, HPA등에게 API를 제공할 수 있다.
    따라서, 아래 kubectl 커맨드를 통해 메트릭을 조회하면 실제로는 메트릭 서버에서 아래와 같이 로그가 보인다. (configured by - –v=5) 
     $ kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes/ip-10-20-10-235.ap-northeast-2.compute.internal" | jq '.'

 $ kubectl logs -f metrics-server-58fd485c7-nnltl -n kube-system
 # srcIP=x-ENI의 주소 중 하나다. 내 로컬 -> API server -> metrics-server로 왔기 때문에. 
 I1222 15:01:10.534483       1 handler.go:143] metrics-server: GET "/apis/metrics.k8s.io/v1beta1/nodes/ip-10-20-10-235.ap-northeast-2.compute.internal" satisfied by gorestful with webservice /apis/metrics.k8s.io/v1beta1
 I1222 15:01:10.534721       1 httplog.go:129] "HTTP" verb="GET" URI="/apis/metrics.k8s.io/v1beta1/nodes/ip-10-20-10-235.ap-northeast-2.compute.internal" latency="432.784µs" userAgent="kubectl/v1.26.0 (linux/amd64) kubernetes/b46a3f8" audit-ID="ae15fa19-9c33-4ebf-8f7d-15931ba355e0" srcIP="10.20.10.23:40266" resp=200

    아래는 HPA를 배포한 리소스에 대한 Metrics Server로그이다.

       # 아래도 주기적으로 관찰되는 로그이며 이는 php-apche hpa에의해 주기적으로 15초마다 조회되는 결과다.  
   I1222 15:01:37.041375       1 handler.go:143] metrics-server: GET "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods" satisfied by gorestful with webservice /apis/metrics.k8s.io/v1beta1
   I1222 15:01:37.041560       1 httplog.go:129] "HTTP" verb="LIST" URI="/apis/metrics.k8s.io/v1beta1/namespaces/default/pods?labelSelector=run%3Dphp-apache" latency="4.32772ms" userAgent="kube-controller-manager/v1.21.14 (linux/amd64) kubernetes/b07006b/system:serviceaccount:kube-system:horizontal-pod-autoscaler" audit-ID="de1229ce-5f8f-4636-8698-0ff3003aca05" srcIP="10.20.11.208:32858" resp=200

   I1222 15:01:52.104167       1 handler.go:143] metrics-server: GET "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods" satisfied by gorestful with webservice /apis/metrics.k8s.io/v1beta1
   I1222 15:01:52.104352       1 httplog.go:129] "HTTP" verb="LIST" URI="/apis/metrics.k8s.io/v1beta1/namespaces/default/pods?labelSelector=run%3Dphp-apache" latency="3.059605ms" userAgent="kube-controller-manager/v1.21.14 (linux/amd64) kubernetes/b07006b/system:serviceaccount:kube-system:horizontal-pod-autoscaler" audit-ID="b57ab53f-0194-41e2-99b1-50c7dc429e03" srcIP="10.20.11.208:32858" resp=200

   I1222 15:02:07.127963       1 handler.go:143] metrics-server: GET "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods" satisfied by gorestful with webservice /apis/metrics.k8s.io/v1beta1
   I1222 15:02:07.128194       1 httplog.go:129] "HTTP" verb="LIST" URI="/apis/metrics.k8s.io/v1beta1/namespaces/default/pods?labelSelector=run%3Dphp-apache" latency="4.949269ms" userAgent="kube-controller-manager/v1.21.14 (linux/amd64) kubernetes/b07006b/system:serviceaccount:kube-system:horizontal-pod-autoscaler" audit-ID="5f545f72-bb1b-4cce-a7e6-a5681235f1bc" srcIP="10.20.11.208:32858" resp=200

   I1222 15:02:22.205786       1 handler.go:143] metrics-server: GET "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods" satisfied by gorestful with webservice /apis/metrics.k8s.io/v1beta1
   I1222 15:02:22.206026       1 httplog.go:129] "HTTP" verb="LIST" URI="/apis/metrics.k8s.io/v1beta1/namespaces/default/pods?labelSelector=run%3Dphp-apache" latency="3.820725ms" userAgent="kube-controller-manager/v1.21.14 (linux/amd64) kubernetes/b07006b/system:serviceaccount:kube-system:horizontal-pod-autoscaler" audit-ID="b591499b-e8f3-4313-9f7a-ab2395eaac70" srcIP="10.20.11.208:32858" resp=200
  5. kube-controller(HPA controller)가 metrics API의 데이터를 보고 RS replica를 조정함 (15초)

2. HPA 테스트

  1. 아래와 같이 php-apache에 대한 Deployment, Service, HPA 리소스를 배포한다.

     apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: php-apache1
 spec:
   selector:
     matchLabels:
       run: php-apache1
   replicas: 2
   template:
     metadata:
       labels:
         run: php-apache1
     spec:
       containers:
       - name: php-apache1
         image: registry.k8s.io/hpa-example
         ports:
         - containerPort: 80
         resources:
           limits:
             cpu: 500m
           requests:
             cpu: 200m
 ---
 apiVersion: v1
 kind: Service
 metadata:
   name: php-apache1
   labels:
     run: php-apache1
 spec:
   ports:
   - port: 80
   selector:
     run: php-apache1
 ---
 apiVersion: autoscaling/v1
 kind: HorizontalPodAutoscaler
 metadata:
   name: php-apache1
   namespace: default
 spec:
   maxReplicas: 10
   minReplicas: 2
   scaleTargetRef:
     apiVersion: apps/v1
     kind: Deployment
     name: php-apache1
   targetCPUUtilizationPercentage: 50

  2. 아래와 같이 curl을 요청할 수 있는 Deployment를 배포한다.

     apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: netutil
   labels:
     app: netutil
 spec:
   replicas: 1
   selector:
     matchLabels:
       app: netutil
   template:
     metadata:
       labels:
         app: netutil
     spec:
       containers:
       - name: netutil
         image: praqma/network-multitool

HPA 로직

HPA는 desired수를 아래와 같이 계산한다.
desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )]

만약 HPA에 CPU기반 scale 설정이 되었으나 파드 Manifest에 request.cpu가 정의되지 않은 경우에는 HPA가 동작하지 않는다.

부하 & 결과 확인

Test1

부하 발생

$ kubectl exec -it netutil-669f67cb94-2hdns -- /bin/bash
$ while true; do curl http://php-apache1; sleep 0.3; done

결과 확인

$ while true; do date; k get hpa php-apache1; sleep 10; done
#################### Result ####################
Thu Jan  5 09:57:28 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   0%/50%    2         10        2          39h
Thu Jan  5 09:57:38 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   26%/50%   2         10        2          39h
Thu Jan  5 09:57:48 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   26%/50%   2         10        2          39h
Thu Jan  5 09:57:58 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   52%/50%   2         10        2          39h
Thu Jan  5 09:58:08 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   51%/50%   2         10        2          39h
Thu Jan  5 09:58:18 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   51%/50%   2         10        2          39h
Thu Jan  5 09:58:29 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   52%/50%   2         10        2          39h
Thu Jan  5 09:58:39 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   52%/50%   2         10        2          39h
Thu Jan  5 09:58:49 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   52%/50%   2         10        2          39h
Thu Jan  5 09:58:59 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   50%/50%   2         10        2          39h
Thu Jan  5 09:59:09 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   51%/50%   2         10        2          39h
Thu Jan  5 09:59:19 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   51%/50%   2         10        2          39h
Thu Jan  5 09:59:29 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   52%/50%   2         10        2          39h
Thu Jan  5 09:59:40 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   56%/50%   2         10        2          39h
Thu Jan  5 09:59:50 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   56%/50%   2         10        2          39h
Thu Jan  5 10:00:00 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   49%/50%   2         10        3          39h
Thu Jan  5 10:00:10 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   33%/50%   2         10        3          39h
Thu Jan  5 10:00:20 KST 2023

$ while true; do date; k get po -l run=php-apache1; sleep 5; done
#################### Result ####################
Thu Jan  5 09:59:33 KST 2023
NAME                          READY   STATUS    RESTARTS   AGE
php-apache1-8445df799-bdlln   1/1     Running   0          39h
php-apache1-8445df799-jnmtx   1/1     Running   0          39h
Thu Jan  5 09:59:36 KST 2023
NAME                          READY   STATUS              RESTARTS   AGE
php-apache1-8445df799-67jjv   0/1     ContainerCreating   0          0s
php-apache1-8445df799-bdlln   1/1     Running             0          39h
php-apache1-8445df799-jnmtx   1/1     Running             0          39h
Thu Jan  5 09:59:39 KST 2023
NAME                          READY   STATUS    RESTARTS   AGE
php-apache1-8445df799-67jjv   1/1     Running   0          3s
php-apache1-8445df799-bdlln   1/1     Running   0          39h
php-apache1-8445df799-jnmtx   1/1     Running   0          39h
  • 51%, 52% 구간

    결과를 확인해보면 중간 51%, 52%구간에서는 Target 임계치인 50%를 넘었지만 파드가 Autoscaling되지 않았다. ceil((52/50) * 2) = ceil(2.08) = 3 이 나오는 것임에도 불구하고 증가하지 않은 이유는 무엇일까?

    공식문서에서는 다음과 같이 이야기하고있다. 비율이 1에 충분히 가깝다면 스케일링 건너뛴다. 허용오차는 0.1의 default를 가지고 있다. (--horizontal-pod-autoscaler-tolerance)

    The control plane skips any scaling action if the ratio is sufficiently close to 1.0 (within a globally-configurable tolerance, 0.1 by default).

    따라서, 2.08은 0.1만큼의 허용오차에서 무시가 된다.

  • 56% 구간

    파드는 정확하게 2.1을 넘긴 56% 구간이 되자마자 파드가 하나 증가하였다.

Test2

부하 발생

$ kubectl exec -it netutil-669f67cb94-2hdns -- /bin/bash
$ while true; do curl http://php-apache1; sleep 0.1; done

결과 확인

$ while true; do date; k get hpa php-apache1; sleep 10; done
#################### Result ####################
Thu Jan  5 10:28:49 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   0%/50%    2         10        2          40h
Thu Jan  5 10:28:59 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   0%/50%    2         10        2          40h
Thu Jan  5 10:29:09 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   16%/50%   2         10        2          40h
Thu Jan  5 10:29:19 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   16%/50%   2         10        2          40h
Thu Jan  5 10:29:30 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   91%/50%   2         10        2          40h
Thu Jan  5 10:29:40 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   93%/50%   2         10        4          40h
Thu Jan  5 10:29:50 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   93%/50%   2         10        4          40h
Thu Jan  5 10:30:00 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   45%/50%   2         10        4          40h
Thu Jan  5 10:30:10 KST 2023
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache1   Deployment/php-apache1   48%/50%   2         10        4          40h

$ while true; do date; k get po -l run=php-apache1; sleep 5; done
#################### Result ####################
NAME                          READY   STATUS    RESTARTS   AGE
php-apache1-8445df799-67jjv   1/1     Running   0          29m
php-apache1-8445df799-bdlln   1/1     Running   0          39h
Thu Jan  5 10:29:23 KST 2023
NAME                          READY   STATUS              RESTARTS   AGE
php-apache1-8445df799-67jjv   1/1     Running             0          29m
php-apache1-8445df799-bdlln   1/1     Running             0          39h
php-apache1-8445df799-ffff6   0/1     ContainerCreating   0          1s
php-apache1-8445df799-r5z8p   0/1     ContainerCreating   0          1s
Thu Jan  5 10:29:26 KST 2023
NAME                          READY   STATUS    RESTARTS   AGE
php-apache1-8445df799-67jjv   1/1     Running   0          29m
php-apache1-8445df799-bdlln   1/1     Running   0          39h
php-apache1-8445df799-ffff6   1/1     Running   0          4s
php-apache1-8445df799-r5z8p   1/1     Running   0          4s
  • 91~93% 구간

    결과를 확인해보면 91~93% 구간이 확인되자 바로 Pod는 4개로 증가하였다.
    ceil(2 * (93/50)) = ceil(2* 1.86) = ceil(3.72) = 4

  • 48% 구간

    파드가 4개로 증가하고 48%를 유지한다. 이 때는 아래와 같이 desiredReplica수가 currentReplica 수와 같기 때문에 유지한다.
    ceil(4 * (48/50)) = ceil(4* 0.96) = ceil(3.84) = 4




Ref: kubernetes-sigs/metrics-server Ref: High Availability Ref: metric-server 가용성 Ref: Resource metrics pipeline

Tag: [ hpa  kubernetes  eks  aws  ]