Kubernetes Volume

Container Storage Interface(CSI)

Storage Provider가 플러그인을 한 번만 개발하면 여러 Container Orchestration에서 동작할 수 있도록 하는 표준 Container Storage Interface에 대한 정의


A CO interacts with an Plugin through RPCs. Each SP MUST provide:


How to work

k8s-workflow-csi-driver


EBS/EFS CSI Driver의 구성요소

## EFS CSI Controller
- efs-plugin
- csi-provisioner
- liveness-probe

## EFS CSI Node
- efs-plugin
- csi-driver-registrar
- liveness-probe


## EBS CSI Controller
- ebs-plugin
- csi-provisioner
- csi-attacher
- csi-resizer
- liveness-probe
- csi-snapshotter (CSI Snapshot controller와 같이 사용, Addon 설치)

## EBS CSI Node
- ebs-plugin
- node-driver-registrar
- liveness-probe

CSI Volume Plugins in Kubernetes Design Doc

[Leader & Lease]

Leader 선출에 의해 동작하는 애플리케이션은 lease 오브젝트를 가지고 있다.

# Lease objects
$ kubectl get leases -A
NAMESPACE         NAME                                                      HOLDER                                                                                   AGE
......
kube-system       ebs-csi-aws-com                                           1725345475903-4152-ebs-csi-aws-com                                                       664d
kube-system       efs-csi-aws-com                                           1725345522671-8163-efs-csi-aws-com                                                       648d
......

# Describe leases
$ kubectl describe lease ebs-csi-aws-com -n kube-system
Name:         ebs-csi-aws-com
Namespace:    kube-system
Labels:       <none>
Annotations:  <none>
API Version:  coordination.k8s.io/v1
Kind:         Lease
Metadata:
  Creation Timestamp:  2022-11-09T02:12:09Z
  Resource Version:    380944711
  UID:                 e8e8f63a-8556-408b-a1f2-628bdd3e2ca6
Spec:
  Acquire Time:            2024-09-03T06:39:03.304069Z
  Holder Identity:         1725345475903-4152-ebs-csi-aws-com
  Lease Duration Seconds:  15
  Lease Transitions:       130
  Renew Time:              2024-09-04T01:41:37.828660Z
Events:                    <none>

# Describe Rolebinding  
$ kubectl describe rolebinding ebs-csi-leases-rolebinding -n kube-system
Name:         ebs-csi-leases-rolebinding
Labels:       app.kubernetes.io/component=csi-driver
              app.kubernetes.io/managed-by=EKS
              app.kubernetes.io/name=aws-ebs-csi-driver
              app.kubernetes.io/version=1.34.0
Annotations:  <none>
Role:
  Kind:  Role
  Name:  ebs-csi-leases-role
Subjects:
  Kind            Name                   Namespace
  ----            ----                   ---------
  ServiceAccount  ebs-csi-controller-sa  kube-system

# Describe Role
$ kubectl describe role ebs-csi-leases-role -n kube-system
Name:         ebs-csi-leases-role
Labels:       app.kubernetes.io/component=csi-driver
              app.kubernetes.io/managed-by=EKS
              app.kubernetes.io/name=aws-ebs-csi-driver
              app.kubernetes.io/version=1.34.0
Annotations:  <none>
PolicyRule:
  Resources                   Non-Resource URLs  Resource Names  Verbs
  ---------                   -----------------  --------------  -----
  leases.coordination.k8s.io  []                 []              [get watch list delete update create]

Lease 오브젝트를 업데이트하는 주체는 리더다.

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "6b69f448-d54b-4c91-aee7-8adb4bca75ce",
  "stage": "ResponseComplete",
  "requestURI": "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/ebs-csi-aws-com",
  "verb": "update",
  "user": {
    "username": "system:serviceaccount:kube-system:ebs-csi-controller-sa",
    "uid": "04b51e47-d1db-4c65-a501-7a6eb6a6772a",
    "groups": [
      "system:serviceaccounts",
      "system:serviceaccounts:kube-system",
      "system:authenticated"
    ],
    "extra": {
      "authentication.kubernetes.io/credential-id": [
        "JTI=c5d82535-2f29-40ec-8b93-47b5d997ef41"
      ],
      "authentication.kubernetes.io/node-name": [
        "ip-10-20-129-248.ap-northeast-2.compute.internal"
      ],
      "authentication.kubernetes.io/node-uid": [
        "79f64dda-e2a8-4fa4-9ccf-b89c1d586f03"
      ],
      "authentication.kubernetes.io/pod-name": [
        "ebs-csi-controller-67988b76d8-f46gv"
      ],
      "authentication.kubernetes.io/pod-uid": [
        "d703f4ee-8654-416c-ad8d-73b918c2980f"
      ]
    }
  },
  "sourceIPs": [
    "10.20.130.198"
  ],
  "userAgent": "csi-provisioner/v0.0.0 (linux/amd64) kubernetes/$Format",

}

Lease 오브젝트를 주기적으로 업데이트하여 리더의 health를 확인하고 만약 리더가 health 체크를 하지 못하는 경우 새로운 리더를 선출한다.

Leader가 아닌 파드의 로그

$ kubectl logs -f ebs-csi-controller-67988b76d8-ptmzv -c ebs-plugin  -n kube-system

I0903 06:38:44.006320       1 leaderelection.go:250] attempting to acquire leader lease kube-system/ebs-csi-aws-com...

참고




References