Istio

1. Istio service mesh

Istio Architecture

Istio는 컨트롤 플레인과 데이터 플레인으로 나뉜다.

istio-architecture001.png

Istio Component


2. Install Istio

Istio를 설치하는 방법에는 여러가지 방법이 있고 istioctl과 helm을 통한 설치방법을 소개하지만 여기서는 Helm을 이용하기로 한다. Helm의 values파일을 변경하여 필요한 값을 Customize하여 사용하기 위해서다.

2.1. Install with istioctl

Install Download

curl -L https://istio.io/downloadIstio | sh -
cd istio-1.19.3
cp bin/istioctl /usr/local/bin

Install

istioctl install --set profile=demo -y
istioctl uninstall --set profile=demo -y

Injection Configuration

kubectl label namespace default istio-injection=enabled

Deploy the sample application

kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml


2.2. Install with helm

base-values.yaml
istiod-values.yaml
gateway-values.yaml

gateway-values.yaml
ingress-gateway를 위해

  1. Service(NLB)의 설정파일 실제 Listener로 설정될 포트는 서비스 포트이며 타겟 포트로 설정될 포트는 Gateway 서버포트와 일치해야 한다.
  2. 헬스체크에 대해서는 고민이 필요하다. 15021 포트로 헬스체크를 하면 istio-ingressgateway 파드 자체에 대한 헬스체크는 가능하나 backend 서비스에 자체에 대한 헬스체크는 불가능하다. 만약 traffic-port로 변경할 경우 healthcheck-path 값을 통해 backend 중 하나에는 트래픽을 보낼 수 있지만 여러 서비스로는 보낼 수 없다. 그래서 traffic-port를 tcp로 변경하고 traffic-port로 설정하면 각 서비스에 대한 헬스체크가 가능하다(path는 제외하였지만).
service:
  # Type of service. Set to "None" to disable the service entirely
  type: LoadBalancer
  ports:
  # - name: status-port
  #   port: 15021
  #   protocol: TCP
  #   targetPort: 15021
  # - name: https
  #   port: 443
  #   protocol: TCP
  #   targetPort: 10012
  # 실제 트래픽 전달을 위해 사용 하는 포트 (awssdk)
  - name: https
    port: 443
    protocol: TCP
    targetPort: 80
  # 실제 트래픽 전달을 위해 사용 하는 포트 (bookinfo)
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-name: k8s-istiosys-ingressgateway
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-02e6d788fad8afdcf, subnet-020255d69e8c814da #az1-extelb, az3-extelb
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
    service.beta.kubernetes.io/aws-load-balancer-attributes: load_balancing.cross_zone.enabled=true
    service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: ELBSecurityPolicy-TLS13-1-2-2021-06
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:ap-northeast-2:558846430793:certificate/250015a4-4753-4a97-b536-88a6e6aaaf73
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
    
    # SET 1
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "15021"  # traffic-port "15021"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: /healthz/ready
    # SET 2
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: traffic-port  # traffic-port "15021"

    service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: "2"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: "3"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: "20"
    service.beta.kubernetes.io/aws-load-balancer-security-groups: sg-0f94ae1a6d9ba9d69
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: auto-delete=no
  loadBalancerIP: ""
  loadBalancerSourceRanges: []
  externalTrafficPolicy: ""
  externalIPs: []
  ipFamilyPolicy: ""
  ipFamilies: []

설정한 파일을 가지고 총 3개의 Helm 차트를 설치한다.

  1. istio-base
    • CRD, WebhookConfiguration 등이 설치된다.
  2. istiod
  3. istio-ingressgateway
helm repo add istio https://istio-release.storage.googleapis.com/charts
helm repo update
kubectl create namespace istio-system

helm upgrade -i istio-base istio/base -n istio-system -f /Users/bys/workspace/kubernetes/istio/bys-dev-eks-main/helm/base-values.yaml
helm upgrade -i istiod istio/istiod -n istio-system -f /Users/bys/workspace/kubernetes/istio/bys-dev-eks-main/helm/istiod-values.yaml
helm upgrade -i istio-ingressgateway istio/gateway -n istio-system -f /Users/bys/workspace/kubernetes/istio/bys-dev-eks-main/helm/gateway-values.yaml


2.2. Istio object

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: my-gateway
  namespace: some-config-namespace
spec:
  selector:
    app: my-gateway-controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - uk.bookinfo.com
    - eu.bookinfo.com
    tls:
      httpsRedirect: true # sends 301 redirect for http requests
  - port:
      number: 443
      name: https-443
      protocol: HTTPS
    hosts:
    - uk.bookinfo.com
    - eu.bookinfo.com
    tls:
      mode: SIMPLE # enables HTTPS on this port
      serverCertificate: /etc/certs/servercert.pem
      privateKey: /etc/certs/privatekey.pem
  - port:
      number: 9443
      name: https-9443
      protocol: HTTPS
    hosts:
    - "bookinfo-namespace/*.bookinfo.com"
    tls:
      mode: SIMPLE # enables HTTPS on this port
      credentialName: bookinfo-secret # fetches certs from Kubernetes secret
  - port:
      number: 9080
      name: http-wildcard
      protocol: HTTP
    hosts:
    - "*"
  - port:
      number: 2379 # to expose internal service via external port 2379
      name: mongo
      protocol: MONGO
    hosts:
    - "*"

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews-route
spec:
  hosts:
  - reviews.prod.svc.cluster.local
  http:
  - name: "reviews-v2-routes"
    match:
    - uri:
        prefix: "/wpcatalog"
    - uri:
        prefix: "/consumercatalog"
    rewrite:
      uri: "/newcatalog"
    route:
    - destination:
        host: reviews.prod.svc.cluster.local
        subset: v2
  - name: "reviews-v1-route"
    route:
    - destination:
        host: reviews.prod.svc.cluster.local
        subset: v1
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: reviews-destination
spec:
  host: reviews.prod.svc.cluster.local
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2

이 외에도 기타 여러 설정 등을 통해 추가적인 Traffic 제어가 가능하다.


2.3. Kiali dashbaord

Istio 메시의 시각화를 위해 사용할 수 있다. 추후 Sample을 배포하고 흐름을 볼 수 있도록 미리 설치를 진행한다.

Install Kiali

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.21/samples/addons/kiali.yaml

istioctl dashboard kiali


2.4 Deploy Sample Application

kubectl label namespace default istio-injection=enabled

## Deploy sample Deployments, Services 
kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml

## Deploy sample Gateway, VirtualService
kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

정상적으로 배포가 되었는지 확인하기 위한 작업

export INGRESS_NAME=istio-ingressgateway
export INGRESS_NS=istio-system

kubectl get svc "$INGRESS_NAME" -n "$INGRESS_NS"

export INGRESS_HOST=$(kubectl -n "$INGRESS_NS" get service "$INGRESS_NAME" -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
export INGRESS_PORT=$(kubectl -n "$INGRESS_NS" get service "$INGRESS_NAME" -o jsonpath='{.spec.ports[?(@.name=="http")].port}')

export GATEWAY_URL=$INGRESS_HOST:$INGRESS_PORT

curl http://$GATEWAY_URL/productpage
# watch -n 1 curl -o /dev/null -s -w %{http_code} $GATEWAY_URL/productpage


2.5 Deploy Application(개인용)

기존의 존재하던 Application에 Gateway, VirtualService 추가 구성.

awssdk-gateway

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: awssdk-gateway
  namespace: aws
spec:
  # The selector matches the ingress gateway pod labels.
  # If you installed Istio using Helm following the standard documentation, this would be "istio=ingress"
  selector:
    # istio: ingressgateway # use istio default controller
    istio: ingressgateway # use istio default controller
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "istio-awssdk.bys.asia"

awssdk-virtual-service

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: awssdk-virtual-service
  namespace: aws
spec:
  hosts:
    - "*"
  gateways:
    - awssdk-gateway
  http:
    - match:
        - uri:
            prefix: /storage
      route:
        - destination:
            host: awssdk-storage-dev-svc.aws.svc.cluster.local
            port:
              number: 10011
    - match:
        - uri:
            prefix: /iam
      route:
        - destination:
            host: awssdk-iam-dev-svc.aws.svc.cluster.local
            port:
              number: 10012

curl -v https://istio-awssdk.bys.asia/storage/v2/s3/buckets
curl -v https://istio-awssdk.bys.asia/iam/v2/sts/id



10. Troubleshooting

1. Kiali 구성 시, 기존 Prometheus 연결

Kiali dashboard를 사용하기 위해서는 prometheus 구성이 필수이지만 기존 prometheus가 설치된 경우 istio-system에 존재하지 않기 때문에 주소를 등록해주어야 한다.
kiali ConfigMap의 external_services에 custom 설치된 prometheus URL을 추가하여 설정한다.

# kubectl edit cm kiali -n istio-system

......
      external_services:
        custom_dashboards:
          enabled: true
        istio:
          root_namespace: istio-system
        tracing:
          enabled: false
        prometheus:
          url: http://prometheus-server.prometheus.svc.cluster.local/
2. Envoy Access Logging

Gateway, VirtualService 등을 구성하고 503 오류가 발생하여 istio-ingressgateway에서 발생한 오류인지 혹은 각 서비스 Envoy 프록시에서 발생한 오류인지를 확인하려고 kubectl logs 커맨드를 사용하였으나 Access 로그가 보이지 않았다.
Envoy access log를 활성화 하기 위해서는 아래의 Telemetry를 배포한다.

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  accessLogging:
    - providers:
      - name: envoy

📚 References

[1] Istio Architecture - 공식 아키텍처 문서

[2] Kiali Troubleshooting - Kiali FAQ