NGINX Plus Ingress Controller + NAP 설치 가이드 with AWS EKS : Technical Service Center

구성 환경
- AWS - EKS ( 2 Worker Node )
- Docker Registry : docker.com > 개인 Private Registry 이용
- ELK - 모니터링
- nginx Plus 인증서 필요

Flow
- Nginx Plus Ingres Controller Image 사전 준비
- NGINX Ingress Controller w/NAP 배포
- Test POD/Service/Ingress Resource 배포 및 정상 동작 확인
- K8S - CRD Resource 생성 - NAP Signature 작성 / NAP Logging Policy 작성
- Test Ingress Resource 에 NAP CRD Resource ADD 및 서비스 확인

NAP Ingres Controller Image 사전 준비
- F5 Private Registry Access 를 위한 인증서 사전 작업

mkdir /etc/docker/certs.d/private-registry.nginx.com
cp nginx-repo.crt /etc/docker/certs.d/private-registry.nginx.com/client.cert
cp nginx-repo.key /etc/docker/certs.d/private-registry.nginx.com/client.key

NGINX Ingress Controller with NAP Image 버전 검색 및 Image Pull
- 인증서 경로 설정 필요

curl <https://private-registry.nginx.com/v2/nginx-ic-nap/nginx-plus-ingress/tags/list> --key /etc/docker/certs.d/private-registry.nginx.com/client.key --cert /etc/docker/certs.d/private-registry.nginx.com/client.cert | jq

[root@ip-10-10-10-217 private-registry.nginx.com]# curl <https://private-registry.nginx.com/v2/nginx-ic-nap/nginx-plus-ingress/tags/list> --key /etc/docker/certs.d/private-registry.nginx.com/client.key --cert /etc/docker/certs.d/private-registry.nginx.com/client.cert | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   362  100   362    0     0    305      0  0:00:01  0:00:01 --:--:--   305
{
  "name": "nginx-ic-nap/nginx-plus-ingress",
  "tags": [
    "1.12.0-ubi",
    "1.12.0",
    "1.12.1-ubi",
    "1.12.1",
    "1.12.2-ubi",
    "1.12.2",
    "1.12.3-ubi",
    "1.12.3",
    "1.12.4-ubi",
    "1.12.4",
    "2.0.0-ubi",
    "2.0.0",
    "2.0.1-ubi",
    "2.0.1",
    "2.0.2-ubi",
    "2.0.2",
    "2.0.3-ubi",
    "2.0.3",
    "2.1.0-ubi",
    "2.1.0",
    "2.1.1-ubi",
    "2.1.1",
    "2.1.2-ubi",
    "2.1.2",
    "2.2.0-ubi",
    "2.2.0",
    "2.2.1-ubi",
    "2.2.1",
    "2.2.2-ubi",
    "2.2.2"
  ]
}

NAP Image > Private Docker Registry Upload
- Private Registry 미 구성 관계로, Docker - Private Registry 이용

docker pull private-registry.nginx.com/nginx-ic-nap/nginx-plus-ingress:2.2.0-ubi
docker tag private-registry.nginx.com/nginx-ic-nap/nginx-plus-ingress:2.2.0-ubi sometime4231/elk/nginx-plus-ingress:2.2.0-ubi
docker push $REGISTRY/elk/nginx-ingress-nap/nginx-plus-ingress:2.2.0-ubi

Nginx Ingress 설치 전 사전작업
- OSS 설치와 유사함
- Ingress git 다운로드

# Download
git clone <https://github.com/nginxinc/kubernetes-ingress.git>
cd kubernetes-ingress/deployments/

# Git Check-Out - 버전 확인 // 220526 기준, 2.2.0
git checkout v2.2.0

Ingress 관련 k8s resource 구성

kubectl apply -f common/ns-and-sa.yaml
#namespace/nginx-ingress created
#serviceaccount/nginx-ingress created
kubectl apply -f common/default-server-secret.yaml
#secret/default-server-secret created
kubectl apply -f common/nginx-config.yaml
#configmap/nginx-config created
kubectl apply -f rbac/rbac.yaml
#clusterrole.rbac.authorization.k8s.io/nginx-ingress created
#clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress created
kubectl apply -f rbac/ap-rbac.yaml  ## NAP 용 추가**
#clusterrole.rbac.authorization.k8s.io/nginx-ingress-app-protect created
#clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-app-protect created

# NAP Definition Custom  Resource 
kubectl apply -f common/crds/k8s.nginx.org_virtualservers.yaml
#customresourcedefinition.apiextensions.k8s.io/virtualservers.k8s.nginx.org created
kubectl apply -f common/crds/k8s.nginx.org_virtualserverroutes.yaml
#customresourcedefinition.apiextensions.k8s.io/virtualserverroutes.k8s.nginx.org created
kubectl apply -f common/crds/k8s.nginx.org_transportservers.yaml
#customresourcedefinition.apiextensions.k8s.io/transportservers.k8s.nginx.org created
kubectl apply -f common/crds/k8s.nginx.org_policies.yaml
#customresourcedefinition.apiextensions.k8s.io/policies.k8s.nginx.org created

# NAP Default CRD Apply 
kubectl apply -f common/crds/appprotect.f5.com_aplogconfs.yaml
kubectl apply -f common/crds/appprotect.f5.com_appolicies.yaml
kubectl apply -f common/crds/appprotect.f5.com_apusersigs.yaml

Ingress Controller 배포

# Image Pull
docker pull $REGISTRY/elk:nic-nap-2.2.0-ubi

vi deployment/nginx-plus-ingress.yaml
>> 61 Line 주석 해제 " - -enable-app-protect "

kubectl apply -f deployment/nginx-plus-ingress.yaml
#ingressclass.networking.k8s.io/nginx created

AWS ELB 생성

kubectl apply -f service/loadbalancer-aws-elb.yaml
#service/nginx-ingress created
kubectl apply -f common/nginx-config.yaml > 내용 추가 후 업데이트
#configmap/nginx-config configured

배포 완료 이후

#k get svc -A
nginx-ingress   nginx-ingress   LoadBalancer   172.20.87.209   aa317044bfe80461086eaaf8e1c48aae-925891627.ap-northeast-2.elb.amazonaws.com   80:31024/TCP,443:32004/TCP   108s

Test POD/Service/Ingress Resource 배포 및 정상 동작 확인
- POD Deploy

apiVersion: apps/v1
kind: Deployment
metadata:
  name: f5-hello-world-web
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: f5-hello-world-web
  template:
    metadata:
      labels:
        app: f5-hello-world-web
    spec:
      containers:
      - env:
        - name: service_name
          value: f5-hello-world-web
        image: f5devcentral/f5-hello-world:latest
        imagePullPolicy: IfNotPresent
        name: f5-hello-world-web
        ports:
        - containerPort: 8080
          protocol: TCP

Service Deploy

apiVersion: v1
kind: Service
metadata:
  name: f5-hello-world-web
  namespace: default
  labels:
    app: f5-hello-world-web
spec:
  ports:
  - name: f5-hello-world-web
    port: 8080
    protocol: TCP
    targetPort: 8080
  type: ClusterIP
  selector:
    app: f5-hello-world-web

Ingress Deploy
- Host : AWS ELB A record

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: f5-hello-world-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: aa317044bfe80461086eaaf8e1c48aae-925891627.ap-northeast-2.elb.amazonaws.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: f5-hello-world-web
            port:
              number: 8080

K8S - CRD Resource 생성 : NAP Signature 작성 / NAP Logging Policy 작성
- Annotation 추가해줘야함
  - ingress Deply 했던 YAML 에 추가해야함
    - 앞서 선언한 CRD 에 대해 명시필요 ( Annotation )

#cat Ingress Deploy.yaml 

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: f5-hello-world-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    appprotect.f5.com/app-protect-policy: "test-sig"   ## Policy
    appprotect.f5.com/app-protect-enable: "True"                      ## NAP Enable - True / False
    appprotect.f5.com/app-protect-security-log-enable: "True"         ## NAP Logging - True / False
    appprotect.f5.com/app-protect-security-log: "default/logconf"     ## NAP Log Policy - Loggung
    appprotect.f5.com/app-protect-security-log-destination: "syslog:server=10.10.10.70:5144"  ## NAP Log Syslog Server
spec:
  rules:
  - host: aa317044bfe80461086eaaf8e1c48aae-925891627.ap-northeast-2.elb.amazonaws.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: f5-hello-world-web
            port:
              number: 8080

Sample NAP Policy ( test-sig.yaml )
- CRD 검증 : k describe appolicy test-sig

---
apiVersion: appprotect.f5.com/v1beta1
kind: APPolicy
metadata:
  name: test-sig
spec:
  policy:
    template:
      name: POLICY_TEMPLATE_NGINX_BASE
    applicationLanguage: utf-8
    name: policy_name
    enforcementMode: blocking

Sample Log Config ( ap-logconf.yaml)
- CRD 검증 : k describe aplogconf logconf

apiVersion: appprotect.f5.com/v1beta1
kind: APLogConf
metadata:
  name: logconf
spec:
  content:
    format: default
    max_message_size: 64k
    max_request_size: any
  filter:
    request_type: all

k8s CRD Resource 확인

[root@ip-10-10-10-217 sample_apps]# kubectl get crd
NAME                                         CREATED AT
aplogconfs.appprotect.f5.com                 2022-05-27T01:53:26Z
appolicies.appprotect.f5.com                 2022-05-27T01:53:27Z
apusersigs.appprotect.f5.com                 2022-05-27T01:53:29Z
eniconfigs.crd.k8s.amazonaws.com             2022-05-26T07:47:23Z
policies.k8s.nginx.org                       2022-05-27T00:42:51Z
securitygrouppolicies.vpcresources.k8s.aws   2022-05-26T07:47:27Z
transportservers.k8s.nginx.org               2022-05-27T00:42:49Z
virtualserverroutes.k8s.nginx.org            2022-05-27T00:42:49Z
virtualservers.k8s.nginx.org                 2022-05-27T00:42:47Z
[root@ip-10-10-10-217 sample_apps]# kubectl get aplogconfs
NAME      AGE
logconf   3d4h
[root@ip-10-10-10-217 sample_apps]# kubectl get appolicies
NAME       AGE
test-sig   3d4h

Test Ingress Resource 에 NAP CRD Resource ADD 및 서비스 확인

[root@ip-10-10-10-217 sample_apps]# k get ing
NAME                     CLASS    HOSTS                                                                         ADDRESS   PORTS   AGE
f5-hello-world-ingress   <none>   aa317044bfe80461086eaaf8e1c48aae-925891627.ap-northeast-2.elb.amazonaws.com             80      3d5h
[root@ip-10-10-10-217 sample_apps]# k describe ing
Name:             f5-hello-world-ingress
Namespace:        default
Address:
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host                                                                         Path  Backends
  ----                                                                         ----  --------
  aa317044bfe80461086eaaf8e1c48aae-925891627.ap-northeast-2.elb.amazonaws.com
                                                                               /   f5-hello-world-web:8080 (10.10.10.204:8080,10.10.10.230:8080)
Annotations:                                                                   appprotect.f5.com/app-protect-enable: True
                                                                               appprotect.f5.com/app-protect-policy: test-sig
                                                                               appprotect.f5.com/app-protect-security-log: default/logconf
                                                                               appprotect.f5.com/app-protect-security-log-destination: syslog:server=10.10.10.70:5144
                                                                               appprotect.f5.com/app-protect-security-log-enable: True
                                                                               kubernetes.io/ingress.class: nginx
Events:                                                                        <none>
[root@ip-10-10-10-217 sample_apps]#

Test : ELB A Record 로 접속 이후 확인

NGINX Plus Ingress Controller + NAP 설치 가이드 with AWS EKS 인쇄

관련 아티클