NGINX with Prometheus Exporter



사전 설정

Prometheus 구성 *

  • Official URL : https://prometheus.io/docs/prometheus/latest/installation/
  • Scrape config 설정 필요 ( 모니터링 대상 NGINX 추가 )
    • sample Config

      # my global config

      global:

        scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.

        evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

        # scrape_timeout is set to the global default (10s).


      # Alertmanager configuration

      alerting:

        alertmanagers:

          - static_configs:

              - targets:

                # - alertmanager:9093


      # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.

      rule_files:

        # - "first_rules.yml"

        # - "second_rules.yml"


      # A scrape configuration containing exactly one endpoint to scrape:

      # Here it's Prometheus itself.

      scrape_configs:

        # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

        - job_name: "prometheus"


          # metrics_path defaults to '/metrics'

          # scheme defaults to 'http'.


          static_configs:

            - targets: ["localhost:9090"]

            - targets: ['10.250.11.99:9113']




Grafana Deploy

  • Dashboard 활용 시 사전 Deploy 필요 

Go Install (Build)

  • Build 시, go 설치 필요

    ## 설치 참고 URL : <https://go.dev/doc/install>
    wget <https://go.dev/dl/go1.19.1.linux-amd64.tar.gz>

    ## 설치
    rm -rf /usr/local/go && tar -C /usr/local -xzf go1.19.1.linux-amd64.tar.gz

    ## Profile 수정
    cat <<EOF >> /etc/profile
    export PATH=$PATH:/usr/local/go/bin
    EOF

    ## Profile 수정 적용
    source /etc/profile

    ## Go 설치확인
    go version
    go version go1.19.1 linux/amd64


NGINX OSS

  • stub_status On

  • Sample Code

    # /etc/nginx/conf.d/http-8080.conf
    server {
    listen 8080;
    location /stub_status { ## OSS Monitoring URI
    stub_status on;
    }
    }


NGINX Plus

  • API Access On

  • Sample Code

    # /etc/nginx/conf.d/http-8080.conf
    server {
    listen 8080;
    location /api { ## Plus API Access URI
    api write=on;
    allow 0.0.0.0/0; ## API ACL IP List
    access_log /dev/null;
    }
    }

NGINX - Prometheus-NJS 활용

  • NGINX Plus에서 직접 Prometheus 메트릭 엔드포인트를 노출
  • Prometheus Exporter 를 활용하지 않고, 3rd Module로 Prometheus 연동

3rd Modules Install

  • Code

    yum search nginx-plus-module-prometheus --show-duplicate

    ## NGINX plus R25 버전 기준
    yum install nginx-plus-module-njs-r25
    yum install nginx-plus-module-prometheus-25+1.3.3-1.el7.ngx.noarch


  • Install 결과 (LOG)

    Installing : nginx-plus-module-njs-25+0.7.3-1.el7.ngx.x86_64                                                                                                                                                 1/1
    ----------------------------------------------------------------------

    The njs dynamic modules for NGINX Plus have been installed.
    To enable these modules, add the following to /etc/nginx/nginx.conf
    and reload nginx:

    load_module modules/ngx_http_js_module.so;
    load_module modules/ngx_stream_js_module.so;

    Please refer to the modules documentation for further details:
    <https://nginx.org/en/docs/njs/>
    <https://nginx.org/en/docs/http/ngx_http_js_module.html>
    <https://nginx.org/en/docs/stream/ngx_stream_js_module.html>

    ----------------------------------------------------------------------
    Verifying : nginx-plus-module-njs-25+0.7.3-1.el7.ngx.x86_64 1/1

    Installed:
    nginx-plus-module-njs.x86_64 0:25+0.7.3-1.el7.ngx

    Complete!
    ============================================================================

    Installing : nginx-plus-module-prometheus-25+1.3.3-1.el7.ngx.noarch 1/1
    ----------------------------------------------------------------------

    The Prometheus exporter NJS module for NGINX Plus has been installed.
    To enable this module, add the following to /etc/nginx/nginx.conf:

    load_module modules/ngx_http_js_module.so;
    load_module modules/ngx_stream_js_module.so;

    then configure nginx in a way similar with the provided example here:

    /usr/share/doc/nginx-plus-module-prometheus/examples/nginx.conf

    and reload nginx.

    Metrics suitable for Prometheus will be available at the "/metrics"
    location (by default).

    Please refer to the module documentation for further details:
    <https://docs.nginx.com/nginx/admin-guide/dynamic-modules/prometheus-njs/>

    ----------------------------------------------------------------------
    Verifying : nginx-plus-module-prometheus-25+1.3.3-1.el7.ngx.noarch 1/1

    Installed:
    nginx-plus-module-prometheus.noarch 0:25+1.3.3-1.el7.ngx

    Complete!


NGINX Config 설정

  • Sample Config (nginx.conf)

    load_module modules/ngx_http_js_module.so;
    load_module modules/ngx_stream_js_module.so;


  • Sample Config (temp.conf)

    js_import /usr/share/nginx-plus-module-prometheus/prometheus.js;

    server {
    listen 9000;
    location = /metrics {
    set $prom_keyval "http_upstream_keyval";
    js_content prometheus.metrics;
    }
    }


Prometheus 연동 확인

  • Config
    • prometheus.yml 에 NGINX - Prometheus-NJS Module 설치 IP 기입
  • Prometheus 확인

NGINX - Prometheus Exporter 구성 (Build)

Prometheus Exporter Download

  • Code

    wget [<https://github.com/nginxinc/nginx-prometheus-exporter/archive/refs/heads/main.zip>](<https://github.com/nginxinc/nginx-prometheus-exporter/archive/refs/heads/main.zip>)

    cd /root/nginx-prometheus-exporter-main/


Prometheus Exporter Build

  • Code

    make


Prometheus Exporter Execute

  • Code

     

    # NGINX Plus 

    ./nginx-prometheus-exporter -nginx.plus -nginx.scrape-uri=http://127.0.0.1:8080/api &


    # NGINX OSS 

    ./nginx-prometheus-exporter -nginx.scrape-uri=http://127.0.0.1:8080/stub_status &





NGINX - Prometheus Exporter 구성 (Docker) *

Prometheus Exporter Image Pull or Build

  • Image Pull or Build

  • Code - Image Build

    wget <https://github.com/nginxinc/nginx-prometheus-exporter/archive/refs/heads/main.zip>
    unzip main.zip
    cd nginx-prometheus-exporter
    make container


  • Code - Docker Image Run

    # NGINX Plus with Image Pull
    docker run -p 9113:9113 nginx/nginx-prometheus-exporter:0.10.0 -nginx.plus -nginx.scrape-uri=http://<nginx-plus>:8080/api

    # NGINX Plus with Local Image
    docker run -p 9113:9113 adf2bedcc588 -nginx.plus -nginx.scrape-uri=http://10.250.11.99:8080/api

    # OSS with Image Pull
    $ docker run -p 9113:9113 nginx/nginx-prometheus-exporter:0.10.0 -nginx.scrape-uri=http://<nginx>:8080/stub_status


모니터링

Prometheus 에서 모니터링

  • Web 에서 모니터링 수행

  • CAP

Grafana Dashboard 연동

  • Dashbard 를 통한 모니터링

Grafana Dashboard Source update

Grafana Dashboard Create


ETC

  • OS Service 등록
    • Process 실행 시, shell logout 시, 모니터링 불가

    • code

      #vi /etc/systemd/system/nginx-prometheus.service

      [Unit]
      Description=NGINX with Prometheus Exporter

      [Service]
      Type=simple
      ExecStart=/home/cco/nginx-prometheus-exporter-main/nginx-prometheus-exporter -nginx.plus -nginx.scrape-uri=http://127.0.0.1:8080/api &

      Restart=on-failure

      [Install]
      WantedBy=multi-user.target

    • Run Command

      # systemctl start nginx-prometheus

      # systemctl status nginx-prometheus
      ● nginx-prometheus.service - NGINX with Prometheus Exporter
      Loaded: loaded (/etc/systemd/system/nginx-prometheus.service; disabled; vendor preset: disabled)
      Active: active (running) since 수 2023-04-05 14:46:47 KST; 8s ago
      Main PID: 15162 (nginx-prometheu)
      CGroup: /system.slice/nginx-prometheus.service
      └─15162 /home/cco/nginx-prometheus-exporter-main/nginx-prometheus-exporter -nginx.plus -n...

      4월 05 14:46:47 nginx.master.cco systemd[1]: Started NGINX with Prometheus Exporter.
      4월 05 14:46:47 nginx.master.cco nginx-prometheus-exporter[15162]: NGINX Prometheus Exporter vers...1
      4월 05 14:46:47 nginx.master.cco nginx-prometheus-exporter[15162]: 2023/04/05 14:46:47 Starting...
      4월 05 14:46:47 nginx.master.cco nginx-prometheus-exporter[15162]: 2023/04/05 14:46:47 Listening ...3
      4월 05 14:46:47 nginx.master.cco nginx-prometheus-exporter[15162]: 2023/04/05 14:46:47 NGINX Prom...d
      Hint: Some lines were ellipsized, use -l to show in full.