NGINX with Prometheus Exporter

NGINX Metric 에 대해 Prometheus 에서 모니터링 가능하게 함
nginx-prometheus-exporter 활용하여 구성
- Official URL : https://github.com/nginxinc/nginx-prometheus-exporter
NGINX Prometheus-NJS 3rd Modules 활용하여 구성
- Official URL : https://docs.nginx.com/nginx/admin-guide/dynamic-modules/prometheus-njs/
NGINX OSS vs NGINX Plus 차이점
- NGINX OSS : stub_status 기준 모니터링 가능
- NGINX Plus : 100+ Metric 기준 모니터링 가능

사전 설정

Prometheus 구성 *

Official URL : https://prometheus.io/docs/prometheus/latest/installation/

Scrape config 설정 필요 ( 모니터링 대상 NGINX 추가 )

sample Config

# my global config

global:

scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.

evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

# scrape_timeout is set to the global default (10s).

# Alertmanager configuration

alerting:

alertmanagers:

- static_configs:

- targets:

# - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.

rule_files:

# - "first_rules.yml"

# - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:

# Here it's Prometheus itself.

scrape_configs:

# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

- job_name: "prometheus"

# metrics_path defaults to '/metrics'

# scheme defaults to 'http'.

static_configs:

- targets: ["localhost:9090"]

- targets: ['10.250.11.99:9113']

Grafana Deploy

Dashboard 활용 시 사전 Deploy 필요

Go Install (Build)

Build 시, go 설치 필요

## 설치 참고 URL : <https://go.dev/doc/install>
wget <https://go.dev/dl/go1.19.1.linux-amd64.tar.gz>

## 설치
rm -rf /usr/local/go && tar -C /usr/local -xzf go1.19.1.linux-amd64.tar.gz

## Profile 수정
cat <<EOF >> /etc/profile
export PATH=$PATH:/usr/local/go/bin
EOF

## Profile 수정 적용
source /etc/profile

## Go 설치확인
go version
go version go1.19.1 linux/amd64

NGINX OSS

stub_status On
Sample Code

# /etc/nginx/conf.d/http-8080.conf
server {
listen 8080;
location /stub_status { ## OSS Monitoring URI
stub_status on;
}
}

NGINX Plus

API Access On
Sample Code

# /etc/nginx/conf.d/http-8080.conf
server {
listen 8080;
location /api { ## Plus API Access URI
api write=on;
allow 0.0.0.0/0; ## API ACL IP List
access_log /dev/null;
}
}

NGINX - Prometheus-NJS 활용

NGINX Plus에서 직접 Prometheus 메트릭 엔드포인트를 노출
Prometheus Exporter 를 활용하지 않고, 3rd Module로 Prometheus 연동

3rd Modules Install

Code

yum search nginx-plus-module-prometheus --show-duplicate

## NGINX plus R25 버전 기준
yum install nginx-plus-module-njs-r25
yum install nginx-plus-module-prometheus-25+1.3.3-1.el7.ngx.noarch

Install 결과 (LOG)

Installing : nginx-plus-module-njs-25+0.7.3-1.el7.ngx.x86_64 1/1
----------------------------------------------------------------------

The njs dynamic modules for NGINX Plus have been installed.
To enable these modules, add the following to /etc/nginx/nginx.conf
and reload nginx:

load_module modules/ngx_http_js_module.so;
load_module modules/ngx_stream_js_module.so;

Please refer to the modules documentation for further details:
<https://nginx.org/en/docs/njs/>
<https://nginx.org/en/docs/http/ngx_http_js_module.html>
<https://nginx.org/en/docs/stream/ngx_stream_js_module.html>

----------------------------------------------------------------------
Verifying : nginx-plus-module-njs-25+0.7.3-1.el7.ngx.x86_64 1/1

Installed:
nginx-plus-module-njs.x86_64 0:25+0.7.3-1.el7.ngx

Complete!
============================================================================

Installing : nginx-plus-module-prometheus-25+1.3.3-1.el7.ngx.noarch 1/1
----------------------------------------------------------------------

The Prometheus exporter NJS module for NGINX Plus has been installed.
To enable this module, add the following to /etc/nginx/nginx.conf:

load_module modules/ngx_http_js_module.so;
load_module modules/ngx_stream_js_module.so;

then configure nginx in a way similar with the provided example here:

/usr/share/doc/nginx-plus-module-prometheus/examples/nginx.conf

and reload nginx.

Metrics suitable for Prometheus will be available at the "/metrics"
location (by default).

Please refer to the module documentation for further details:
<https://docs.nginx.com/nginx/admin-guide/dynamic-modules/prometheus-njs/>

----------------------------------------------------------------------
Verifying : nginx-plus-module-prometheus-25+1.3.3-1.el7.ngx.noarch 1/1

Installed:
nginx-plus-module-prometheus.noarch 0:25+1.3.3-1.el7.ngx

Complete!

NGINX Config 설정

Sample Config (nginx.conf)

load_module modules/ngx_http_js_module.so;
load_module modules/ngx_stream_js_module.so;
Sample Config (temp.conf)

js_import /usr/share/nginx-plus-module-prometheus/prometheus.js;

server {
listen 9000;
location = /metrics {
set $prom_keyval "http_upstream_keyval";
js_content prometheus.metrics;
}
}

Prometheus 연동 확인

Config
- prometheus.yml 에 NGINX - Prometheus-NJS Module 설치 IP 기입
Prometheus 확인

NGINX - Prometheus Exporter 구성 (Build)

Prometheus Exporter Download

Code

wget [<https://github.com/nginxinc/nginx-prometheus-exporter/archive/refs/heads/main.zip>](<https://github.com/nginxinc/nginx-prometheus-exporter/archive/refs/heads/main.zip>)

cd /root/nginx-prometheus-exporter-main/

Prometheus Exporter Build

Code

make

Prometheus Exporter Execute

Code

# NGINX Plus
./nginx-prometheus-exporter -nginx.plus -nginx.scrape-uri=http://127.0.0.1:8080/api &

# NGINX OSS
./nginx-prometheus-exporter -nginx.scrape-uri=http://127.0.0.1:8080/stub_status &

NGINX - Prometheus Exporter 구성 (Docker) *

Official Docker Image : https://hub.docker.com/r/nginx/nginx-prometheus-exporter

Prometheus Exporter Image Pull or Build

Image Pull or Build
Code - Image Build

wget <https://github.com/nginxinc/nginx-prometheus-exporter/archive/refs/heads/main.zip>
unzip main.zip
cd nginx-prometheus-exporter
make container

Code - Docker Image Run

# NGINX Plus with Image Pull
docker run -p 9113:9113 nginx/nginx-prometheus-exporter:0.10.0 -nginx.plus -nginx.scrape-uri=http://<nginx-plus>:8080/api

# NGINX Plus with Local Image
docker run -p 9113:9113 adf2bedcc588 -nginx.plus -nginx.scrape-uri=http://10.250.11.99:8080/api

# OSS with Image Pull
$ docker run -p 9113:9113 nginx/nginx-prometheus-exporter:0.10.0 -nginx.scrape-uri=http://<nginx>:8080/stub_status

모니터링

Prometheus 에서 모니터링

Web 에서 모니터링 수행
CAP

Grafana Dashboard 연동

Dashbard 를 통한 모니터링

Grafana Dashboard Source update

Prometheus Data 를 Grafana 에 연결해줘야함 ( https://prometheus.io/docs/visualization/grafana/#using )
CAP
GUI >> Configuration - Data Sources - Add Data Source
URL : 그라파나 IP ( 127.0.0.1 X )

Grafana Dashboard Create

GUI DashBoard 생성
Dashboard Source : https://github.com/nginxinc/nginx-prometheus-exporter/blob/main/grafana/dashboard.json
Grafana Dasnboard Create
Grafana Dashboard Update
- 상세 Metric 에 대해서 설정 확인 필요
Grafana Dashboard -설정 완료 화면 (Sample)
- 각NGINX Plus Instance에 대해 모니터링수행 가능

ETC

OS Service 등록

Process 실행 시, shell logout 시, 모니터링 불가

code

#vi /etc/systemd/system/nginx-prometheus.service

[Unit]
Description=NGINX with Prometheus Exporter

[Service]
Type=simple
ExecStart=/home/cco/nginx-prometheus-exporter-main/nginx-prometheus-exporter -nginx.plus -nginx.scrape-uri=http://127.0.0.1:8080/api &

Restart=on-failure

[Install]
WantedBy=multi-user.target

Run Command

# systemctl start nginx-prometheus

# systemctl status nginx-prometheus
● nginx-prometheus.service - NGINX with Prometheus Exporter
Loaded: loaded (/etc/systemd/system/nginx-prometheus.service; disabled; vendor preset: disabled)
Active: active (running) since 수 2023-04-05 14:46:47 KST; 8s ago
Main PID: 15162 (nginx-prometheu)
CGroup: /system.slice/nginx-prometheus.service
└─15162 /home/cco/nginx-prometheus-exporter-main/nginx-prometheus-exporter -nginx.plus -n...

4월 05 14:46:47 nginx.master.cco systemd[1]: Started NGINX with Prometheus Exporter.
4월 05 14:46:47 nginx.master.cco nginx-prometheus-exporter[15162]: NGINX Prometheus Exporter vers...1
4월 05 14:46:47 nginx.master.cco nginx-prometheus-exporter[15162]: 2023/04/05 14:46:47 Starting...
4월 05 14:46:47 nginx.master.cco nginx-prometheus-exporter[15162]: 2023/04/05 14:46:47 Listening ...3
4월 05 14:46:47 nginx.master.cco nginx-prometheus-exporter[15162]: 2023/04/05 14:46:47 NGINX Prom...d
Hint: Some lines were ellipsized, use -l to show in full.

Technical Service Center

NGINX Plus - Prometheus & Grafana 연동 인쇄

NGINX with Prometheus Exporter

사전 설정

Prometheus 구성 *

Grafana Deploy

Go Install (Build)

NGINX OSS

NGINX Plus

NGINX - Prometheus-NJS 활용

3rd Modules Install

NGINX Config 설정

Prometheus 연동 확인

NGINX - Prometheus Exporter 구성 (Build)

Prometheus Exporter Download

Prometheus Exporter Build

Prometheus Exporter Execute

NGINX - Prometheus Exporter 구성 (Docker) *

Prometheus Exporter Image Pull or Build

모니터링

Prometheus 에서 모니터링

Grafana Dashboard 연동

Grafana Dashboard Source update

Grafana Dashboard Create

ETC

NGINX Plus - Prometheus & Grafana 연동 인쇄

NGINX with Prometheus Exporter

사전 설정

Prometheus 구성 *

Grafana Deploy

Go Install (Build)

NGINX OSS

NGINX Plus

NGINX - Prometheus-NJS 활용

3rd Modules Install

NGINX Config 설정

Prometheus 연동 확인

NGINX - Prometheus Exporter 구성 (Build)

Prometheus Exporter Download

Prometheus Exporter Build

Prometheus Exporter Execute

NGINX - Prometheus Exporter 구성 (Docker) *

Prometheus Exporter Image Pull or Build

모니터링

Prometheus 에서 모니터링

Grafana Dashboard 연동

Grafana Dashboard Source update

Grafana Dashboard Create

ETC

관련 아티클