Docker

General Information

Exporter Ports

Exporter

Port

API

nginx_api_proxy:8081/metrics

nginx

nginx_container:9113/metrics

TFSS

bio_container:8501/metrics/tfss

Blackbox

blackbox_container:9115

PostgreSQL

postgres_container:9187/metrics

You can find the already prepared exporter configuration in our repository. After deployment, replace container names with actual addresses. For example, if your nginx container is at 192.168.8.8, change - targets: ["nginx_container:9113"] to - targets: ["192.168.8.8:9113"].

Healthcheck Paths

API

GET http://api:8000/api/version

GET http://api:8000/api/healthcheck

BIO

GET http://bio:8501/v1/models/{{model_name}}

POST http://bio:8501/v1/models/dummy:predict

method: POST
body: '{"inputs": {"images_bytes": [{"b64": "aaa"}]}}'

API

To enable API metrics, in the docker/template/nginx.yml file, uncomment 9081:8081:

ports:
  - "9080:8080"
  - "9081:8081"

Prometheus Job

  - job_name: "oz-api"
    scheme: http
    metrics_path: /metrics
    file_sd_configs:
      - files:
          - sd/oz-api.yml

The sd/oz-api.yml file (instead of placeholders, put your container names and host address):

- targets:
    - api_container1:8081
  labels:
    host: api1
    role: oz-api
    ozforensics: api
    oz_service: "true"
#- targets:
#    - api_container2:8081
#  labels:
#    host: api1
#    role: oz-api
#    ozforensics: api
#    oz_service: "true"

Metrics Overview

Metric

Description

Type

Labels

oz_api_versions_total (also available as oz_api_versions_created, oz_api_info_created, oz_api_info_total)

Displays API version info when you call /api/version

counter

lamb

core – core version

oz_api – API version

oz_api_analyses_total

Number of processed analyses

counter

analysis_result – analysis status (FINISHED, FAILED)

analysis_type – analysis type (QUALITY, BIOMETRY, DOCUMENTS)

company_id – company identifier

resolution_status – analysis resolution (SUCCESS, DECLINED)

oz_api_analyses_created

Number of analyses that have been initiated

gauge

analysis_result – analysis status (FINISHED, FAILED)

analysis_type – analysis type (QUALITY, BIOMETRY, DOCUMENTS)

company_id – company identifier

resolution_status – analysis resolution (SUCCESS, DECLINED)

oz_api_analyses_in_progress

Number of analyses currently processing

gauge

analysis_type – analysis type (QUALITY, BIOMETRY, DOCUMENTS)

company_id – company identifier

oz_api_analyse_duration_seconds_bucket

A histogram for analysis duration. Default buckets: 0.1,0.5,1,1.5,2,2.5,3,3.5,4,4.5,5,6,7,8,10,12,15,20,30,inf

histogram

analysis_result – analysis status (FINISHED, FAILED)

analysis_type – analysis type (QUALITY, BIOMETRY, DOCUMENTS)

resolution_status – analysis resolution (SUCCESS, DECLINED)

company_id – company identifier

le – bucket size

oz_api_analyse_duration_seconds_count

Sum of analyses' durations

counter

analysis_result – analysis status (FINISHED, FAILED)

analysis_type – analysis type (QUALITY, BIOMETRY, DOCUMENTS)

resolution_status – analysis status (SUCCESS, DECLINED)

company_id – company identifier

oz_api_analyse_duration_seconds_sum

Number of analyses whose durations were counted

counter

analysis_result – analysis resolution (FINISHED, FAILED)

analysis_type – analysis type (QUALITY, BIOMETRY, DOCUMENTS)

resolution_status – analysis resolution (SUCCESS, DECLINED)

company_id – company identifier

oz_api_analyse_duration_seconds_created

Timestamp of beginning of duration metric counting

django_http_requests_total_by_transport_total

Number of requests by transport protocol

counter

transport – protocol (HTTP, HTTPS)

uid – user identifier

django_http_responses_before_middlewares_total

Number of Django responses before running middleware

counter

uid – user identifier

django_http_exceptions_total_by_type_total

Number of Django exceptions by type

counter

exception_type – type of exception

uid – user identifier

django_http_requests_latency_seconds_by_view_method_bucket

Histogram of request processing latency by views

histogram

method – HTTP method

uid – user identifier

view_name – view name

le – bucket size

django_http_requests_latency_seconds_by_view_method_count

Number of request processing durations by views

counter

method – HTTP method

uid – user identifier

view_name – view name

django_http_requests_latency_seconds_by_view_method_sum

Sum of request processing durations by views

counter

method – HTTP method

uid – user identifier

view_name – view name

Illustrative screenshots:

Grafana Alerts

groups:
  - name: API alerts
    rules:
      - alert: API is down
        expr: probe_success{job="blackbox-api-healthcheck"} != 1
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "Oz API has been down or showing errors for more than 30 seconds!"
          description: "Critical: Oz API service is down on the host {{ $labels.instance }}!"

      - alert: Absent API metrics!
        expr: absent(oz_api_versions_total)
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Absent API metrics!!!"
          description: "Critical: check API containers, API might be not working properly!!!"

      - alert: High total API analysis requests
        expr: sum (rate(oz_api_analyses_total{}[1m])) > 1
        labels:
          severity: warning
        annotations:
          summary: "Too many API analysis requests per second!"
          description: "API analysis request rate is {{ $value }} rps."

      - alert: High failed API analysis rate
        expr: sum by (analysis_result)(rate(oz_api_analyses_total{analysis_result="FAILED"}[1m])) > 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Failed API analyses detected"
          description: "API analysis failure rate is {{ $value }} rps in the last minute."

      - alert: High API analysis duration
        expr: histogram_quantile(0.95,sum by (le, analysis_type) (rate(oz_api_analyse_duration_seconds_bucket{}[1m]))) > 7
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.analysis_type }} analyses are going slow!"
          description: "The duration of 95% {{ $labels.analysis_type }} analyses is longer than {{ $value }} seconds in the last 10 minutes."

      - alert: API exception errors
        expr: sum by (exception_type)(rate(django_http_exceptions_total_by_type_total{exception_type!~"AuthCredentialsExpired|AuthCredentialsInvalid|AuthCredentialsIsNotProvided|AuthForbidden|InvalidBodyStructureError|InvalidParamValueErrorNotExistError|OSError|InvalidParamValueError|NotExistError"}[1m])) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.exception_type }} in API exception errors"
          description: "{{ $value }} rps in API returned {{ $labels.exception_type }} in the last 5 minutes."

TFSS

To enable metrics, set TFSS_PROMETHEUS="true" in /configs/env/tfss.env.

To change the default metrics path, uncomment TFSS_METRICS_URI="/metrics/tfss" in /configs/env/tfss.env and define the path.

Prometheus Job

Instead of placeholder, put your container name.

  - job_name: "tfss"
    metrics_path: /metrics/tfss
    scheme: http
    static_configs:
      - targets: ["bio_container:8501"]
        labels:
          bio: ozforensics
          ozforensics: true

Metrics Overview

Metric

Description

:tensorflow:serving:request_count

Total number of requests to TFSS

:tensorflow:serving:request_latency_bucket

Histogram of request processing time

:tensorflow:serving:request_latency_sum

Sum of request processing durations

:tensorflow:serving:request_latency_count

Total number of requests whose durations were counted

:tensorflow:cc:saved_model:load_attempt_count

Loaded models

Illustrative screenshots:

TFSS alerts are covered in the Blackbox section below.

nginx

Prometheus Job

Instead of placeholder, put your container name.

  - job_name: "nginx"
    static_configs:
      - targets: ["nginx_container:9113"]
        labels:
          nginx: ozforensics
          ozforensics: true

Metrics Overview

Metric

Description

nginx_up

Shows whether nginx is running: 1 – service is up, 0 – service is down

nginx_connections_accepted

Number of connections accepted by nginx

nginx_connections_handled

Number of connections handled by nginx

nginx_connections_active

Active nginx connections

Illustrative screenshots:

Grafana Alerts

groups:
  - name: NGINX alerts
    rules:
      - alert: nginx is down
        expr: nginx_up != 1
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "nginx has been down for more than 30 seconds!"
          description: "Critical: nginx service is down on the host {{ $labels.instance }}!"

      - alert: nginx not all connections are handled
        expr: rate (nginx_connections_handled[5m]) / rate (nginx_connections_accepted[5m]) <1
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "nginx has issues with handling connections"
          description: "Critical: not all accepted connections have been handled by nginx on the host {{ $labels.instance }} for more than 3 minutes!"

      - alert: "High number of active connections in nginx"
        expr: nginx_connections_active > 3000
        for: 3m
        labels:
          severity: critical
        annotations:
          summary: "High number of active connections in nginx"
          description: "Critical: nginx has had too many active connections on the host {{ $labels.instance }} for more than 3 minutes!"

Blackbox

Blackbox probes verify that API and TFSS are responding correctly. You need to configure the modules, SD target files, and Prometheus jobs as shown below.

Configuration

modules:
  tfss_models_probe:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: []  # Defaults to 2xx
      method: GET
      fail_if_ssl: false
      fail_if_not_ssl: false
      fail_if_body_not_matches_regexp:
        - "\"error_code\": \"OK\""
        - "\"state\": \"AVAILABLE\""
      tls_config:
        insecure_skip_verify: false
      preferred_ip_protocol: "ip4"

  tfss_predict_probe:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: []  # Defaults to 2xx
      method: POST
      body: '{"inputs": {"images_bytes": [{"b64": "aaa"}]}}'
      follow_redirects: true
      fail_if_ssl: false
      fail_if_not_ssl: false
      fail_if_body_not_matches_regexp:
        - '\"outputs\": 1\.0'
      tls_config:
        insecure_skip_verify: false
      preferred_ip_protocol: "ip4"

  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: []
      method: GET
      preferred_ip_protocol: "ip4"
      tls_config:
        insecure_skip_verify: false

  api_healthcheck:
    prober: http
    timeout: 5s
    http:
      method: GET
      valid_status_codes: []
      fail_if_body_not_matches_regexp:
        - '{"generally_healthy": true,(| "celery": (\{\}|\{(?:\s*"[^"]+": "ok",)* "[^"]+": "ok"\s*\}),) "db": "ok"(|, "jwt": {"status": "ok"})}'

SD Files Configuration for Blackbox (instead of placeholders, put your container names)

blackbox_tfss_models.yml:

[
  {
    "targets": ["http://bio_container:8501/v1/models/inquisitor_c"],
    "labels":
      {
        "host": "bio:8501",
        "role": "oz-bio",
        "ssl_alert": "false",
        "http_code": "200",
        "model": "inquisitor_c",
        "ozforensics": "true",
      },
  },
]

blackbox_tfss_probe.yml:

[
  {
    "targets": ["http://bio_container:8501/v1/models/dummy:predict"],
    "labels":
      {
        "host": "bio:8501",
        "role": "oz-bio-balancer",
        "ssl_alert": "false",
        "http_code": "200",
        "model": "dummy",
        "ozforensics": "true",
      },
  },
]

blackbox-exporter-ozforensics.yml:

[
  {
    "targets": ["http://api_container:8000/api/version"],
    "labels":
      {
        "host": "blackbox_container:9115",
        "role": "API",
        "ssl_alert": "false",
        "http_code": "200",
        "ozforensics": "true",
      },
  },
]

blackbox-api-healthcheck.yml:

[
  {
    "targets": ["http://api_container:8000/api/healthcheck"],
    "labels":
      {
        "job": "blackbox-api-healthcheck",
        "host": "api:8000/api/healthcheck",
        "role": "oz-api",
        "ssl_alert": "false",
        "http_code": "200",
        "ozforensics": "true",
      },
  },
]

Prometheus Job

Instead of placeholders, put your container names and host addresses.

  - job_name: "BlackBox_tfss_models"
    scrape_interval: 30s
    metrics_path: /probe
    scheme: http
    params:
      module: [tfss_models_probe]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox_container:9115
    file_sd_configs:
      - files:
          - sd/BlackBox_tfss_models.yml

  - job_name: "BlackBox_tfss_probe"
    scrape_interval: 30s
    metrics_path: /probe
    scheme: http
    params:
      module: [tfss_predict_probe]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox_container:9115
    file_sd_configs:
      - files:
          - sd/BlackBox_tfss_probe.yml

  - job_name: 'BlackBox_exporter_ozforensics'
    scrape_interval: 60s
    metrics_path: /probe
    params:
      module: [http_2xx]
    relabel_configs:
     - source_labels: [__address__]
       target_label: __param_target
     - source_labels: [__param_target]
       target_label: instance
     - target_label: __address__
       replacement: blackbox_container:9115
    file_sd_configs:
      - files:
        - sd/blackbox-exporter-ozforensics.yml

  - job_name: 'BlackBox_api_healthcheck_ozforensics'
    scrape_interval: 60s
    metrics_path: /probe
    scheme: http
    params:
      module: [api_healthcheck]
    relabel_configs:
     - source_labels: [__address__]
       target_label: __param_target
     - source_labels: [__param_target]
       target_label: instance
     - target_label: __address__
       replacement: blackbox_container:9115
    file_sd_configs:
      - files:
        - sd/blackbox-api-healthcheck.yml

Metrics Overview

Metric

Description

probe_success

Displays whether URL call has been finished successfully (Blackbox received an answer with required regular expression)

Grafana Alerts

For API:

groups:
  - name: API alerts
    rules:
      - alert: API is down
        expr: probe_success{job="blackbox-api-healthcheck"} != 1
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "OZ API has been down or showing errors for more than 30 seconds!"
          description: "Critical: OZ API service is down on the host {{ $labels.instance }}!"

For TFSS:

groups:
  - name: tfss alerts
    rules:
      - alert: model inquisitor_c error
        expr: probe_success{model="inquisitor_c"} != 1
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "The inquisitor_c model returns error!"
          description: "Critical: the TFSS service model returns error on the host {{ $labels.instance }}!"
      - alert: model dummy error
        expr: probe_success{model="dummy"} != 1
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "The dummy model returns error!"
          description: "Critical: the TFSS service model returns error on the host {{ $labels.instance }}!"

PostgreSQL

Prometheus Job

Instead of placeholder, put your container name.

 - job_name: "postgres"
    static_configs:
      - targets: ["postgres_container:9187"]
        labels:
          postgres: ozforensics
          ozforensics: true

Grafana Alerts

groups:
  - name: POSTGRESQL alerts
    rules:
      - alert: Postgresql is down
        expr: pg_up != 1
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "PostgreSQL has been down for more than 30 seconds!"
          description: "Critical: Postgres service is down on the host {{ $labels.instance }}!"
      - alert: Postgres exporter errors
        expr: pg_exporter_last_scrape_error == 1
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Postgres Exporter is down or showing errors"
          description: "Critical: postgres-exporter is not running or showing errors on the host {{ $labels.instance }}"

Other Alerts

up displays whether the last metrics collection went successfully: 1 if yes, 0 if no.

groups:
  - name: exporter alerts
    rules:
      - alert: Scrape error or instance is down
        expr: up != 1
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.job }} {{ $labels.host }} scrape error or instance is down!"
          description: "Critical: last scrape on {{ $labels.instance }} finished with error!"

Optional Prometheus Jobs

Node-exporter

  - job_name: "node_exporter"
    metrics_path: /node_exporter/metrics
    scheme: http
    file_sd_configs:
      - files:
        - sd/node-exporter.yml

The node-exporter.yml file (instead of placeholder, put your host address):

[
  {
    "targets": ["node:9113"],
    "labels":
      {
        "job": "node_exporter",
        "host": "api",
        "role": "oz-api",
        "ozforensics": "true",
      },
  },
]

cAdvisor

  - job_name: cadvisor
    metrics_path: /metrics
    file_sd_configs:
      - files:
        - sd/cadvisor.yml

The cadvisor.yml file (instead of placeholder, put your host address):

[
  {
    "targets": ["cadvisor_container:8081"],
    "labels":
      {
        "job": "cadvisor",
        "host": "api",
        "role": "oz-api",
        "ozforensics": "true",
      },
  },
]