> For the complete documentation index, see [llms.txt](https://doc.ozforensics.com/oz-knowledge/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://doc.ozforensics.com/oz-knowledge/guides/administrator-guide/monitoring/api-v6-and-above/docker.md). # Docker ## General Information ### Exporter Ports | Exporter | Port | | ---------- | -------------------------------- | | API | nginx\_api\_proxy:8081/metrics | | nginx | nginx\_container:9113/metrics | | TFSS | bio\_container:8501/metrics/tfss | | Blackbox | blackbox\_container:9115 | | PostgreSQL | postgres\_container:9187/metrics | You can find the already prepared exporter configuration in [our repository](https://gitlab.com/oz-forensics/public/oz-engineering-on-prem-docker-monitoring). After deployment, replace container names with actual addresses. For example, if your nginx container is at 192.168.8.8, change `- targets: ["nginx_container:9113"]` to `- targets: ["192.168.8.8:9113"]`. ### Healthcheck Paths **API** `GET http://api:8000/api/version` `GET http://api:8000/api/healthcheck` **BIO** `GET http://bio:8501/v1/models/{{model_name}}` `POST http://bio:8501/v1/models/dummy:predict` ```json method: POST body: '{"inputs": {"images_bytes": [{"b64": "aaa"}]}}' ``` *** ## API To enable API metrics, in the [docker/template/nginx.yml](https://gitlab.com/oz-forensics/docker-compose/api6/-/blob/main/docker/templates/nginx.yml) file, uncomment `9081:8081`: ```bash ports: - "9080:8080" - "9081:8081" ``` ### Prometheus Job {% code expandable="true" %} ```bash - job_name: "oz-api" scheme: http metrics_path: /metrics file_sd_configs: - files: - sd/oz-api.yml ``` {% endcode %} The `sd/oz-api.yml` file (instead of placeholders, put your container names and host address): {% code expandable="true" %} ```yml - targets: - api_container1:8081 labels: host: api1 role: oz-api ozforensics: api oz_service: "true" #- targets: # - api_container2:8081 # labels: # host: api1 # role: oz-api # ozforensics: api # oz_service: "true" ``` {% endcode %} ### Metrics Overview | Metric | Description | Type | Labels | | ----------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | --------- | -------------------------------------------------------------------- | | `oz_api_versions_total` (also available as `oz_api_versions_created`, `oz_api_info_created`, `oz_api_info_total`) | Displays API version info when you call `/api/version` | counter | `lamb` | | | | | `core` – core version | | | | | `oz_api` – API version | | `oz_api_analyses_total` | Number of processed analyses | counter | `analysis_result` – analysis status (`FINISHED`, `FAILED`) | | | | | `analysis_type` – analysis type (`QUALITY`, `BIOMETRY`, `DOCUMENTS`) | | | | | `company_id` – company identifier | | | | | `resolution_status` – analysis resolution (`SUCCESS`, `DECLINED`) | | `oz_api_analyses_created` | Number of analyses that have been initiated | gauge | `analysis_result` – analysis status (`FINISHED`, `FAILED`) | | | | | `analysis_type` – analysis type (`QUALITY`, `BIOMETRY`, `DOCUMENTS`) | | | | | `company_id` – company identifier | | | | | `resolution_status` – analysis resolution (`SUCCESS`, `DECLINED`) | | `oz_api_analyses_in_progress` | Number of analyses currently processing | gauge | `analysis_type` – analysis type (`QUALITY`, `BIOMETRY`, `DOCUMENTS`) | | | | | `company_id` – company identifier | | `oz_api_analyse_duration_seconds_bucket` | A histogram for analysis duration. Default buckets: `0.1,0.5,1,1.5,2,2.5,3,3.5,4,4.5,5,6,7,8,10,12,15,20,30,inf` | histogram | `analysis_result` – analysis status (`FINISHED`, `FAILED`) | | | | | `analysis_type` – analysis type (`QUALITY`, `BIOMETRY`, `DOCUMENTS`) | | | | | `resolution_status` – analysis resolution (`SUCCESS`, `DECLINED`) | | | | | `company_id` – company identifier | | | | | `le` – bucket size | | `oz_api_analyse_duration_seconds_count` | Sum of analyses' durations | counter | `analysis_result` – analysis status (`FINISHED`, `FAILED`) | | | | | `analysis_type` – analysis type (`QUALITY`, `BIOMETRY`, `DOCUMENTS`) | | | | | `resolution_status` – analysis status (`SUCCESS`, `DECLINED`) | | | | | `company_id` – company identifier | | `oz_api_analyse_duration_seconds_sum` | Number of analyses whose durations were counted | counter | `analysis_result` – analysis resolution (`FINISHED`, `FAILED`) | | | | | `analysis_type` – analysis type (`QUALITY`, `BIOMETRY`, `DOCUMENTS`) | | | | | `resolution_status` – analysis resolution (`SUCCESS`, `DECLINED`) | | | | | `company_id` – company identifier | | `oz_api_analyse_duration_seconds_created` | Timestamp of beginning of duration metric counting | | | | `django_http_requests_total_by_transport_total` | Number of requests by transport protocol | counter | `transport` – protocol (HTTP, HTTPS) | | | | | `uid` – user identifier | | `django_http_responses_before_middlewares_total` | Number of Django responses before running middleware | counter | `uid` – user identifier | | `django_http_exceptions_total_by_type_total` | Number of Django exceptions by type | counter | `exception_type` – type of exception | | | | | `uid` – user identifier | | `django_http_requests_latency_seconds_by_view_method_bucket` | Histogram of request processing latency by views | histogram | `method` – HTTP method | | | | | `uid` – user identifier | | | | | `view_name` – view name | | | | | `le` – bucket size | | `django_http_requests_latency_seconds_by_view_method_count` | Number of request processing durations by views | counter | `method` – HTTP method | | | | | `uid` – user identifier | | | | | `view_name` – view name | | `django_http_requests_latency_seconds_by_view_method_sum` | Sum of request processing durations by views | counter | `method` – HTTP method | | | | | `uid` – user identifier | | | | | `view_name` – view name | Illustrative screenshots: {% tabs %} {% tab title="Analyses rate per type" %}
{% endtab %} {% tab title="Analyses rate per resolution" %}
{% endtab %} {% tab title="Analyses rate per result" %}
{% endtab %} {% tab title="Analyses duration" %}

Analyses duration divided by average value / analysis type and quantile / analysis type. Discontinuity is caused by metric counter which starts for each analysis separately

{% endtab %} {% tab title="Django exceptions rate" %}
{% endtab %} {% endtabs %} ### Grafana Alerts ```bash groups: - name: API alerts rules: - alert: API is down expr: probe_success{job="blackbox-api-healthcheck"} != 1 for: 30s labels: severity: critical annotations: summary: "Oz API has been down or showing errors for more than 30 seconds!" description: "Critical: Oz API service is down on the host {{ $labels.instance }}!" - alert: Absent API metrics! expr: absent(oz_api_versions_total) for: 1m labels: severity: critical annotations: summary: "Absent API metrics!!!" description: "Critical: check API containers, API might be not working properly!!!" - alert: High total API analysis requests expr: sum (rate(oz_api_analyses_total{}[1m])) > 1 labels: severity: warning annotations: summary: "Too many API analysis requests per second!" description: "API analysis request rate is {{ $value }} rps." - alert: High failed API analysis rate expr: sum by (analysis_result)(rate(oz_api_analyses_total{analysis_result="FAILED"}[1m])) > 0 for: 1m labels: severity: warning annotations: summary: "Failed API analyses detected" description: "API analysis failure rate is {{ $value }} rps in the last minute." - alert: High API analysis duration expr: histogram_quantile(0.95,sum by (le, analysis_type) (rate(oz_api_analyse_duration_seconds_bucket{}[1m]))) > 7 for: 10m labels: severity: warning annotations: summary: "{{ $labels.analysis_type }} analyses are going slow!" description: "The duration of 95% {{ $labels.analysis_type }} analyses is longer than {{ $value }} seconds in the last 10 minutes." - alert: API exception errors expr: sum by (exception_type)(rate(django_http_exceptions_total_by_type_total{exception_type!~"AuthCredentialsExpired|AuthCredentialsInvalid|AuthCredentialsIsNotProvided|AuthForbidden|InvalidBodyStructureError|InvalidParamValueErrorNotExistError|OSError|InvalidParamValueError|NotExistError"}[1m])) > 0 for: 5m labels: severity: warning annotations: summary: "{{ $labels.exception_type }} in API exception errors" description: "{{ $value }} rps in API returned {{ $labels.exception_type }} in the last 5 minutes." ``` *** ## TFSS To enable metrics, set `TFSS_PROMETHEUS="true"` in [/configs/env/tfss.env](https://gitlab.com/oz-forensics/docker-compose/api6/-/blob/main/configs/env/tfss.env). To change the default metrics path, uncomment `TFSS_METRICS_URI="/metrics/tfss"` in [/configs/env/tfss.env](https://gitlab.com/oz-forensics/docker-compose/api6/-/blob/main/configs/env/tfss.env) and define the path. ### Prometheus Job Instead of placeholder, put your container name. {% code expandable="true" %} ```bash - job_name: "tfss" metrics_path: /metrics/tfss scheme: http static_configs: - targets: ["bio_container:8501"] labels: bio: ozforensics ozforensics: true ``` {% endcode %} ### Metrics Overview | Metric | Description | | ----------------------------------------------- | ----------------------------------------------------- | | `:tensorflow:serving:request_count` | Total number of requests to TFSS | | `:tensorflow:serving:request_latency_bucket` | Histogram of request processing time | | `:tensorflow:serving:request_latency_sum` | Sum of request processing durations | | `:tensorflow:serving:request_latency_count` | Total number of requests whose durations were counted | | `:tensorflow:cc:saved_model:load_attempt_count` | Loaded models | Illustrative screenshots: {% tabs %} {% tab title="Model request rate" %}

Total number of requests to TFSS per second and per minute, divided by models

{% endtab %} {% tab title="Model latency ($quantile-quantile)" %}

0.95 (or the one you've defined in $quantile) quantile of request processing duration in TFSS

{% endtab %} {% tab title="\[TFSS] Model latency (AVG)" %}

Average analysis processing time

{% endtab %} {% tab title="\[TFSS] HTTP probe status" %}

The built-in blackbox sends requests to check states of models, should be 1; if not, probe_success in TFSS is not working properly

{% endtab %} {% endtabs %} TFSS alerts are covered in the [Blackbox](#blackbox) section below. *** ## nginx ### Prometheus Job Instead of placeholder, put your container name. ```bash - job_name: "nginx" static_configs: - targets: ["nginx_container:9113"] labels: nginx: ozforensics ozforensics: true ``` ### Metrics Overview | Metric | Description | | ---------------------------- | ---------------------------------------------------------------------- | | `nginx_up` | Shows whether nginx is running: 1 – service is up, 0 – service is down | | `nginx_connections_accepted` | Number of connections accepted by nginx | | `nginx_connections_handled` | Number of connections handled by nginx | | `nginx_connections_active` | Active nginx connections | Illustrative screenshots: {% tabs %} {% tab title="Request rate" %}

Total number of requests to nginx

{% endtab %} {% tab title="Active connections" %}

State of nginx connections. Check that there are no stale connections

{% endtab %} {% tab title="Processed connections rate" %}

Processing success rate. The numbers of accepted and handled connections should be equal

{% endtab %} {% endtabs %} ### Grafana Alerts ```bash groups: - name: NGINX alerts rules: - alert: nginx is down expr: nginx_up != 1 for: 30s labels: severity: critical annotations: summary: "nginx has been down for more than 30 seconds!" description: "Critical: nginx service is down on the host {{ $labels.instance }}!" - alert: nginx not all connections are handled expr: rate (nginx_connections_handled[5m]) / rate (nginx_connections_accepted[5m]) <1 for: 2m labels: severity: critical annotations: summary: "nginx has issues with handling connections" description: "Critical: not all accepted connections have been handled by nginx on the host {{ $labels.instance }} for more than 3 minutes!" - alert: "High number of active connections in nginx" expr: nginx_connections_active > 3000 for: 3m labels: severity: critical annotations: summary: "High number of active connections in nginx" description: "Critical: nginx has had too many active connections on the host {{ $labels.instance }} for more than 3 minutes!" ``` *** ## Blackbox Blackbox probes verify that API and TFSS are responding correctly. You need to configure the modules, SD target files, and Prometheus jobs as shown below. ### Configuration {% code expandable="true" %} ```bash modules: tfss_models_probe: prober: http timeout: 5s http: valid_status_codes: [] # Defaults to 2xx method: GET fail_if_ssl: false fail_if_not_ssl: false fail_if_body_not_matches_regexp: - "\"error_code\": \"OK\"" - "\"state\": \"AVAILABLE\"" tls_config: insecure_skip_verify: false preferred_ip_protocol: "ip4" tfss_predict_probe: prober: http timeout: 5s http: valid_status_codes: [] # Defaults to 2xx method: POST body: '{"inputs": {"images_bytes": [{"b64": "aaa"}]}}' follow_redirects: true fail_if_ssl: false fail_if_not_ssl: false fail_if_body_not_matches_regexp: - '\"outputs\": 1\.0' tls_config: insecure_skip_verify: false preferred_ip_protocol: "ip4" http_2xx: prober: http timeout: 5s http: valid_status_codes: [] method: GET preferred_ip_protocol: "ip4" tls_config: insecure_skip_verify: false api_healthcheck: prober: http timeout: 5s http: method: GET valid_status_codes: [] fail_if_body_not_matches_regexp: - '{"generally_healthy": true,(| "celery": (\{\}|\{(?:\s*"[^"]+": "ok",)* "[^"]+": "ok"\s*\}),) "db": "ok"(|, "jwt": {"status": "ok"})}' ``` {% endcode %}
SD Files Configuration for Blackbox (instead of placeholders, put your container names) `blackbox_tfss_models.yml`: {% code expandable="true" %} ```yml [ { "targets": ["http://bio_container:8501/v1/models/inquisitor_c"], "labels": { "host": "bio:8501", "role": "oz-bio", "ssl_alert": "false", "http_code": "200", "model": "inquisitor_c", "ozforensics": "true", }, }, ] ``` {% endcode %} `blackbox_tfss_probe.yml`: {% code expandable="true" %} ```yml [ { "targets": ["http://bio_container:8501/v1/models/dummy:predict"], "labels": { "host": "bio:8501", "role": "oz-bio-balancer", "ssl_alert": "false", "http_code": "200", "model": "dummy", "ozforensics": "true", }, }, ] ``` {% endcode %} `blackbox-exporter-ozforensics.yml`: {% code expandable="true" %} ```yml [ { "targets": ["http://api_container:8000/api/version"], "labels": { "host": "blackbox_container:9115", "role": "API", "ssl_alert": "false", "http_code": "200", "ozforensics": "true", }, }, ] ``` {% endcode %} `blackbox-api-healthcheck.yml`: {% code expandable="true" %} ```yml [ { "targets": ["http://api_container:8000/api/healthcheck"], "labels": { "job": "blackbox-api-healthcheck", "host": "api:8000/api/healthcheck", "role": "oz-api", "ssl_alert": "false", "http_code": "200", "ozforensics": "true", }, }, ] ``` {% endcode %}
### Prometheus Job Instead of placeholders, put your container names and host addresses. {% code expandable="true" %} ```bash - job_name: "BlackBox_tfss_models" scrape_interval: 30s metrics_path: /probe scheme: http params: module: [tfss_models_probe] relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox_container:9115 file_sd_configs: - files: - sd/BlackBox_tfss_models.yml - job_name: "BlackBox_tfss_probe" scrape_interval: 30s metrics_path: /probe scheme: http params: module: [tfss_predict_probe] relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox_container:9115 file_sd_configs: - files: - sd/BlackBox_tfss_probe.yml - job_name: 'BlackBox_exporter_ozforensics' scrape_interval: 60s metrics_path: /probe params: module: [http_2xx] relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox_container:9115 file_sd_configs: - files: - sd/blackbox-exporter-ozforensics.yml - job_name: 'BlackBox_api_healthcheck_ozforensics' scrape_interval: 60s metrics_path: /probe scheme: http params: module: [api_healthcheck] relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox_container:9115 file_sd_configs: - files: - sd/blackbox-api-healthcheck.yml ``` {% endcode %} ### Metrics Overview | Metric | Description | | --------------- | ----------------------------------------------------------------------------------------------------------------------- | | `probe_success` | Displays whether URL call has been finished successfully (Blackbox received an answer with required regular expression) | ### Grafana Alerts For API: ```bash groups: - name: API alerts rules: - alert: API is down expr: probe_success{job="blackbox-api-healthcheck"} != 1 for: 30s labels: severity: critical annotations: summary: "OZ API has been down or showing errors for more than 30 seconds!" description: "Critical: OZ API service is down on the host {{ $labels.instance }}!" ``` For TFSS: ```bash groups: - name: tfss alerts rules: - alert: model inquisitor_c error expr: probe_success{model="inquisitor_c"} != 1 for: 30s labels: severity: critical annotations: summary: "The inquisitor_c model returns error!" description: "Critical: the TFSS service model returns error on the host {{ $labels.instance }}!" - alert: model dummy error expr: probe_success{model="dummy"} != 1 for: 30s labels: severity: critical annotations: summary: "The dummy model returns error!" description: "Critical: the TFSS service model returns error on the host {{ $labels.instance }}!" ``` *** ## PostgreSQL ### Prometheus Job Instead of placeholder, put your container name. {% code expandable="true" %} ```bash - job_name: "postgres" static_configs: - targets: ["postgres_container:9187"] labels: postgres: ozforensics ozforensics: true ``` {% endcode %} ### Grafana Alerts ```bash groups: - name: POSTGRESQL alerts rules: - alert: Postgresql is down expr: pg_up != 1 for: 30s labels: severity: critical annotations: summary: "PostgreSQL has been down for more than 30 seconds!" description: "Critical: Postgres service is down on the host {{ $labels.instance }}!" - alert: Postgres exporter errors expr: pg_exporter_last_scrape_error == 1 for: 10m labels: severity: critical annotations: summary: "Postgres Exporter is down or showing errors" description: "Critical: postgres-exporter is not running or showing errors on the host {{ $labels.instance }}" ``` *** ## Other Alerts `up` displays whether the last metrics collection went successfully: 1 if yes, 0 if no. ```bash groups: - name: exporter alerts rules: - alert: Scrape error or instance is down expr: up != 1 for: 30s labels: severity: critical annotations: summary: "{{ $labels.job }} {{ $labels.host }} scrape error or instance is down!" description: "Critical: last scrape on {{ $labels.instance }} finished with error!" ``` *** ## Optional Prometheus Jobs ### Node-exporter ```bash - job_name: "node_exporter" metrics_path: /node_exporter/metrics scheme: http file_sd_configs: - files: - sd/node-exporter.yml ``` The `node-exporter.yml` file (instead of placeholder, put your host address): ```yml [ { "targets": ["node:9113"], "labels": { "job": "node_exporter", "host": "api", "role": "oz-api", "ozforensics": "true", }, }, ] ``` ### cAdvisor ```bash - job_name: cadvisor metrics_path: /metrics file_sd_configs: - files: - sd/cadvisor.yml ``` The `cadvisor.yml` file (instead of placeholder, put your host address): ```yml [ { "targets": ["cadvisor_container:8081"], "labels": { "job": "cadvisor", "host": "api", "role": "oz-api", "ozforensics": "true", }, }, ] ``` *** ## Telegram Alerting Make sure you replace all placeholders. {% file src="/files/lk8knqBBkhMqLNJ396xE" %} Then, in `./alertmanager`, create `templates/` and add the template file. {% file src="/files/29YHPJRTGMS9AhGPa2A8" %} *** API 6 no longer requires Celery or Redis, so the corresponding metrics are not covered in this guide. You can customize the alerts according to your needs. Please proceed to [our repository](https://gitlab.com/oz-forensics/public/oz-engineering-monitoring-configs/-/tree/helm-0.10.20/alerts?ref_type=heads) to find the alert files. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://doc.ozforensics.com/oz-knowledge/guides/administrator-guide/monitoring/api-v6-and-above/docker.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.