Docker
General Information
Exporter Ports
API
nginx_api_proxy:8081/metrics
nginx
nginx_container:9113/metrics
TFSS
bio_container:8501/metrics/tfss
Blackbox
blackbox_container:9115
PostgreSQL
postgres_container:9187/metrics
You can find the already prepared exporter configuration in our repository. After deployment, replace container names with actual addresses. For example, if your nginx container is at 192.168.8.8, change - targets: ["nginx_container:9113"] to - targets: ["192.168.8.8:9113"].
Healthcheck Paths
API
GET http://api:8000/api/version
GET http://api:8000/api/healthcheck
BIO
GET http://bio:8501/v1/models/{{model_name}}
POST http://bio:8501/v1/models/dummy:predict
method: POST
body: '{"inputs": {"images_bytes": [{"b64": "aaa"}]}}'API
To enable API metrics, in the docker/template/nginx.yml file, uncomment 9081:8081:
Prometheus Job
The sd/oz-api.yml file (instead of placeholders, put your container names and host address):
Metrics Overview
oz_api_versions_total (also available as oz_api_versions_created, oz_api_info_created, oz_api_info_total)
Displays API version info when you call /api/version
counter
lamb
core – core version
oz_api – API version
oz_api_analyses_total
Number of processed analyses
counter
analysis_result – analysis status (FINISHED, FAILED)
analysis_type – analysis type (QUALITY, BIOMETRY, DOCUMENTS)
company_id – company identifier
resolution_status – analysis resolution (SUCCESS, DECLINED)
oz_api_analyses_created
Number of analyses that have been initiated
gauge
analysis_result – analysis status (FINISHED, FAILED)
analysis_type – analysis type (QUALITY, BIOMETRY, DOCUMENTS)
company_id – company identifier
resolution_status – analysis resolution (SUCCESS, DECLINED)
oz_api_analyses_in_progress
Number of analyses currently processing
gauge
analysis_type – analysis type (QUALITY, BIOMETRY, DOCUMENTS)
company_id – company identifier
oz_api_analyse_duration_seconds_bucket
A histogram for analysis duration. Default buckets: 0.1,0.5,1,1.5,2,2.5,3,3.5,4,4.5,5,6,7,8,10,12,15,20,30,inf
histogram
analysis_result – analysis status (FINISHED, FAILED)
analysis_type – analysis type (QUALITY, BIOMETRY, DOCUMENTS)
resolution_status – analysis resolution (SUCCESS, DECLINED)
company_id – company identifier
le – bucket size
oz_api_analyse_duration_seconds_count
Sum of analyses' durations
counter
analysis_result – analysis status (FINISHED, FAILED)
analysis_type – analysis type (QUALITY, BIOMETRY, DOCUMENTS)
resolution_status – analysis status (SUCCESS, DECLINED)
company_id – company identifier
oz_api_analyse_duration_seconds_sum
Number of analyses whose durations were counted
counter
analysis_result – analysis resolution (FINISHED, FAILED)
analysis_type – analysis type (QUALITY, BIOMETRY, DOCUMENTS)
resolution_status – analysis resolution (SUCCESS, DECLINED)
company_id – company identifier
oz_api_analyse_duration_seconds_created
Timestamp of beginning of duration metric counting
django_http_requests_total_by_transport_total
Number of requests by transport protocol
counter
transport – protocol (HTTP, HTTPS)
uid – user identifier
django_http_responses_before_middlewares_total
Number of Django responses before running middleware
counter
uid – user identifier
django_http_exceptions_total_by_type_total
Number of Django exceptions by type
counter
exception_type – type of exception
uid – user identifier
django_http_requests_latency_seconds_by_view_method_bucket
Histogram of request processing latency by views
histogram
method – HTTP method
uid – user identifier
view_name – view name
le – bucket size
django_http_requests_latency_seconds_by_view_method_count
Number of request processing durations by views
counter
method – HTTP method
uid – user identifier
view_name – view name
django_http_requests_latency_seconds_by_view_method_sum
Sum of request processing durations by views
counter
method – HTTP method
uid – user identifier
view_name – view name
Illustrative screenshots:





Grafana Alerts
TFSS
To enable metrics, set TFSS_PROMETHEUS="true" in /configs/env/tfss.env.
To change the default metrics path, uncomment TFSS_METRICS_URI="/metrics/tfss" in /configs/env/tfss.env and define the path.
Prometheus Job
Instead of placeholder, put your container name.
Metrics Overview
:tensorflow:serving:request_count
Total number of requests to TFSS
:tensorflow:serving:request_latency_bucket
Histogram of request processing time
:tensorflow:serving:request_latency_sum
Sum of request processing durations
:tensorflow:serving:request_latency_count
Total number of requests whose durations were counted
:tensorflow:cc:saved_model:load_attempt_count
Loaded models
Illustrative screenshots:




probe_success in TFSS is not working properlyTFSS alerts are covered in the Blackbox section below.
nginx
Prometheus Job
Instead of placeholder, put your container name.
Metrics Overview
nginx_up
Shows whether nginx is running: 1 – service is up, 0 – service is down
nginx_connections_accepted
Number of connections accepted by nginx
nginx_connections_handled
Number of connections handled by nginx
nginx_connections_active
Active nginx connections
Illustrative screenshots:



Grafana Alerts
Blackbox
Blackbox probes verify that API and TFSS are responding correctly. You need to configure the modules, SD target files, and Prometheus jobs as shown below.
Configuration
Prometheus Job
Instead of placeholders, put your container names and host addresses.
Metrics Overview
probe_success
Displays whether URL call has been finished successfully (Blackbox received an answer with required regular expression)
Grafana Alerts
For API:
For TFSS:
PostgreSQL
Prometheus Job
Instead of placeholder, put your container name.
Grafana Alerts
Other Alerts
up displays whether the last metrics collection went successfully: 1 if yes, 0 if no.
Optional Prometheus Jobs
Node-exporter
The node-exporter.yml file (instead of placeholder, put your host address):
cAdvisor
The cadvisor.yml file (instead of placeholder, put your host address):
Telegram Alerting
Make sure you replace all placeholders.
Then, in ./alertmanager, create templates/ and add the template file.
API 6 no longer requires Celery or Redis, so the corresponding metrics are not covered in this guide.
You can customize the alerts according to your needs. Please proceed to our repository to find the alert files.
Last updated
Was this helpful?
