Terms and Definitions

Components' Description

Oz API components:

• APP is the API front app that receives REST requests, performs preprocessing, and creates tasks for other API components.

• Celery is the asynchronous task queue. API has 6 celery queues:

  • Celery-default processes system-wide tasks.

  • Celery-maintenance processes maintenance tasks.

  • Celery-tfss processes analysis tasks.

  • Celery-resolution checks for completion of all nested analyses within a folder and changes folder status.

  • Celery-preview_convert creates a video preview for media.

  • Celery-beat is a CronJob for managing maintenance celery tasks.

  • Celery-Flower is a Celery metrics collector.

  • Celery-regula (optional) processes document analysis tasks.

• Redis is a message broker and result backend for Celery.

• RabbitMQ (optional) can be used as a message broker for Celery instead of Redis.

• Nginx serves static media files for external HTTP(s) requests.

• O2N (optional) processes the Blacklist analysis.

• Statistic (optional) provides statistics' collection for Web UI.

• Web UI provides the web interface.

BIO-Updater checks for models updates and downloads new models.

Oz BIO (TFSS) runs TensorFlow with AI models and makes decisions for incoming media.

Deployment Scenarios

The deployment scenario depends on the workload you expect.

Autoscaling is implemented on the basis of ClusterAutoscaler and must be supported by your infrastructure.

Small business or PoC

Please find the installation guide here: Docker.

• Type of containerization: Docker,

• Type of installation: Docker compose,

• Autoscaling/HA: none.

Requirements

Software

• Docker 19.03+,

• Podman 4.4+,

• Python 3.4+.

Storage

• Depends on image quality and required archive depth.

• May be count as: [average image size] * 2 * [analyses per day] * [archive depth in days].

Staff qualification:

• Basic knowledge of Linux and Docker.

Deployment

1. Single node.

Resources:

• 1 node,

• 16 CPU/32 RAM.

1. Two nodes.

Resources:

• 2 nodes,

• 16 CPU/32 RAM for the first node; 8 CPU/16 RAM for the second node.

Medium Load

Please find the installation guide here: Docker.

• Type of containerization: Docker/Podman,

• Type of installation: Docker compose,

• Autoscaling/HA: manual scaling; HA is partially supported.

Requirements

Computational resources

Depending on load, you can change the number of nodes. However, for 5+ nodes, we recommend that you proceed to the High Load section.

• From 2 to 4 Docker nodes (see schemes):

  • 2 Nodes:

    • 24 CPU/32 RAM per node.

  • 3 Nodes:

    • 16 CPU/24 RAM per node.

  • 4 Nodes:

    • 8 CPU/16 RAM for two nodes (each),

    • 16 CPU/24 RAM for two nodes (each).

We recommend using external self-managed PostgreSQL database and NFS share.

Software

• Docker 19.03+,

• Podman 4.4+,

• Python 3.4+.

Storage

• Depends on image quality and required archive depth.

• May be count as: [average image size] * 2 * [analyses per day] * [archive depth in days].

Staff qualification:

• Advanced knowledge of Linux, Docker, and Postgres.

Deployment

2 nodes:

3 nodes:

4 nodes:

High Load

Please find the installation guide here: Kubernetes.

• Type of containerization: Type of containerization: Docker containers with Kubernetes orchestration,

• Type of installation: Helm charts,

• Autoscaling/HA: supports autoscaling; HA for most components.

Requirements

Computational resources

3-4 nodes. Depending on load, you can change the number of nodes.

• 16 CPU/32 RAM Nodes for the BIO pods,

• 8+ CPU/16+ RAM Nodes for all other workload.

We recommend using external self-managed PostgreSQL database.

Software

• Docker 19.03+,

• Python 3.4+.

Storage

• Depends on image quality and required archive depth.

• May be count as: [average image size] * 2 * [analyses per day] * [archive depth in days].

Staff qualification:

• Advanced knowledge of Linux, Docker, Kubernetes, and Postgres.

Deployment Scheme

By default, all API methods are published without restrictions, that may possess security threats. For accessing API methods from Internet, we recommend enabling limitations on WAF, border L7 balancer, etc.

API

The information below is relevant for Oz API 5.2.

For Oz API with Mobile SDK, make sure these methods are accessible from the Internet:

You may need to extend this list depending on how Oz API has been integrated into your infrastructure.

Web SDK

For Web SDK, make sure these methods are accessible from the Internet. Your Web SDK URL is the Web Adapter URL you have received from us.

Hardware and Software Requirements

To launch the services, you'll require:

• CPU: 16 cores,

• RAM: 32 GB,

• Disk: 100 GB, SSD,

• Linux-compatible OS,

• Docker 19.03+ (or Podman 4.4+),

• Docker Compose 1.27+ (or podman-compose 1.2.0+, if you use Podman).

Distribution Package Contents

The package you get consists of the following directories and files:

Installation

Installing TFSS and the API on the Same Host

1. Put the license file in ./configs/tfss/license.key.

2. Unzip the file that contains models into the ./data/tfss/models directory.

3. Before starting system configuration, we recommend running the host readiness check scripts. Navigate to the checkers directory and run the pre-checker-all.sh script.

1. Set the initial passwords and values:

1. For this configuration, run all services on a single host:

Installing TFSS and the API on Separate Hosts

TFSS Host

1. Create a directory and unzip the distribution package into it. The package contains Docker Compose manifests and directories with the configuration files required for operation.

2. Put the license file in ./configs/tfss/license.key.

3. Unzip the file that contains models into the ./data/tfss/models directory.

4. Before starting system configuration, we recommend running the host readiness check scripts. Navigate to the checkers directory and run the pre-checker-all.sh script.
5. For this configuration, run TFSS service on a separate host:

API Host

1. Create a directory and unzip the distribution package into it. The package contains Docker Compose manifests and directories with the configuration files required for operation.

2. Before starting system configuration, we recommend running the host readiness check scripts. Navigate to the checkers directory and run the pre-checker-all.sh script.

2. For this configuration, run all services on a single host:

To install Oz product via Kubernetes, consider using Helm charts.

• Oz API and related components: Helm chart.

  • API 5.2: version 0.11.x,

  • API 5.3 (regulatory update for Kazakhstan): 0.12.x.

• Web SDK: Helm chart. Please note: the version of the chart is not tied to the Web SDK version.

Database Creation

For testing purposes, the database installed and created automatically by the chart is sufficient. However, for production, we strongly recommend using a separate, self-managed database.

Recommended PostgreSQL version: 15.5.

Create a database using the script(s) below.

    CREATE ROLE <<USERNAME>> WITH LOGIN PASSWORD '<<PASSWORD>>'
      INHERIT
      CONNECTION LIMIT -1
      NOSUPERUSER
      NOCREATEDB
      NOCREATEROLE
      NOREPLICATION ;
    CREATE DATABASE <<DB_NAME>>
      WITH OWNER = <<USERNAME>>
          ENCODING = 'UTF8'
          TABLESPACE = pg_default
          CONNECTION LIMIT = -1
          LC_COLLATE='en_US.UTF-8'
          LC_CTYPE='en_US.UTF-8'
          TEMPLATE template0;
    \c <<DB_NAME>>
    CREATE EXTENSION IF NOT EXISTS pgcrypto;

    CREATE ROLE <<O2N_USERNAME>> WITH LOGIN PASSWORD '<<O2N_PASSWORD>>'
      INHERIT
      CONNECTION LIMIT -1
      NOSUPERUSER
      NOCREATEDB
      NOCREATEROLE
      NOREPLICATION ;
    CREATE DATABASE <<O2N_DB_NAME>>
      WITH OWNER = <<O2N_USERNAME>>
          ENCODING = 'UTF8'
          TABLESPACE = pg_default
          CONNECTION LIMIT = -1
          LC_COLLATE='en_US.UTF-8'
          LC_CTYPE='en_US.UTF-8'
          TEMPLATE template0;
    \c <<O2N_DB_NAME>>
    CREATE EXTENSION IF NOT EXISTS pgcrypto;
    CREATE EXTENSION IF NOT EXISTS vector;

To increase performance, consider using this list of indexes:

CREATE INDEX CONCURRENTLY gw_analyse_abstract_company_id_idx ON public.gw_analyse_abstract USING btree (company_id);
CREATE INDEX CONCURRENTLY gw_analyse_abstract_folder_id ON public.gw_analyse_abstract USING btree (folder_id);
CREATE INDEX CONCURRENTLY gw_analyse_abstract_time_updated_state_idx ON public.gw_analyse_abstract USING btree (time_updated) WHERE (state = 'PROCESSING'::analyse_state);
CREATE INDEX CONCURRENTLY idx_gw_analyse_abstract_state_time_updated ON public.gw_analyse_abstract USING btree (state, time_updated);
CREATE INDEX CONCURRENTLY gw_analyse_collection_collection_id_idx ON public.gw_analyse_collection USING btree (collection_id);
CREATE INDEX CONCURRENTLY gw_analyse_collection_person_media_association_person_id_idx ON public.gw_analyse_collection_person_media_association USING btree (person_id);
CREATE INDEX CONCURRENTLY gw_analyse_cpma_media_association_id_idx ON public.gw_analyse_collection_person_media_association USING btree (media_association_id);
CREATE INDEX CONCURRENTLY gw_analyse_result_group_image_analyse_id_idx ON public.gw_analyse_result_group_image USING btree (analyse_id);
CREATE INDEX CONCURRENTLY gw_analyse_result_group_image_image_id_idx ON public.gw_analyse_result_group_image USING btree (image_id);
CREATE INDEX CONCURRENTLY gw_analyse_result_image_media_association_id_idx ON public.gw_analyse_result_image USING btree (media_association_id);   
CREATE INDEX CONCURRENTLY gw_analyse_source_media_association_analyse_id_idx ON public.gw_analyse_source_media_association USING btree (analyse_id);
CREATE INDEX CONCURRENTLY gw_analyse_source_media_association_source_image_id_idx ON public.gw_analyse_source_media_association USING btree (source_image_id);   
CREATE INDEX CONCURRENTLY gw_analyse_source_media_association_source_shots_set_id_idx ON public.gw_analyse_source_media_association USING btree (source_shots_set_id);
CREATE INDEX CONCURRENTLY gw_analyse_source_media_association_source_video_id_idx ON public.gw_analyse_source_media_association USING btree (source_video_id);
CREATE INDEX CONCURRENTLY gw_collection_company_id_idx ON public.gw_collection USING btree (company_id);
CREATE INDEX CONCURRENTLY gw_collection_creator_id_idx ON public.gw_collection USING btree (creator_id);
CREATE INDEX CONCURRENTLY gw_collection_person_collection_id_idx ON public.gw_collection_person USING btree (collection_id);
CREATE INDEX CONCURRENTLY gw_collection_person_image_person_id_idx ON public.gw_collection_person_image USING btree (person_id);
CREATE INDEX CONCURRENTLY gw_folder_company_id_idx ON public.gw_folder USING btree (company_id) WHERE (NOT is_archive);
CREATE INDEX CONCURRENTLY gw_folder_event_session_id_idx ON public.gw_folder USING btree (((meta_data ->> 'event_session_id'::text))) WHERE (NOT is_archive);
CREATE INDEX CONCURRENTLY gw_folder_resolution_author_id_idx ON public.gw_folder USING btree (resolution_author_id);
CREATE INDEX CONCURRENTLY gw_folder_resolution_comment_fts_idx ON public.gw_folder USING gin (to_tsvector('english'::regconfig, (resolution_comment)::text));
CREATE INDEX CONCURRENTLY gw_folder_time_created_folder_id_idx ON public.gw_folder USING btree (time_created DESC, folder_id DESC);
CREATE INDEX CONCURRENTLY gw_folder_time_created_folder_id_nonarchive_idx ON public.gw_folder USING btree (time_created DESC, folder_id DESC) WHERE (is_archive = false);
CREATE INDEX CONCURRENTLY gw_folder_time_created_idx ON public.gw_folder USING btree (time_created DESC);
CREATE INDEX CONCURRENTLY gw_folder_time_created_nonarchive_idx ON public.gw_folder USING btree (time_created DESC) WHERE (is_archive = false);
CREATE INDEX CONCURRENTLY gw_folder_user_id_idx ON public.gw_folder USING btree (user_id);
CREATE INDEX CONCURRENTLY idx_gw_folder_transaction_id ON public.gw_folder USING btree (company_id, ((meta_data ->> 'transaction_id'::text))) WHERE (NOT is_archive);
CREATE INDEX CONCURRENTLY idx_gw_folder_user_id_time_created ON public.gw_folder USING btree (user_id, time_created);
CREATE INDEX CONCURRENTLY gw_folder_image_folder_id_idx ON public.gw_folder_image USING btree (folder_id);
CREATE INDEX CONCURRENTLY gw_folder_report_author_id_idx ON public.gw_folder_report USING btree (author_id);
CREATE INDEX CONCURRENTLY gw_folder_report_folder_id_idx ON public.gw_folder_report USING btree (folder_id);
CREATE INDEX CONCURRENTLY gw_folder_report_report_template_id_idx ON public.gw_folder_report USING btree (report_template_id);
CREATE INDEX CONCURRENTLY idx_gw_folder_shots_set_folder_id ON public.gw_folder_shots_set USING btree (folder_id);
CREATE INDEX CONCURRENTLY gw_folder_video_folder_id_idx ON public.gw_folder_video USING btree (folder_id);
CREATE INDEX CONCURRENTLY gw_logging_event_record_session_id_idx ON public.gw_logging_event_record USING btree (session_id);
CREATE INDEX CONCURRENTLY gw_logging_event_record_time_created_idx ON public.gw_logging_event_record USING btree (time_created);
CREATE INDEX CONCURRENTLY gw_logging_event_record_timemark_idx ON public.gw_logging_event_record USING btree (timemark);
CREATE INDEX CONCURRENTLY gw_logging_event_session_time_created_idx ON public.gw_logging_event_session USING btree (time_created);
CREATE INDEX CONCURRENTLY gw_logging_event_session_user_id_idx ON public.gw_logging_event_session USING btree (user_id);
CREATE INDEX CONCURRENTLY gw_logging_event_session_time_updated_idx ON gw_logging_event_session(time_updated);
CREATE INDEX CONCURRENTLY gw_report_template_company_id_idx ON public.gw_report_template USING btree (company_id);
CREATE INDEX CONCURRENTLY gw_report_template_name_company_id_idx ON public.gw_report_template USING btree (name, company_id);
CREATE INDEX CONCURRENTLY gw_report_template_attachment_filename_report_template_id_idx ON public.gw_report_template_attachment USING btree (filename, report_template_id);
CREATE INDEX CONCURRENTLY gw_report_template_attachment_report_template_id_idx ON public.gw_report_template_attachment USING btree (report_template_id);
CREATE INDEX CONCURRENTLY gw_role_abstract_user_company_id_idx ON public.gw_role_abstract_user USING btree (company_id);
CREATE INDEX CONCURRENTLY gw_role_email_restore_code_user_id_idx ON public.gw_role_email_restore_code USING btree (user_id);
CREATE INDEX CONCURRENTLY gw_role_session_old_access_token_idx ON public.gw_role_session USING btree (old_access_token);
CREATE INDEX CONCURRENTLY gw_role_session_user_id_idx ON public.gw_role_session USING btree (user_id);
CREATE INDEX CONCURRENTLY gw_role_user_photo_user_id_idx ON public.gw_role_user_photo USING btree (user_id);
CREATE INDEX CONCURRENTLY gw_shots_set_frame_shots_set_id_idx ON public.gw_shots_set_frame USING btree (shots_set_id);
CREATE INDEX CONCURRENTLY gw_utils_audit_actor_id_idx ON public.gw_utils_audit USING btree (actor_id);
CREATE INDEX CONCURRENTLY gw_utils_media_company_id_idx ON public.gw_utils_media USING btree (company_id);

Chart Deployment

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc-api
provisioner: efs.csi.aws.com
allowVolumeExpansion: true
#Use Retain if you are going to use PVC for static, or your static will be deleted on chart uninstall
reclaimPolicy: Retain 
mountOptions:
  - tls
  - iam
parameters:
  provisioningMode: efs-ap
  fileSystemId: fs-____________
  directoryPerms: "770"
  uid: "1000"
  gid: "1000"
  basePath: ""

To deploy in Kubernetes, download the chart version you require and adjust the values.yaml file. This file specifies parameters for deployment of Oz products.

API

Adjust the values.yaml file, setting the following mandatory parameters before deployment:

• ozDockerHubCreds: you'll receive them from Oz Engineer.

• UserParams:

  • URLs:

    • apiURL: URL for API. May be internal, if you use Web SDK only. For Mobile SDKs, should be public. Please refer to this article for more information.

  • DB: must be set, if you use an external PostgreSQL server. For details, please check Database Creation.

    • use_chart_postgres: false by default. Enables internal PostgreSQL server (not recommended for production).

    • postgresUser: same as <<USERNAME>>.

    • postgresHost: the hostname of your PostgreSQL server.

    • postgresDB: same as <<DB_NAME>>.

    • postgresUserPassword: same as <<PASSWORD>>.

    • postgresPort: 5432 by default.

    • o2nDB:

      • use_chart_o2nDB: false by default. Enables internal PostgreSQL server (not recommended for production).

      • startinit: true by default. Enables database init scripts. Set to false after chart is deployed.

      • creds:

        • postgresHost: the hostname of your PostgreSQL server with O2N database.

        • postgresPort: 5432 by default.

        • postgresDB: same as <<O2N_DB_NAME>>.

        • postgresUser: same as <<O2N_USERNAME>>.

        • postgresUserPassword: Same as <<O2N_PASSWORD>>.

  • Creds:

    • apiAdminLogin: login for new (default) user for API. Will be created on the first run.

    • apiAdminPass: password for the default user.

    • webUILocalAdminLogin: local Admin for Web UI. Should differ from apiAdminLogin.

    • webUILocalAdminPass: password for webUILocalAdminLogin.

  • BIO:

    • licenseKey: you'll receive it from Oz Engineer / Sales.

    • clientToken: you'll receive it from Oz Engineer.

  • pvc:

    • api:

      • static:

        • storageClassName: RWX StorageClass.

        • size: Expected size for PV.

• Params:

  • Global:

    • startinits: false by default. Set to true on the first run, then, after successful deployment, change back to false.

To adjust API behavior, you might want to change other parameters. Please refer to comments in the values.yaml file.

TFSS (BIO)

BIO is a part of the API chart. The BIO pods require separate nodes for each pod. To ensure BIO resides on dedicated nodes, you can use affinity and tolerations.

The BIO behavior can be customized via Params -> global -> affinity in values.yaml.

The default parameters are listed below:

# We recommend using separate nodes for API and BIO pods,
# and separate node for each BIO pod.
 ## Default affinity is:
 ## Try to start API pods on nodes with "oz:api" label, BIO pods on nodes with "oz:bio" label
affinity: 
  API:
  # You may add additional keys, or comment the whole section to disable affinity
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        preference:
          matchExpressions:
          - key: oz
            operator: In
            values:
            - api
    # nodeName: "myNode" #Node name can be changed. Uncomment to use

    # nodeSelector: ##NodeSelector can be changed. Uncomment to use
    #   oz: api

    # fill tolerations according to node taints, uncomment to use
    #tolerations:
    #- key: nodegroup
    #  operator: Equal
    #  value: api
    #  effect: NoSchedule

  BIO:
  # You may add additional keys, or comment the whole section to disable affinity
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        preference:
          matchExpressions:
          - key: oz
            operator: In
            values:
            - bio

    # nodeName: "myNode" ## Node name can be changed. Uncomment to use.

    # nodeSelector: ## NodeSelector can be changed. Uncomment to use.
    #   oz: bio

    # fill tolerations according to node taints, uncomment to use.
    #tolerations:
    #- key: nodegroup
    #  operator: Equal
    #  value: bio
    #  effect: NoSchedule

The example of chart deployment via Helm:

helm install oz-api https://chartmuseum.infra.ozforensics.ai/charts/oz-k8s-0.11.28.tgz --namespace oz-api --values custom-api-values.yaml

Web SDK

For proper deployment, Web SDK requires an API service account. Pre-create a user for Web SDK with the CLIENT type and is_service flag set. Please refer to User Roles for more details.

Adjust the values.yaml file, setting the following mandatory parameters before deployment:

• ozDockerHubCreds: you'll receive them from Oz Engineer.

• UserParams:

  • URLs:

    • apiURL: API URL. Can be an internal API URL.

    • webSDKURL: WebSDK url that will be used for public access.

  • Creds:

    • AdminLogin: login of the user that should be pre-created in API. Do not use the default admin login.

    • AdminPass: password of the pre-created user.

• PVC:

  • persistentStorage: false be default. Set to true if you use more than 1 Web SDK pod.

  • storageClassName: RWX StorageClass.

• Params:

  • websdk:

    • license: should contain your Web SDK license. You'll receive it from Oz Engineer / Sales.

# Format:
# Remark: Indentations are critical for yaml.
    license: |-
      {
          "payload_b64": "",
          "signature": "",
          "enc_public_key": ""
      }

To adjust API behavior, you might want to change other parameters. Please refer to comments in the values.yaml file.

The example of chart deployment via Helm:

helm install oz-websdk https://chartmuseum.infra.ozforensics.ai/charts/oz-k8s-sdk-1.5.1+onPremise.tgz --namespace oz-websdk --values custom-websdk-values.yaml