ml-models-triton-inference-server-oke

Model serving is the process of deploying a machine learning (ML) model into production, so that it can make predictions on new, unseen data. It involves loading the trained model from a model repository, deploying it to a server and providing an API endpoint for inference service. Model serving is a critical step in the ML pipeline because it enables organizations to operationalize their ML models to solve real-world problems.

When designing the serving architecture, consider the following major concepts:

• Scalability: Scale model serving capability to handle bursts of prediction traffic efficiently.
  
  
• Flexibility: Serve multiple models with different ML frameworks and switch between models easily.
  
  
• Performance: Infer with low latency for real-time applications and high throughput for offline applications.
  
  

Oracle Container Engine for Kubernetes

Oracle Container Engine for Kubernetes (OKE) is a managed service that enables you to run Kubernetes clusters on Oracle Cloud Infrastructure (OCI), allowing containerized applications to be run at scale without the complexity of managing Kubernetes infrastructure. OKE also supports heterogenous clusters, so you can create both CPU and GPU compute node pools to run different types of workloads in the same cluster.

All GPU worker nodes provisioned with OKE images or platform images come with the NVIDIA CUDA software stack preinstalled. To deploy an application, such as NVIDIA Triton Inference Server, on a GPU node, just specify the number of GPUs required in the Kubernetes pod specifications.