How to install KServe using Open Data Hub

KServe is a highly scalable and standards-based model inference platform on Kubernetes for scalable AI. Open Data Hub (ODH) now seamlessly incorporates KServe as an integral component within its ecosystem. In earlier iterations, utilizing KServe necessitated the execution of a relatively intricate installation script. However, with the release of Open Data Hub 2.5, significant strides have been made to streamline and alleviate the complexities associated with this process. Users can now experience a more user-friendly and efficient installation procedure, underscoring our commitment to enhancing accessibility and usability within the Open Data Hub framework.

This article will delve into a simplified method for installing KServe. The installation process can be summarized as follows:

1. Install required operators:
   • Red Hat OpenShift Serverless
   • Red Hat OpenShift Service Mesh
   • Open Data Hub
2. Deploy service mesh control plane by creating DataScienceClusterInitialization.
3. Deploy knative-serving by creating DataScienceCluster.
4. Install KServe.

Installing required operators

Users are responsible for the installation of operators on Red Hat OpenShift. To facilitate this process, users are required to install the KServe-dependent operators via OperatorHub. Figure 1 illustrates the successful installation of all three operators.

Deploy service mesh control planes with DataScienceClusterInitialization

Open Data Hub's DataScienceClusterInitialization refers to the process of setting up and initializing a cluster environment tailored for ODH components. After installing the necessary operators, Open Data Hub's DataScienceClusterInitialization makes service mesh control plane deployment easy. A DSCI is automatically created when you install the Open Data Hub operator. When the DataScienceClusterInitialization is created, it triggers the creation of a ServiceMeshControlPlane, which is an Istio object that configures the service mesh control plane.

This setup results in the deployment of three primary pods in the istio-system namespace (Figure 2):

• istiod pod for managing configurations and policies.

• istio-ingressgateway pod for handling incoming traffic, and optionally.

• istio-egressgateway pod for managing outgoing traffic. 

These components ensure robust traffic management, security, and observability for the services within the mesh.

The managementState field in ODH specifies whether a particular component is being managed (Managed) by ODH, left unmanaged (Unmanaged) for users to configure directly, or marked for removal (Removed) to be deleted.

• Managed: This means that ODH creates and manages the object. ODH handles its lifecycle, including creation, configuration, and deletion.
• Unmanaged: In this case, ODH does not manage the object, and users are responsible for its configuration. ODH does not interfere with the object's lifecycle; it remains as configured by the user.
• Removed: This indicates that ODH intends to delete the associated object. It marks the object for removal upon cluster uninstallation, effectively uninstalling it from the cluster.

These management states allow users to control how ODH interacts with specific components, allowing for tailored management approaches based on individual needs and preferences.

Deploying knative-serving with DataScienceCluster

Post Istio installation, the subsequent step involves confirming completion by verifying that all three pods are in the "Ready" state.

  oc wait --for=condition=ready pod -l app=istiod -n istio-system --timeout=10s
  oc wait --for=condition=ready pod -l app=istio-ingressgateway -n istio-system --timeout=10s
  oc wait --for=condition=ready pod -l app=istio-egressgateway -n istio-system --timeout=10s

To streamline this process, Open Data Hub's DataScienceCluster facilitates the installation of essential objects such as the gateway and service for knative-serving, net-istio controller, and seamless integration with ISTIO.

Beginning with Open Data Hub 2.5, a new specification has been introduced to the kserve field. An illustrative example is provided in Figure 3, featuring the creation and application of a Self-Signed Certificate to the gateway. However, users have the flexibility to furnish their preferred certificates, contributing to a more tailored and user-friendly experience.

Apply user's certificate with DataScienceCluster

If you have a user's certificate to apply, follow these straightforward steps:

1. Place or create a certificate secret. Generate or create a certificate secret in the istio-system namespace. Replace <your-secret-name> with a suitable name for your secret, and <path-to-your-certificate> with the actual path to the certificate file.

   oc create secret tls <your-secret-name> --cert=path/to/cert/file --key=path/to/key/file  -n istio-system

2. Apply DataScienceCluster. Use Provided instead of SelfSigned in the DataScienceCluster to integrate the user's certificate. Ensure to adjust any specific configurations as needed.

         serving:
           ingressGateway:
             certificate:
               type: Provided
               secretName:  <your-secret-name> 
           managementState: Managed

Now, your Knative gateway is configured with the user's certificate, providing secure communication.

Deploy KServe

After completing the knative-serving installation, follow these simple steps to deploy KServe:

1. Update DataScienceCluster configuration. Locate the DataScienceCluster configuration and look for the Kserve field within the configuration.

2. Set managementState to Managed. Locate the managementState field for the KServe installation within the DataScienceCluster configuration. Change its value from Remove to Managed.

   kserve:
     managementState: Managed

3. Verify successful KServe installation. After a successful installation of KServe, ensure the following pods are running in each namespace (Figure 4).

Conclusion

We hope that in the future, you will find it easy to install KServe through Open Data Hub and deploy and use your desired models effortlessly. With the enhancements introduced in Open Data Hub, the installation process and integration have been significantly streamlined, allowing you to set up a robust AI model deployment environment without the complexities.

This guide aims to help you smoothly complete the installation and configuration of KServe, alleviating potential challenges in the model deployment process. We look forward to seeing how the continued innovation and improvements in Open Data Hub will enhance the efficiency and productivity of your AI and machine learning projects.

Explore the possibilities with KServe, effortlessly deploy AI models, and achieve great success in your endeavors. Should you have any questions or require further assistance, our community and support channels are always available to help. Wishing you success in your AI model deployments!