Common development hurdles and Red Hat’s solution: An introduction

Before diving into the solution, it’s crucial to first understand the issues developers face in implementing AI applications. Then, we will outline the solution architecture we’ll execute to address these challenges.

Prerequisites:

• Hardware requirements:
  • Development machine with at least 16GB RAM (32GB recommended).
  • Storage space: Minimum 50GB free disk space for models and training data.
  • GPU support (optional but recommended for faster training).
• Software requirements:
  • Python 3.9+ installed with pip.
  • Git for cloning repositories.
  • Terminal/command line access.
  • Text editor capable of editing YAML files.
• Technical knowledge:
  • Basic Python and virtual environment management.
  • Command line proficiency for running installation and training commands.
  • Understanding of YAML syntax for creating taxonomy files.
  • File system navigation and directory structure management.
• Account access:
  • Internet connection for downloading models and dependencies.
  • GitHub access for taxonomy repository references (Read-only).

In this lesson, you will:

• Identify common AI development challenges that enterprises face when building production AI applications.
• Understand Red Hat's AI portfolio architecture and how InstructLab, RHEL AI, and OpenShift AI work together.
• Install and configure InstructLab in a Python virtual environment on your local machine.
• Create a custom taxonomy with telecommunications domain knowledge using YAML format.
• Train and customize a foundation model with your domain-specific knowledge using InstructLab.
• Test your fine-tuned model to validate the learning and knowledge integration.
• Convert models to GPT-Generated Unified Format (GGUF) format for optimized inference and compatibility across platforms.

The current state of AI: Development challenges

To begin, let's examine the pain points that AI developers commonly face:

• Limited access to customizable foundation models: Many organizations struggle to adapt general-purpose large language models (LLMs) to their specific business domains without extensive machine learning (ML) expertise.
• Lengthy development cycles: Taking AI projects from experimentation to production often involves complex handoffs between data scientists and operations teams.
• Infrastructure complexity: Managing the specialized hardware and software requirements for AI workloads adds significant overhead.
• Governance and security concerns: Ensuring models comply with organizational policies and security standards remains challenging.
• Deployment and scaling limitations: Many AI projects fail when transitioning from pilot to production due to scalability issues.

Red Hat's AI portfolio addresses these challenges head-on, providing a comprehensive platform for developing, training, deploying, and managing AI applications across hybrid environments.

Red Hat's AI portfolio: An overview

Red Hat offers a complete ecosystem for AI development:

• InstructLab: A community-driven project that simplifies LLM tuning and customization.
• Red Hat Enterprise Linux (RHEL) AI: A foundation model platform combining Granite models with InstructLab in a bootable RHEL image.
• Red Hat OpenShift AI: A comprehensive platform for managing the complete AI/ML lifecycle at scale.

Let's explore how InstructLab and RHEL AI work together with a practical use case.

Use case: Building a customer support AI assistant with domain-specific knowledge

For our example, let's consider a telecommunications company that wants to create an AI-powered customer support assistant that can access and utilize its proprietary product knowledge base. This assistant needs to:

• Handle common customer inquiries about telecom products.
• Answer questions specific to the company's services, policies, and technical specifications.

Solution architecture

Our solution will leverage Red Hat's AI portfolio to create a comprehensive workflow consisting of three phases. We will:

1. Use InstructLab to customize a foundation model with telecom-specific knowledge.
2. Deploy the model using RHEL AI for experimentation and initial testing.
3. Scale and productionize the application with OpenShift AI.

Let's break down the implementation details. The rest of this learning path will walk you through each step of this process.

Phase 1: Customize the foundation model with InstructLab

InstructLab provides a user-friendly approach to enhancing LLMs with specific domain knowledge without requiring extensive machine learning expertise.

Installation and setup

First, let's install and set up InstructLab with a Python virtual environment:

# Create a new directory for our project
mkdir telecom-ai && cd telecom-ai

# Create a Python virtual environment
python3 -m venv --upgrade-deps venv

# Activate the virtual environment
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Clear pip cache (recommended)
pip cache remove llama_cpp_python

# Install InstructLab
pip install instructlab

# Initialize InstructLab configuration
ilab config init

During the initialization, InstructLab will ask for the path to the taxonomy repository and model locations. You can accept the defaults by pressing Enter.

Create the telecom knowledge taxonomy

Next, we'll create a taxonomy of telecommunications knowledge. Create the necessary directories and files:

# Create the required directory structure
mkdir -p ~/.local/share/instructlab/taxonomy/knowledge/telecom

Create a file called qna.yaml in this directory with the following content:

version: 3
domain: telecom
document_outline: Overview of telecommunications technologies and services
created_by: mrigankapaul
seed_examples:
  - context: |
      Fiber optic internet is a broadband connection that uses fiber optic cables made of thin strands
      of glass or plastic to transmit data using light. These cables can transmit data at speeds up to
      10 Gbps or higher, making them significantly faster than traditional copper connections.
    questions_and_answers:
      - question: What is fiber optic internet?
        answer: |
          Fiber optic internet is a broadband connection that uses fiber optic cables made of thin strands
          of glass or plastic to transmit data using light signals instead of electrical signals.
      - question: What materials are fiber optic cables made from?
        answer: Fiber optic cables are made from thin strands of glass or plastic that can transmit light signals.
      - question: How fast can fiber optic internet transmit data?
        answer: |
          Fiber optic internet can transmit data at speeds up to 10 Gbps or higher, which is significantly
          faster than traditional copper-based connections.
  - context: |
      5G is the fifth generation of cellular network technology, offering significantly faster data
      transmission speeds, lower latency, and greater capacity than previous generations. 5G networks
      operate on higher frequency bands and use smaller cell sites than 4G networks.
    questions_and_answers:
      - question: What is 5G technology?
        answer: 5G is the fifth generation of cellular network technology that offers faster speeds, lower latency, and greater capacity than previous generations.
      - question: How does 5G differ from previous cellular network generations?
        answer: 5G operates on higher frequency bands, uses smaller cell sites, and provides significantly faster data transmission speeds and lower latency than 4G and earlier generations.
      - question: What are the advantages of 5G for mobile users?
        answer: 5G provides mobile users with dramatically faster download and upload speeds, more reliable connections, and the ability to connect more devices simultaneously.
  - context: |
      VoIP (Voice over Internet Protocol) is a technology that allows voice calls to be made over
      internet connections rather than traditional phone lines. It converts analog voice signals into
      digital data packets that are transmitted over IP networks.
    questions_and_answers:
      - question: What is VoIP technology?
        answer: VoIP (Voice over Internet Protocol) is a technology that allows voice calls to be made over internet connections rather than traditional phone lines.
      - question: How does VoIP work?
        answer: VoIP works by converting analog voice signals into digital data packets that are transmitted over IP networks, then converted back to voice at the receiving end.
      - question: What are the benefits of using VoIP over traditional phone services?
        answer: VoIP offers benefits such as lower costs (especially for long-distance calls), greater flexibility, enhanced features like video conferencing, and the ability to make calls from multiple devices.
  - context: |
      Satellite internet provides connectivity through communications satellites rather than land-based
      cables. It's particularly valuable for remote areas where traditional infrastructure is limited or
      absent. Modern satellite internet services can offer download speeds of up to 100 Mbps.
    questions_and_answers:
      - question: What is satellite internet?
        answer: Satellite internet is a connectivity solution that transmits data through communications satellites orbiting Earth rather than through land-based cables.
      - question: Where is satellite internet most valuable?
        answer: Satellite internet is most valuable in remote or rural areas where traditional cable or fiber infrastructure is limited or entirely absent.
      - question: What speeds can modern satellite internet achieve?
        answer: Modern satellite internet services can offer download speeds of up to 100 Mbps, though they typically have higher latency than land-based connections.
  - context: |
      Network latency refers to the time delay between when data is sent and when it's received across
      a network, typically measured in milliseconds. Low latency is crucial for applications requiring
      real-time interaction, such as video calls, online gaming, and industrial control systems.
    questions_and_answers:
      - question: What is network latency?
        answer: Network latency is the time delay between when data is sent and when it's received across a network, typically measured in milliseconds.
      - question: Why is low latency important in telecommunications?
        answer: Low latency is important for applications requiring real-time interaction, such as video calls, online gaming, financial trading, and industrial control systems.
      - question: What factors can increase network latency?
        answer: Network latency can be increased by physical distance between endpoints, network congestion, routing equipment performance, and the use of certain transmission media like satellite connections.
document:
  repo: https://github.com/instructlab/taxonomy.git
  commit: main
  patterns:
    - README.md

Be careful about formatting when creating this file:

• Make sure there are no trailing spaces at the end of lines.
• Ensure proper indentation.
• Add a new line at the end of the file.

Verify and process the taxonomy

Now, let's verify our taxonomy is properly formatted:

# Verify our taxonomy changes
ilab taxonomy diff

If all is well, you'll see a message indicating your taxonomy is valid. Next, we'll generate synthetic training data and train the model:

# Download a model to work with (if not already done during initialization)
ilab model download

# Generate synthetic training data
ilab data generate

# Train the model
ilab model train

# Convert model for macOS compatibility (required for Apple Silicon)
ilab model convert --model-dir ~/.local/share/instructlab/checkpoints/instructlab-granite-7b-lab-mlx-q

The training process may take some time, depending on your hardware. Once complete, we can test the enhanced model.

InstructLab's approach enables us to efficiently improve the model's understanding of telecommunications concepts with minimal training data and computing resources.

Test your fine-tuned model

After completing the training and conversion, you can test your model using the chat interface:

ilab model chat --model ./instructlab-granite-7b-lab-trained/instructlab-granite-7b-lab-Q4_K_M.gguf

This starts an interactive session where you can evaluate how well your model has learned the telecom domain. Here are a few effective questions to test your model:

• "What is fiber optic internet and how fast can it be?"

  • Tests if the model can combine information from multiple related question-answer pairs.

• "How does 5G technology compare to previous generations?"

  • Checks the model's understanding of comparative technological information.

• "Can you explain how VoIP technology works and its advantages?"

  • Tests if the model can synthesize technical functionality with business benefits.

• "Why might someone in a rural area consider satellite internet?"

  • Evaluates the model's understanding of use case scenarios for specific technologies.

• "What telecommunications technology would be best for real-time gaming?"

  • Application question requiring the model to consider latency requirements across technologies.

• "What's the difference between fiber and satellite internet?"

  • Tests cross-context learning by requiring comparison between two different technology sections.

Start with questions directly from your seed examples, then gradually introduce questions requiring synthesis across multiple contexts. This progression will help you evaluate both knowledge retention and the model's ability to generalize from its training.

Now that we’ve tested the model, let’s move on to deployment with RHEL AI.