Build an AI Inferencing Solution With TensorRt and PyTorch

AI inference workloads are increasingly demanding, requiring low latency, high throughput, and cost-efficiency at scale. Whether working with computer vision or natural language AI models, processing power and efficiency are key; inference workloads must be able to handle real-time predictions while maintaining optimal resource utilization. Choosing the right infrastructure and optimization tools can dramatically impact both performance and operational costs.

This guide shows how to build and benchmark a complete AI inferencing solution using TensorRT and PyTorch on Akamai Cloud’s NVIDIA RTX 4000 Ada GPU instances. NVIDIA RTX 4000 Ada GPU instances are available across global core compute regions, delivering the specialized hardware required for heavy AI workloads. Using the steps in this guide, you can:

• Deploy an RTX 4000 Ada GPU instance using Akamai Cloud infrastructure
• Run an AI inference workload using PyTorch
• Optimize your model with TensorRT for performance gains
• Measure latency and throughput

The primary AI model used in this guide is a ResNet50 computer vision (CV) model. However, the techniques used can be applied to other model architectures like object detection (YOLO; You Only Look Once) models, speech recognition systems (OpenAI’s Whisper), and large language models (LLMs) like ChatGPT, Llama, or Claude.

AI Inferencing

What is AI Inference?

AI inference occurs after model training - it’s the point at which the AI model generates an “opinion” or decision based on how it was trained. Think of inference like how people have the ability to form a point of view based on prior knowledge and experience.

Consider an AI model trained on a data set that includes millions of images of dogs. If given a new image of a dog not in the data set, the AI model uses inference to determine information about the new dog (i.e. the dog’s breed).

The goal of AI inference is to generate an educated, accurate result from a well-trained model with speed and efficiency.

What is Computer Vision (CV)?

Computer vision is a type of artificial intelligence that interprets images and outputs physical information about what is detected in the image. This guide uses a CV model (ResNet50) that uses a specific pre-trained set of images and runs an example image against that set. In this example, inferencing occurs when the CV model returns information about the example image (i.e. “this is a picture of a dog”) based on its pre-trained knowledge base of millions of sample images.

What are TensorRT and PyTorch?

TensorRt

TensorRT is an API and tool ecosystem by NVIDIA that includes inference compilers, runtimes, and deep learning model optimizations. TensorRT is trained on all major frameworks and is used to improve performance on NVIDIA GPUs using techniques like kernel auto-tuning, dynamic tensor memory management, and multi-stream execution. It directly integrates with PyTorch using the TensorRT Framework Integrations API to achieve up to 6x faster inferencing.

PyTorch

PyTorch is an open-source machine learning framework based on the Torch library and developed by Meta AI for training deep learning models. PyTorch is written in Python and integrates with TensorRT through Torch-TensorRT, so developers can optimize PyTorch models without changing existing codebases. PyTorch integrates with CUDA (Compute Unified Device Architecture) to take advantage of parallel computing architectures found in NVIDIA GPUs.

Before You Begin

The following prerequisites are recommended before starting the implementation steps in this tutorial:

• An Akamai Cloud account with the ability to deploy GPU instances
• The Linode CLI configured with proper permissions
• An understanding of Python virtual environments and package management
• General familiarity of deep learning concepts and models

Architecture Diagram

1. User connects to the NVIDIA RTX 4000 Ada GPU instance via SSH.

2. CUDA (Compute Unified Device Architecture) and NVIDIA drivers are installed via CUDA keyring to ensure the latest stable versions are running.

3. PyTorch, TensorRT, and their dependencies are installed in a Python Virtual Environment (venv) to prevent any conflicts with system-wide packages.

4. An inferencing script written in Python is created. The script imports a pre-trained AI model (ResNet50) and benchmarks the inference time against sample images to test GPU performance.

5. The inference script outputs the average time per inference across a number of runs for a given sample image.

Deploy an NVIDIA RTX 4000 Ada Instance

Akamai’s NVIDIA RTX 4000 Ada GPU instances can be deployed using Cloud Manager or the Linode CLI. Deployment instructions and technical requirements:

• Cloud Manager deployment: For instructions on deploying a GPU instance via the Cloud Manager, see our Create a Linode guide.

• CLI deployment: For guidance on deploying a GPU instance using the Linode CLI, see the Create a Linode section of our API documentation.

• Distribution: Select the latest stable Ubuntu version (Ubuntu 24.04 LTS as of this writing)

• Plan type: All RTX 4000 Ada GPU plan types support the AI inference workload in this guide

• Region: For a list of GPU region availability, see our Choose a Data Center guide. See our API documentation to see a region’s service availability using the Linode API or CLI.

Set Up Your Development Environment

Once your GPU is fully deployed, connect to your instance to update system packages and install system dependencies. It is recommended to first follow the steps in our Set up and secure a Linode guide to configure a limited user with sudo access and secure your sever.

Update Packages

1. Log into your instance via SSH. Replace user with your sudo username and IP_ADDRESS with your Linode instance’s IP address:

   ssh user@IP_ADDRESS

2. Update your system and install build tools and system dependencies:

   sudo apt update && sudo apt install -y \
       build-essential \
       gcc \
       wget \
       gnupg \
       software-properties-common \
       python3-pip \
       python3-venv

3. Download and install NVIDIA CUDA keyring so you get the latest stable drivers and toolkits:

   wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
   sudo dpkg -i cuda-keyring_1.1-1_all.deb

4. Update system packages after the keyring is installed:

   sudo apt update

Install NVIDIA Drivers and CUDA Toolkit

1. Install the NVIDIA driver repository along with the latest drivers compatible with the RTX 4000 Ada card:

   sudo apt install -y cuda

2. Reboot your instance to complete installation of the driver:

   sudo reboot

3. After the reboot is complete, log back into your instance:

   ssh user@IP_ADDRESS

4. Use the following command to verify successful driver installation:

   nvidia-smi

   This displays basic information about your RTX 4000 Ada instance and its driver version. Your driver and software versions may vary based on release date:

   +-----------------------------------------------------------------------------------------+
   | NVIDIA-SMI 575.57.08              Driver Version: 575.57.08      CUDA Version: 12.9     |
   |-----------------------------------------+------------------------+----------------------+
   | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
   | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
   |                                         |                        |               MIG M. |
   |=========================================+========================+======================|
   |   0  NVIDIA RTX 4000 Ada Gene...    On  |   00000000:00:02.0 Off |                  Off |
   | 30%   35C    P8              4W /  130W |       2MiB /  20475MiB |      0%      Default |
   |                                         |                        |                  N/A |
   +-----------------------------------------+------------------------+----------------------+
   
   +-----------------------------------------------------------------------------------------+
   | Processes:                                                                              |
   |  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
   |        ID   ID                                                               Usage      |
   |=========================================================================================|
   |  No running processes found                                                             |
   +-----------------------------------------------------------------------------------------+

Configure Your Python Environment

Set up and use a Python Virtual Environment (venv) so that you can isolate Python packages and prevent conflicts with system-wide packages and across projects.

Create the Virtual Environment

1. Using the python3-venv package downloaded during setup, set up the Python Virtual Environment:

   python3 -m venv ~/venv
   source ~/venv/bin/activate

   You can confirm you are using your virtual environment when you see (venv) at the beginning of your command prompt:

   (venv) user@hostname

2. While in your virtual environment, upgrade pip to the latest version to complete the setup:

   (venv)

   pip install --upgrade pip

Install PyTorch and TensorRT

Remain in your virtual environment to install PyTorch, TensorRT, and dependencies. These are the primary AI libraries needed to run your inference workloads.

pip install torch==2.5.1+cu121 torchvision==0.16.1+cu121 torchaudio==2.5.1+cu121 --index-url https://download.pytorch.org/whl/cu121
pip install requests
pip install nvidia-pyindex
pip install nvidia-tensorrt
pip install torch-tensorrt -U

Run and Benchmark the ResNet50 Inference Model

Create and run a Python script using a pre-trained ResNet50 computer vision model. Running this script tests to make sure the environment is configured correctly while providing a way to evaluate GPU performance using a real-world example. This example script is a foundation that can be adapted for other inference model architectures.

1. Using a text editor such as nano, create the Python script file. Replace inference_test.py with a script tile name of your choosing:

   nano inference_test.py

2. Copy and insert the below code content into the script. In order, the script performs the following actions:

   • Imports the PyTorch framework and its pre-trained models

   • Pulls an example sample image of a dog from PyTorch’s GitHub repository on which to run AI inference

   • Preprocessing for the sample image: Image resizing for compatibility with the ResNet50 model, format conversion for PyTorch, add a “batch dimension” to emulate multiple images, moves the processed data to the GPU

   • Loads the ResNet50 pre-trained model, including a library of images

   • GPU optimization and preparation for benchmarking

   • Runs the inference benchmark 20 times against the ResNet50 AI model

   File: inference_test.py

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49

   # import PyTorch, pre-trained models from torchvision and image utilities import torch import torchvision.models as models import torchvision.transforms as transforms from PIL import Image import requests from io import BytesIO import time # Download a sample image of a dog # You could replace this with a local file or different URL img_url = "https://github.com/pytorch/hub/raw/master/images/dog.jpg" image = Image.open(BytesIO(requests.get(img_url).content)) # Preprocess # Resize and crop to match ResNet50’s input size # ResNet50 is trained on ImageNet where inputs are 224sx224 RGB # Convert to a tensor array so PyTorch can understand it # Use unsqueeze(0) to add a batch dimension, tricks model to think we are sending a batch of # images # Use cuda() to move the data to the GPU transform = transforms.Compose([ transforms.Resize(256), transforms.CenterCrop(224), transforms.ToTensor(), ]) input_tensor = transform(image).unsqueeze(0).cuda() # Load a model (ResNet50) pretrained on the ImageNet dataset containing millions of images model = models.resnet50(pretrained=True).cuda().eval() # Warm-up the GPU # Allows the GPU to optimize the necessary kernels prior to running the benchmark for _ in range(5): _ = model(input_tensor) # Benchmark Inference Time using an average time across 20 inference runs start = time.time() with torch.no_grad(): for _ in range(20): _ = model(input_tensor) end = time.time() print(f"Average inference time: {(end - start) / 20:.4f} seconds")

   When complete, press Ctrl + X to exit nano, Y to save, and Enter to confirm.

3. Run the Python script:

   python inference_test.py

   If everything works correctly, you should see output similar to the below. Time results may vary:

   Average inference time: 0.0025 seconds

   The model runs 20 times, and the total inference time is then divided by 20 to get the average time per inference. This provides an idea of how quickly your GPU can process input using this model.

Accelerate Inferencing with TensorRT

If you want to accelerate your GPU’s inferencing power further, you can use NVIDIA’s optimized inference runtime, TensorRT, to deliver faster inference with lower latency.

1. Add the highlighted line (line 10) to the import section at the top of your inference script to import the TensorRT model (torch_tensorrt) previously installed:

   File: inference_test.py

   1 2 3 4 5 6 7 8 9 10

   # import PyTorch, pre-trained models from torchvision and image utilities import torch import torchvision.models as models import torchvision.transforms as transforms from PIL import Image import requests from io import BytesIO import time import torch_tensorrt

2. Next, add the highlighted code block (lines 35-51) after the Load a model section to load the TensorRT-optimized model:

   File: inference_test.py

   31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51

   # Load a model (ResNet50) pretrained on the ImageNet dataset containing millions of images model = models.resnet50(pretrained=True).cuda().eval() # Compile with TensorRT model = torch_tensorrt.compile( model, inputs=[torch_tensorrt.Input(input_tensor.shape)], enabled_precisions={torch.float} ) # Benchmark TensorRT Inference for _ in range(5): _ = model_trt(input_tensor) start = time.time() with torch.no_grad(): for _ in range(20): _ = model_trt(input_tensor) end = time.time() print(f" TensorRT average inference time: {(end - start) / 20:.4f} seconds")

   Save your changes when complete.

3. Run the script again, and compare the Average inference time output to that of your PyTorch results:

   Average inference time:

As you scale your inference, using TensorRT can help keep your model smaller and more performant.

Next Steps

Try switching out ResNet50 for different model architectures available in torchvision.models, such as:

• efficientnet_b0: Lightweight and accurate
• vit_b_16: Vision Transformer model for experimenting with newer architectures

This can help you see how model complexity affects speed and accuracy.

More Information

You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

• Akamai TechDocs: GPU Linodes
• Link Title 2