How to do a Performance Test for AI Models hosted on NVIDIA infrastructure?

 

 

A performance test is extremely important for an Artificial Intelligence (A.I.) Model just as it is important for an e-commerce website like Amazon.

The NVIDIA Triton server is a gold standard that standardizes AI model deployment and execution across every workload and it is important that it works well for your custom models.

If you/or your company are working with NVIDIA chips, there is a very high probability you are using the Triton server. The Triton server is the secret sauce that helps to get the peak performance out of expensive NVIDIA hardware.

There are some tools provided by NVIDIA as part of it development sdk. You just need to download the correct docker container

1. Triton Model Analyzer - This tool helps to understand the compute and memory requirements for your model. It also uses the Triton Performance Analyzer to test the inference performance of models. (see the next point) 

2. Triton Performance Analyzer -  This tool only concentrates on the inference performance of models. It helps you to fine tune various model configurations like instance groups etc.

3. Gen-AI-Perf - This tool primarily focuses on Generative AI models. 

It currently supports different types of models like Embedding, Ranker and even Vision models.

 4. Traditional Tools - The performance testing tools like Apache JMeter and Locust are still widely used.