Use GPUs that support NVIDIA Multi-Instance GPU (MIG) to improve utilization and reduce hardware requirements. MIG partitions a single physical GPU into multiple isolated instances, enabling multiple models or workloads to run concurrently on the same GPU.
An iQ Studio agent that uses a RAG pipeline runs these models in parallel:
- Embedding model
- Reranker model
- Inference (LLM) model
With MIG-enabled GPUs, assign each model to a separate MIG instance to consolidate workloads on fewer physical GPUs.
GPUs that don’t support MIG require one dedicated GPU per model, even when a model does not fully use available GPU memory. In this configuration, the RAG pipeline requires at least three GPUs. NVIDIA L40s GPUs do not support MIG.