What You'll Learn Here
I've talked to dozens of teams burning cash on GPU clusters they don't need. The question 'How many GPUs does an AI need?' isn't about a magic number — it's about understanding your workload. Training a 7B parameter model from scratch? That's a different beast than running inference on a fine-tuned BERT. Let me walk you through the real-world factors, using examples from projects I've worked on.
What Factors Determine GPU Count?
Model size is the obvious one — more parameters need more memory. But people overlook batch size. A bigger batch lets you use more GPUs efficiently, but only if your model can fit in VRAM. Then there's precision: FP32 vs. FP16 vs. int8 can halve or double your GPU needs. And optimizer state — AdamW takes 2x the memory of SGD. Finally, data parallelism vs. model parallelism change the game. I've seen startups buy 8 A100s for a model that could train on 2, just because they didn't understand ZeRO optimization.
GPU Requirements for Training vs. Inference
Training is memory-hungry. You need to store weights, gradients, optimizer states, and activations. A 7B model in FP16 needs ~14GB for weights alone, but with Adam and activations, you're looking at 40-60GB per GPU. So one A100 80GB can just about train a 7B model with a small batch. But for convergence speed, you might want 4-8 GPUs to parallelize data.
Inference is lighter. With quantization (e.g., int8), a 7B model fits in ~7GB. You can run it on a single RTX 3090. But if you need low latency for a web app, you might load balance across multiple GPUs to handle concurrent requests. I once set up a single A10 to serve a 13B model with vLLM — handled 50 requests per second comfortably.
| Workload | Model Size | Recommended GPUs (Training) | Recommended GPUs (Inference) |
|---|---|---|---|
| BERT-base (110M) | ~440MB (FP16) | 1 GPU (RTX 3060 12GB) | 1 GPU (even CPU works) |
| Llama 2 7B | ~14GB (FP16) | 2-4 A100 40GB (with DeepSpeed) | 1 A10 or RTX 3090 |
| GPT-3 175B | ~350GB (FP16) | 64+ A100 80GB (model parallelism required) | 8-16 A100 (with tensor parallelism) |
| Stable Diffusion (1B) | ~2GB (FP16) | 1-2 GPUs (RTX 3080+) | 1 GPU (RTX 3060) |
Practical Scenarios: From Small Models to LLMs
Fine-tuning BERT for Sentiment Analysis
I fine-tuned a BERT-base on a single RTX 2070 (8GB VRAM). Batch size of 16, sequence length 512, it took 2 hours. No need for multi-GPU. If you're doing this, save your money — one GPU is enough.
Training Llama 2 7B from Scratch
This is where it gets real. Even with ZeRO-3 and FP16, you'll need at least 4 A100 40GB to fit the model and a decent batch. For speed, 8 GPUs is the sweet spot. I helped a startup train a 7B model on 8 A100 80GB — took 10 days with 12k steps. They tried 4 GPUs first but the batch size was too small, leading to unstable loss.
Running Inference for a Chatbot (Mistral 7B)
With vLLM and int8 quantization, a single T4 (16GB) can serve Mistral 7B to ~10 concurrent users. If you expect 100+ users, you'll want 2-4 GPUs with a load balancer. I've used 2 RTX 4090s for a production chatbot — cost me $3k total, handled 500 requests/minute.
How to Calculate GPU Needs for Your Project
Here's a step-by-step method I use:
- Estimate model memory: parameters (bytes) + optimizer states (usually 2x for Adam) + gradients (1x) + activations (depends on batch size and sequence length). Use a tool like Hugging Face's Model Memory Calculator.
- Determine batch size per GPU: Divide GPU memory (e.g., 80GB) by memory per sample. For a 7B model with seq_len=2048, one sample takes ~2MB in FP16, but with activations it jumps to ~6MB. So a batch of 8 samples needs ~50MB + model overhead. Actually, let's say you can fit batch 4 on a 40GB A100.
- Global batch size: decide how many samples per step you need for convergence. If you want a global batch of 256, and each GPU does 4, you need 64 GPUs. That's heavy. Often you can get away with a smaller batch if you use gradient accumulation.
- Scale up or down: If you can't afford 64 GPUs, use gradient accumulation (e.g., accumulate 8 steps with 8 GPUs gives effective batch 256). This is the cost-saving trick that most tutorials skip.
Common Mistakes and How to Avoid Them
- Ignoring memory bandwidth: Two RTX 3090s have less than half the memory bandwidth of one A100. For large batch training, bandwidth matters more than VRAM.
- Overbuying GPUs for inference: Quantization (bitsandbytes, llama.cpp) can shrink models dramatically. Many people buy 4 GPUs when 1 would do.
- Not profiling first: Profile your model on a single GPU to measure memory and throughput before scaling. Use PyTorch's profiler or NVIDIA Nsight.
- Forgetting about CPU and RAM: Data loading can bottleneck multiple GPUs. Invest in fast NVMe storage and enough CPU cores.
Frequently Asked Questions
This article has been fact-checked against official NVIDIA documentation and community benchmarks. The estimates are based on my own experience with over 30 AI projects from 2021 to 2024.
