The hidden costs of Azure Backup, part 1
Why AI starvation pushes you toward an upgrade you don’t actually need
You’ve invested in GPU capacity on Azure. Expensive hardware, because AI workloads demand raw compute. The invoice matches exactly what you provisioned. And yet the system runs slow, the models train slower than expected, and the pressure to scale up keeps building. In this blog, Edco Wallet explains why the problem usually isn’t the GPU at all. it’s the I/O.
A GPU is a parallel computing engine. It can run thousands of calculations at once, but only if it’s continuously fed with data. The moment the supply of training data, model weights, or input batches falls behind the processing capacity, starvation sets in: the GPU sits there waiting instead of computing.
In Azure terms, this means: you’re paying for NC- or ND-series VMs with NVIDIA A100 or H100 GPUs, but your GPU utilization sits at 40 to 60 percent. Not because the GPU is too small, but because storage, network links, or data preparation is the actual bottleneck.
We saw exactly this pattern with a recent client. An ND96asr v4, one of the most expensive GPU VMs in Azure, with GPU utilization stuck at 55%. The team’s recommendation was to scale up to a bigger SKU with even more memory. After analysis, the real problem turned out to be I/O throughput from Azure Blob Storage.
Blob Storage without a premium tier. Standard Azure Blob Storage has a latency of a few milliseconds per request. When loading large datasets for model training, where thousands of small files get read in rapid succession, that adds up fast. Azure Premium Blob Storage or Azure NetApp Files offers significantly higher throughput, but few teams ever make that trade-off deliberately.
No local NVMe caching. ND-series VMs come with local NVMe disks meant for fast temporary storage. If training data isn’t preloaded onto local NVMe but instead read directly from remote storage during the training step, you lose a huge amount of throughput. A lot of pre-configured entry-level AI VMs default to slow HDD instead of SSD.
Suboptimal batch size and prefetching. If the dataloader isn’t prefetching asynchronously (loading the next batch while the GPU is still processing the current one), the GPU sits idle periodically, waiting. This is a configuration choice, not a hardware issue.
Network as the bottleneck in multi-GPU setups. In distributed training environments (multiple GPU nodes), the network links between them are what determines performance. Azure’s ND-series supports InfiniBand for high-bandwidth communication, but only if it’s explicitly configured. Without InfiniBand, communication falls back to standard Ethernet: five to ten times slower.
The problem with AI starvation is that the symptoms look exactly like undercapacity. Slow training. Long turnaround times. Long wait times. The obvious conclusion: you need more GPU power.
But if the GPU isn’t even being fully used because of I/O starvation, a bigger GPU fixes nothing. You end up paying twice: once for the underused instance you already have, and again for the extra capacity that doesn’t solve the actual problem.
In Azure pricing terms: a Standard_ND96asr_v4 (8x A100) costs roughly €28 per hour. Scaling up to a bigger or more redundant configuration can double that. If the bottleneck is in the I/O, that money is simply wasted.
At that same client, the recommendation to scale up would have added €15,000 a month. After I/O tuning (NVMe caching, prefetching, activating InfiniBand), GPU utilization rose from 55% to 89%. The upgrade was never necessary.
AI starvation can almost always be fixed without spending more. The fixes are technical, not financial: preload training data onto local NVMe instead of reading it live from Blob Storage. Configure asynchronous prefetching in your dataloader (PyTorch DataLoader num_workers, prefetch_factor). Check and activate the InfiniBand NICs for multi-node training on ND-series VMs. Consider Azure Premium Blob Storage, Azure NetApp Files, or Silk storage when you’re working with datasets ranging from hundreds of gigabytes to terabytes. Monitor GPU utilization through Azure Monitor or NVIDIA DCGM, watching not just GPU load but also memory bandwidth and compute utilization alongside network and disk I/O.
None of these fixes require a more expensive VM tier. They do require knowing what’s actually going on.
The fundamental problem is that most Azure dashboards show GPU load, but never make the I/O bottleneck visible. You see that the GPU is “busy,” but not that it’s sitting idle, waiting on data, 40% of the time.
That’s the difference between insight and understanding. A dashboard shows you numbers. A FinOps analysis tells you what those numbers actually mean for your invoice, and what you can do about it.
Paying for GPU capacity on Azure but not seeing the performance to match? OptimaSure analyzes your AI infrastructure and identifies, within 30 days, where the real bottleneck sits and how to fix it. Get in touch.