GPU Capacity: CoreWeave, RunPod, Modal vs the Hyperscalers
    AI
    9 min read

    GPU Capacity: CoreWeave, RunPod, Modal vs the Hyperscalers

    Reserved clusters, on-demand pods, per-second serverless GPUs and hyperscaler capacity blocks solve different problems. How to match procurement model, billing, cold starts and quotas to your workload.

    Will Schulz
    Will Schulz
    CEO

    Most teams pick a GPU provider by comparing hourly prices for an H100. That number rarely decides whether a project ships on time. The harder questions are about access: can you get the capacity, how long must you commit, what are you billed for while idle, how long does a cold replica take to answer, and how much paperwork stands between a new account and its first GPU?

    Providers have settled on four procurement models: reserved or committed clusters, on-demand pods and instances, serverless GPU functions billed per second, and hyperscaler capacity products that sit somewhere in between. This post covers how each works on CoreWeave, RunPod, Modal, AWS, Google Cloud and Azure, per their own docs and pricing pages at the time of writing (November 2026). Inference cost techniques and H100 price trends are covered in an earlier post; this one stays on procurement, billing, cold starts, quotas and portability.

    Four Ways to Buy a GPU

    Serverless GPU functions are the most granular option. Modal lists GPUs per second: $0.001097 per second for an H100 SXM5, $0.000694 for an A100 80 GB, and $0.000306 for an A10 [1]. You pay for container load time and input processing, and nothing when the function has scaled to zero. Read the multipliers, though. Modal's page lists region selection at 1.15 to 1.75x base prices and non-preemptible execution at 3x base prices [1]. If your function needs a fixed region or can't tolerate preemption, the headline rate isn't your rate.

    On-demand pods sit one level up. RunPod lists an H100 SXM pod at $4.39/hr on Secure Cloud and from $2.69/hr on Community Cloud, and says pods are billed per second with no commitment [2]. Secure Cloud runs in T3/T4 data centers aimed at production workloads, while Community Cloud connects you to individual providers through a vetted peer-to-peer system [3]. RunPod's serverless product is a third option, with H100 workers listed at $5.89/hr [2].

    Reserved clusters are CoreWeave's core business. Its pricing page lists an 8-GPU NVIDIA HGX H100 instance at $49.24 per instance hour on demand and says it offers "up to 60% discounts over our On-Demand prices for committed usage" [4]. The delivery model is Kubernetes-first. CKS "runs Kubernetes directly on bare metal Nodes," with a managed control plane and a data plane you configure [5]. That suits teams already running GPU workloads on Kubernetes.

    Hyperscaler Capacity: Blocks, Flex-Start and Big VMs

    AWS EC2 Capacity Blocks for ML let you reserve GPU instances for a future window. You can book a start time up to eight weeks out, each block holds up to 64 instances, and you can hold up to 256 instances across blocks [6]. Durations come in 1-day increments up to 14 days, then in 7-day increments up to 182 days [7]. The price depends on supply and demand at purchase time and is charged up front [8]. The fine print shapes how you plan. Cancellations aren't allowed, blocks end at 11:30 AM UTC, and instance termination begins at 11:00 AM UTC on the final day [6]. Savings Plans and Reserved Instance discounts don't apply [8].

    Google Cloud splits the same problem into provisioning models. Spot VMs offer discounts of "up to 91% off" but can be preempted at any time. Flex-start, powered by Dynamic Workload Scheduler, queues your request and runs instances for a minimum of 10 minutes and a maximum of seven days. Its discount is up to 53% for A4, A3, A2 and G4 machine types [9]. For standalone flex-start VMs, the request can wait between 90 seconds and two hours for capacity before it fails. It also needs preemptible quota, not standard quota [10]. For longer runs, calendar-mode future reservations cover workloads of up to 90 days at DWS rates [9].

    Azure takes a different approach: large, tightly coupled VMs. The Standard_ND96isr_H100_v5 has 96 vCPUs and eight 80 GB H100 GPUs. Each GPU gets its own 400 Gbps InfiniBand connection, for 3.2 Tbps of interconnect per VM, and the docs note that live migration isn't supported [11]. You buy this size for multi-node training, not for scale-to-zero inference.

    Billing Granularity Is Where Idle Money Goes

    Per-second billing sounds the same everywhere. It isn't, because providers differ on which seconds they count. EC2 On-Demand bills per second while an instance is running, with a 60-second minimum [12]. RunPod serverless starts billing when a worker starts and rounds up to the nearest second. It bills cold start time spent loading the model, but not the container image pull, which happens while the worker is scaled down. It also bills idle time up to the idle timeout, which defaults to 5 seconds [13]. Modal keeps containers alive for 60 seconds after the last input by default and bills that time [1]. Our GPU cloud pricing table lines up these billing rules next to on-demand list prices taken from each provider's own pricing page.

    The gap between a 5-second and a 60-second default is invisible on a pricing page. On a spiky endpoint it can be most of the bill. A Capacity Block flips the problem around: you prepay the whole window, and the bill shows separate lines for used and unused reserved time [8]. Utilization is your problem from the moment you book.

    Cold Starts Are a Configuration Choice

    Every scale-to-zero platform trades cold starts against idle cost, and each one exposes the dial. On RunPod, flex workers scale to zero and incur cold starts. Active workers run 24/7 and eliminate them [14], but they "incur charges continuously, including when they have no jobs." FlashBoot retains worker state after spin-down for faster revival. It's on by default for new endpoints created in the console and off by default through the REST API [15]. Watch that drift if you provision from infrastructure code.

    Modal offers four controls. scaledown_window sets idle time anywhere from two seconds to twenty minutes. min_containers sets a floor so the function never scales to zero. buffer_containers keeps extra warm containers ready while the function is active. Memory snapshots capture a warmed-up container and reuse that state on later boots [16]. Modal's own guide notes that all of these raise resource consumption [16], so treat them as a latency budget you pay for, not free tuning.

    A practical rule: keep a warm floor when your p99 latency target is tighter than your measured cold start. Then measure cold start with your real weights, because loading model weights into GPU memory is usually the dominant term, and it's the part RunPod bills [13].

    Quotas Are the Real Lead Time

    On a hyperscaler, a new account often can't launch a single GPU instance. AWS sets the default for Running On-Demand P instances to 0 vCPUs, the same for G and VT instances, and the same for the concurrent P5 Capacity Blocks quotas [17]. Quotas are counted in vCPUs, not GPUs [12]. A p5.48xlarge has 192 vCPUs and eight H100s [18], so one instance needs a P quota of at least 192. AWS does raise On-Demand quotas automatically as you use them, and you can request increases through Service Quotas [12]. Instances inside a Capacity Block don't count against your On-Demand limits [6].

    Azure enforces vCPU quota at two tiers per subscription per region: a regional total and a per-VM-family limit. Adjustable requests are "usually" decided within a few minutes. If a request isn't fulfilled, or the quota isn't available, you open a support request [19]. On Google Cloud, remember that flex-start draws on preemptible quota [10], so a standard GPU quota increase won't help it.

    The smaller clouds have limits too. RunPod ties the serverless worker cap to account balance: 5 by default, 10 at $100 or more, and up to 60 at $900 or more [14]. Endpoints default to a max of 3 workers [15], and the default spend limit is $80 per hour across all resources [13]. File quota requests weeks before launch.

    Portability and Data Gravity

    The cheapest capacity is the capacity you can move to. Build one OCI container image with your inference server and pinned CUDA libraries, and keep provider-specific glue thin: a Modal decorator, a RunPod handler, a Kubernetes manifest for CKS. Modal lets you pass an ordered list of GPU types as fallbacks, and it may run an H100 request on an H200 at no extra cost unless you pin it with "H100!" [20]. Pin it when benchmarking.

    Weights are the heavy part: a 70B-parameter model at 2 bytes per parameter is about 140 GB, and every cold start on a new provider has to fetch it. Egress policy varies widely. CoreWeave lists egress and internet data transfer as free [4]. AWS includes 100 GB of free data transfer out to the internet each month [21], so any multi-provider deployment that pulls weights out of S3 will pass that quickly. Keep a copy of the weights next to each compute pool: object storage in the same cloud, or a RunPod network volume, which exists independently of any single pod and can attach to pods or serverless endpoints [3].

    Matching Workload Shape to Model

    1. Batch training or fine-tuning with a fixed end date: use a Capacity Block, a calendar-mode reservation, or a committed CoreWeave cluster. Build checkpointing around hard end times like the 11:00 AM UTC termination start on AWS [6]. 2. Interruptible batch jobs such as evals and embedding backfills: use Spot or flex-start, with retries, and accept the 7-day cap [9]. 3. Bursty or low-volume inference: use serverless per-second GPUs with a short idle window and a small warm floor. 4. Steady, high-utilization inference: at a constant load, a warm serverless floor stops saving money. Compare it against on-demand pods or committed instances at your real utilization, not peak.

    How AW3 Host Fits

    AW3 Host deploys to CoreWeave, RunPod and Modal alongside AWS, GCP, Azure and Kubernetes from one dashboard and API. For the multi-provider setup above, that means weighted routing to shift inference traffic between a serverless endpoint and a reserved pool, failover driven by 60-second health checks when a provider runs out of capacity, canary rollouts when you move a model to new hardware, and daily cost sync with budget policies, so idle warm floors show up before the invoice does.

    AW3 won't get you quota, book a Capacity Block, shorten a cold start, or move 140 GB of weights for you. Those stay with you and your providers. What it removes is the work of running the same service in several places.

    Takeaways

    Pick the procurement model before you pick the provider. Read the multipliers and idle-billing defaults, not just the headline per-second rate. Treat cold start mitigation as a budget line. Request quota weeks early, and remember that AWS counts it in vCPUs. Keep one portable image and a copy of the weights near each compute pool.

    Sources

    1. Modal Pricing — Modal, 2026
    2. Runpod Pricing — Runpod, 2026
    3. Pods overview — Runpod Docs, 2026
    4. CoreWeave Pricing — CoreWeave, 2026
    5. CoreWeave Kubernetes Service (CKS) — CoreWeave Docs, 2026
    6. Capacity Blocks for ML — Amazon EC2 User Guide, 2026
    7. Find and purchase Capacity Blocks — Amazon EC2 User Guide, 2026
    8. Capacity Blocks pricing and billing — Amazon EC2 User Guide, 2026
    9. Compute Engine provisioning models — Google Cloud Documentation, 2026
    10. About Flex-start VMs — Google Cloud Documentation, 2026
    11. ND-H100-v5 size series — Microsoft Learn, 2026
    12. Purchasing On-Demand Instances for Amazon EC2 — Amazon EC2 User Guide, 2026
    13. Serverless pricing — Runpod Docs, 2026
    14. Serverless workers overview — Runpod Docs, 2026
    15. Endpoint configurations — Runpod Docs, 2026
    16. Cold start performance — Modal Docs, 2026
    17. Amazon EC2 instance type quotas — Amazon EC2 Instance Types Guide, 2026
    18. Specifications for Amazon EC2 accelerated computing instances — Amazon EC2 Instance Types Guide, 2026
    19. Increase VM-family vCPU quotas — Microsoft Learn, 2026
    20. GPU acceleration — Modal Docs, 2026
    21. Amazon EC2 On-Demand Pricing — Amazon Web Services, 2026
    GPUCapacity PlanningServerlessMulti-Cloud
    Will Schulz
    Will Schulz
    CEO

    Writing about cloud infrastructure, DevOps best practices, and the future of application deployment. Follow for weekly insights on multi-cloud strategies and platform engineering.

    Stay Updated

    Get the latest articles on cloud infrastructure, DevOps, and platform engineering delivered to your inbox weekly.