Why AI Applications Should Submit Workloads, Not Select GPUs

A developer is building an AI application that needs to run a GPU-backed inference job. The first implementation looks straightforward: # Simplified example provider = CloudGPUProvider(api_key=API_KEY) instance = provider.launch_instance( region="us-east", instance_type="gpu.large", gpu_model="specific-gpu-model", image="registry.example.com/inference:v1", ) provider.run_command( instance_id=instance.id, command="python inference.py --input /data/request.json", ) It works. Then the selected region runs out of capacity. The developer adds another region. The second region does not offer the same instance type, so the application needs a hardware-specific branch. Another provider has available GPUs, but its API uses a different lifecycle model. One provider expects the application to manage virtual machines. Another starts containers directly. A third exposes jobs, but returns logs and artifacts through separate services. The original inference feature gradually becomes an infrastructure orchestration system. Application code now contains: - Provider credentials - Region-selection logic - GPU-model mappings - Capacity checks - Instance lifecycle management - Startup polling - Retry rules - Fallback providers - Log collection - Artifact retrieval - Cleanup procedures The application began with a business requirement: Run this AI workload. It ended with infrastructure-specific code describing exactly where and how the workload should run. That is the wrong abstraction. AI applications should describe the workload they need executed. An infrastructure layer should decide how to satisfy that request. Instead of saying: Launch this exact GPU instance from this exact provider. Applications should be able to say: Execute this workload with these runtime, memory, latency, compatibility, and cost constraints. That shift-from instance provisioning to AI workload execution-removes infrastructure decisions from the application without pretending that hardware requirements do not matter. GPU Selection Leaks Infrastructure Complexity Into the Application Selecting a GPU is not a single decision. It often implies decisions about: - Cloud provider - Region - Availability zone - Instance family - GPU architecture - GPU memory - CPU allocation - System memory - Storage - Container runtime - Driver compatibility - Network configuration - Billing model - Machine lifecycle When an application selects an exact instance type, it inherits all the assumptions attached to that instance. The deployment code may assume that: - The instance is available in the requested region. - The provider API is responding normally. - The selected GPU has sufficient memory. - The container image is compatible with the machine. - The provider can start the workload within the expected time. - Logs can be retrieved through a known endpoint. - The application can safely retry the request. - The provider will continue offering the same instance type. Those assumptions eventually become production dependencies. Provider lock-in Provider lock-in is not limited to contracts or pricing. It also appears in code. An application that directly manages a provider’s instances becomes coupled to that provider’s: - Authentication system - Resource names - Machine lifecycle - Storage model - Networking model - Job states - Logging interfaces - Error semantics - Billing behavior Moving the workload elsewhere requires more than changing an endpoint. It requires rewriting operational logic. Capacity failures A provider can support a GPU model without having that GPU available when the application needs it. Capacity is dynamic. It varies by: - Region - Time - Provider demand - Reservation status - Account limits - Hardware supply - Maintenance events Hard-coding one provider and region turns temporary capacity shortages into application failures. Hardware-specific logic Applications sometimes require exact hardware, but many workloads require capabilities rather than product names. For example, a workload may need: - At least a certain amount of GPU memory - A supported accelerator architecture - A compatible software runtime - A maximum execution cost - A preferred startup time - A minimum CPU or storage allocation Encoding these requirements as a specific instance SKU loses the distinction between what the workload genuinely needs and what happened to work during development. Complex fallback systems Once the primary provider fails, developers begin adding fallback logic: Try provider A in region 1 | +-- unavailable --> try provider A in region 2 | +-- unavailable --> map workload to provider B | +-- incompatible GPU --> try provider C Every application team ends up rebuilding a partial scheduler. The result is usually brittle because capacity discovery, compatibility checking, retries, and provider selection are not the application’s primary concern. Operational overhead Direct provisioning also creates ongoing operational work. Someone must maintain provider SDKs, update instance mappings, monitor API changes, classify provider errors, reconcile abandoned machines, and ensure failed workloads do not continue consuming resources. This overhead becomes increasingly unpredictable as more providers and workload types are added. What a Workload Specification Should Contain A workload specification should describe the work and its constraints. It should not unnecessarily prescribe the infrastructure implementation. Depending on the workload, the specification may communicate: - The container or approved runtime - The command or entry point - Required inputs - Expected outputs - GPU memory requirements - CPU and system-memory requirements - Accelerator compatibility - Maximum execution time - Cost constraints - Latency preferences - Storage requirements - Retry policy - Environment configuration - Geographic or compliance restrictions The exact fields depend on the execution platform. The important principle is that the specification should separate requirements from placement decisions. For example: This workload requires at least 24 GB of GPU memory. is a workload constraint. Launch instance type x9-gpu-24gb in provider regionzone-a . is an infrastructure decision. The first statement leaves room for the execution layer to find compatible capacity. The second prevents the infrastructure layer from choosing an equivalent or better execution path. Before and After Before: instance-based provisioning The application selects the provider, region, instance, and lifecycle. // Simplified illustrative example const provider = new SpecificCloudProvider({ apiKey: process.env.PROVIDER_API_KEY!, }); const machine = await provider.createInstance({ region: "region-a", instanceType: "provider-specific-gpu-instance", imageId: "provider-specific-image", }); await provider.waitUntilReady(machine.id); await provider.execute(machine.id, { command: ["python", "worker.py"], environment: { INPUT_PATH: "/inputs/request.json", OUTPUT_PATH: "/outputs/result.json", }, }); const logs = await provider.getLogs(machine.id); const result = await provider.downloadFile( machine.id, "/outputs/result.json", ); await provider.terminateInstance(machine.id); The application owns the complete infrastructure lifecycle. It must also decide what happens if any step fails. After: workload-level submission The application describes what needs to run. // Pseudocode only. // This is not the verified Jungle Grid API schema. const workload = { runtime: { type: "container", image: "registry.example.com/ai-worker:v2", command: ["python", "worker.py"], }, resources: { accelerator: "gpu", minimumGpuMemoryGb: 24, cpuCores: 8, systemMemoryGb: 32, }, constraints: { maximumRuntimeSeconds: 3600, maximumEstimatedCost: 5.0, }, inputs: [ { name: "request", source: "storage://datasets/request.json", }, ], outputs: [ { name: "result", path: "/outputs/result.json", }, ], }; const execution = await executionLayer.submit(workload); The application owns the workload definition. The execution layer owns placement and execution. Intent-Based Execution Versus Instance-Based Provisioning The distinction can be summarized as follows: | Area | Instance provisioning | Workload submission | |---|---|---| | Primary request | Create a specific machine | Execute a defined workload | | Provider selection | Application | Execution layer | | Region selection | Application | Execution layer, within constraints | | Capacity discovery | Application | Execution layer | | Hardware compatibility | Encoded in instance choice | Expressed as workload constraints | | Retry behavior | Application-specific | Centralized execution policy | | Logs | Provider-specific integration | Normalized workload interface | | Artifacts | Application retrieves from machine | Associated with workload execution | | Failure handling | Rebuilt by every team | Managed by execution infrastructure | | Portability | Low | Higher | | Infrastructure coupling | Strong | Reduced | | Exact hardware control | Direct | Available through explicit constraints where supported | Workload submission does not eliminate infrastructure. It moves infrastructure decisions into a layer designed to make them. Sample Workload Request The following JSON is illustrative pseudocode. It is not presented as the current Jungle Grid API schema. { "runtime": { "type": "container", "image": "registry.example.com/batch-inference:v4", "command": [ "python", "run_inference.py" ] }, "resources": { "accelerator": "gpu", "minimum_gpu_memory_gb": 24, "cpu_cores": 8, "system_memory_gb": 32 }, "constraints": { "maximum_runtime_seconds": 7200, "maximum_estimated_cost": 10, "preferred_startup_latency_seconds": 120 }, "inputs": [ { "name": "dataset", "source": "storage://datasets/inference-batch" } ], "outputs": [ { "name": "predictions", "path": "/outputs/predictions.jsonl" } ] } The application communicates its intent: - Run a particular container. - Provide the required inputs. - Allocate sufficient resources. - Keep executi