Modal

Status

Supported

Kind

Serverless GPU cloud

Provider id

modal

Regions and residency

Region pinning, EU included (for example eu-west)

Use this provider

resource "hardware_profile" "modal_l4" {
  provider = "modal"
  region   = "eu-west"
  accelerator {
    kind   = "gpu"
    model  = "l4"
    count  = 1
    memory = "24GiB"
  }
  memory = "32GiB"
}

Modal is one of the providers amicu supports. It is the remote GPU provider apply can deploy to today; others are in preview. It is the only provider tested end to end on real GPUs.

Overview

A hardware profile with provider = "modal" makes apply deploy the deployment to Modal as an app. Both vllm and llama_cpp runtimes work. The model must come from Hugging Face; the container downloads it.

Setup and credentials

Export these in your shell. Never write values into HCL, and never pass them as flags.

Check the control-plane token with amicu modal auth-check.

apply refuses to start a Modal deployment without the proxy tokens. It needs them to verify that the runtime enforces its own key behind Modal's proxy. Nothing is started if they are missing. See Endpoint auth.

Example: vLLM

resource "model" "qwen_fp8" {
  source {
    registry  = "hugging_face"
    reference = "RedHatAI/Qwen3.5-9B-FP8-dynamic"
  }
  architecture   = "qwen3"
  modalities     = ["text"]
  artifact {
    format       = "safetensors"
    quantization = "fp8"
  }
  context_length = 262144
}

resource "hardware_profile" "modal_2xl4" {
  provider = "modal"
  region   = "eu-west"
  accelerator {
    kind   = "gpu"
    model  = "l4"
    count  = 2
    memory = "24GiB"
  }
  memory = "64GiB"
}

resource "deployment" "qwen_tp2" {
  model    = model.qwen_fp8.id
  hardware = hardware_profile.modal_2xl4.id
  runtime  = "vllm"
  api_key  = secret("QWEN_TP2_API_KEY")
  vllm {
    context_length       = 262144
    port                 = 8000
    tensor_parallel_size = 2
    env {
      NCCL_P2P_DISABLE = "1"
    }
  }
}

accelerator.count becomes Modal's GPU string (L4:2). Amicu does not derive tensor_parallel_size from the count; set it yourself. NCCL_P2P_DISABLE = "1" works around an intermittent NCCL error: unhandled cuda error on GPUs without NVLink, such as the L4. Plain env values are baked into the image, so never put a secret there.

Example: llama.cpp

A GGUF variant.file becomes -hf <source.reference> -hff <file>. variant.size must be an integer byte size string such as "8350MiB". Amicu appends --host 0.0.0.0, --no-mmproj (unless the deployment has a multimodal {} block), and --metrics.

resource "model" "qwen35_9b_gguf" {
  source {
    registry  = "hugging_face"
    reference = "unsloth/Qwen3.5-9B-GGUF"
  }
  architecture   = "qwen3"
  modalities     = ["text"]
  artifact {
    format       = "gguf"
    quantization = "q6_k"
  }
  context_length = 32768
  variant {
    quantization = "q6_k"
    file         = "Qwen3.5-9B-UD-Q6_K_XL.gguf"
    size         = "8350MiB"
  }
}

resource "hardware_profile" "modal_l4" {
  provider = "modal"
  accelerator {
    kind   = "gpu"
    model  = "l4"
    count  = 1
    memory = "24GiB"
  }
  memory = "32GiB"
}

resource "deployment" "qwen35_modal" {
  model    = model.qwen35_9b_gguf.id
  hardware = hardware_profile.modal_l4.id
  runtime  = "llama_cpp"
  api_key  = secret("QWEN35_MODAL_API_KEY")
  llama_cpp {
    context_length = 32768
    port           = 8080
    n_gpu_layers   = -1
  }
}

Flags that refer to local files (--mmproj, --lora, --chat-template-file, a speculative {} draft model) and secret-bearing flags (--hf-token, --api-key) are rejected before any Modal call.

Configuration reference

Regions

region is optional. Pinning a region places the app there and multiplies Modal's base price (about 1.15x for broad regions, about 1.75x for narrow ones, per Modal's published region-selection guide). Each region maps to a residency jurisdiction used by policy checks.

The region also selects the proxy region of the web endpoint: eu, eu-west, eu-north, eu-south, and uk use eu-west; us-west uses us-west; the ap*, jp, and au regions use ap-south; everything else uses us-east. Modal does not allow the routing region of an existing app to change, so changing region on a deployed app requires amicu modal stop <app> and a new apply.

GPU types

Set accelerator.kind = "gpu" and accelerator.model to one of these. The name is upper-cased and passed to Modal. Only GPU accelerators are valid; tpu, cpu, and apple_silicon are rejected before any call.

The GPU string is not validated against Modal's live catalog at parse time. An unsupported name is rejected by Modal at apply time.

Resources and defaults

• The profile's memory becomes the container memory request. cpu { cores = N } becomes the CPU request.
• Defaults: startup timeout 600 s, scaledown window 900 s, container timeout 3600 s, target concurrency 32, no warm containers, no maximum. Override the autoscaling values with a scaling {} block.
• The app is named amicu- plus the deployment id, lower-cased, with runs of characters outside [a-z0-9] replaced by a single -. Deployment chat_modal becomes app amicu-chat-modal. Use this name with amicu modal commands.
• Weights and compile caches live in Modal volumes shared by all amicu deployments in your workspace (amicu-hf-cache, amicu-vllm-cache, amicu-llamacpp-cache). Only the first run of a model is slow.

Calling the endpoint

Two layers of credentials are required: Modal's proxy token and the deployment's key.

curl "$URL/v1/models" \
  -H "Authorization: Bearer $CHAT_API_KEY" \
  -H "Modal-Key: $MODAL_PROXY_TOKEN_ID" \
  -H "Modal-Secret: $MODAL_PROXY_TOKEN_SECRET"

$URL is the endpoint apply printed, or the url: line of amicu modal status. A 503 means the container is cold-starting. Thinking models spend max_tokens on reasoning, so give a generous max_tokens and check finish_reason.

Endpoint auth

Modal puts a proxy in front of the app that requires a Proxy Auth Token. amicu needs that token to prove the runtime enforces its own key behind the proxy. Without MODAL_PROXY_TOKEN_ID and MODAL_PROXY_TOKEN_SECRET, apply stops in preflight with an error that says nothing was started. Create the token in the Modal dashboard under Settings, Proxy Auth Tokens.

The check has an extra step for Modal: gateway credentials without the API key must also be rejected with 401 or 403. vLLM's --api-key protects only /v1/*; /metrics and /health stay open behind the proxy, so the proxy token is the second layer. Use both. See Endpoint auth for the full check.

With the OpenAI Python SDK, pass the proxy headers too:

client = OpenAI(
    base_url=f"{url}/v1",
    api_key=os.environ["CHAT_API_KEY"],
    default_headers={
        "Modal-Key": os.environ["MODAL_PROXY_TOKEN_ID"],
        "Modal-Secret": os.environ["MODAL_PROXY_TOKEN_SECRET"],
    },
)

Modal's proxy rejects Authorization: Bearer as proxy credentials; Modal-Key and Modal-Secret are separate headers.

Monitoring

amicu modal commands read MODAL_TOKEN_ID and MODAL_TOKEN_SECRET, and work without a project directory. <app> is the app name, amicu- plus the deployment id with _ replaced by -.

logs without --follow prints the logs available now, then returns. --follow streams until the app ends or you press Ctrl-C. --since resumes just past a log entry id. A missing app prints no deployment named <app> and exits 1.

metrics scrapes the runtime's /metrics endpoint through the app's Modal URL and needs the proxy token pair. A scaled-to-zero app is woken by the request, so the first call can be slow. For vLLM it shows request counts, token counts, KV cache usage, preemptions, and latency percentiles; for llama.cpp it shows request and token counters and tokens per second (latency percentiles are n/a). Tokens per second needs two scrapes, so use --watch. Per-container hardware metrics (utilization, memory, temperature, power) come from a read-only probe in each container; if it fails, they show n/a. See Monitoring for local services.

Limits

• Only GPU accelerators are valid; tpu, cpu, and apple_silicon are rejected before any call.
• The model must come from Hugging Face; the container downloads it.
• The routing region of an existing app cannot change.
• Flags that refer to local files and secret-bearing flags are rejected before any Modal call.
• Without scaling.max_replicas, a burst of requests can start many GPU containers.

Troubleshooting

Source

Adapter: crates/amicu-providers/src/modal.rs. Dispatch: apply.rs. Status of every provider: Providers.