This machine

Status

Supported

Kind

launchd or systemd service

Provider id

local, local_apple, local_cpu, on_prem

Regions and residency

Wherever the machine is

Use this provider

resource "hardware_profile" "dev_mac" {
  provider = "local_apple"
  accelerator {
    kind   = "apple_silicon"
    model  = "m3_pro"
    count  = 1
    memory = "36GiB"
  }
  memory = "36GiB"
}

Overview

Hardware profiles with provider set to local, local_apple, local_cpu, or on_prem run on the machine where you invoke amicu. The four ids differ in intent, not in how apply runs them:

A profile with no provider also means the local host.

These ids are supported by apply, which runs the runtime on the host you invoke it from. Modal is the only provider tested end to end on real GPUs; the local path is implemented for macOS (launchd) and Linux (systemd), as described below.

How it runs

apply renders the runtime's command line from your llama_cpp {} or vllm {} block, registers it as a per-user service, and waits up to 15 minutes for /health to answer. A service that exits during startup fails within seconds. On failure, apply stops the service and the error gives the log location.

The service label is io.amicu.llama.<deployment id>, for both llama.cpp and vLLM deployments. The server binds to 127.0.0.1 on the port you set. The host is not configurable.

When you change a deployment, apply replaces the running service; it may wait up to 10 seconds for the old one to stop. If something else still answers /health on the port, apply fails with port N is already serving /health. When you remove a deployment from the configuration, apply stops and unloads its service.

Requirements

• llama.cpp runtime: llama-server on PATH. apply resolves it from PATH to an absolute path and fails with llama-server not found on PATH; install with ... if it is missing. Run amicu doctor to confirm it is found.
• vLLM runtime: vllm on PATH; if it is missing, apply fails with vllm not found on PATH; install with pip install vllm``. A native single-process vLLM run needs a supported GPU (NVIDIA CUDA, AMD ROCm, or Intel XPU).
• Model file: for llama.cpp, a GGUF file that you download yourself; amicu does not download weights. Put it at models/<model resource id>/<variant.file> relative to the project directory, or set variant.file to an absolute path. amicu plan shows the resolved absolute path, <project dir>/models/<model id>/<file>, in the launch: line. See the Quickstart for a download command.

Full example

resource "model" "support_chat" {
  source {
    registry  = "hugging_face"
    reference = "Qwen/Qwen2.5-7B-Instruct"
  }
  architecture   = "qwen2"
  modalities     = ["text"]
  artifact {
    format       = "gguf"
    quantization = "q4_k_m"
  }
  context_length = 8192
  variant {
    quantization = "q4_k_m"
    file         = "qwen2.5-7b-instruct-q4_k_m.gguf"
    size         = "4GiB"
  }
}

resource "hardware_profile" "dev_mac" {
  provider = "local_apple"
  accelerator {
    kind   = "apple_silicon"
    model  = "m3_pro"
    count  = 1
    memory = "36GiB"
  }
  memory = "36GiB"
}

resource "deployment" "support_local" {
  model    = model.support_chat.id
  hardware = hardware_profile.dev_mac.id
  runtime  = "llama_cpp"
  api_key  = secret("LOCAL_API_KEY")
  llama_cpp {
    context_length = 8192
    port           = 8000
    n_gpu_layers   = -1
  }
}

On Apple Silicon, n_gpu_layers = -1 offloads every layer to Metal. Lower it, or use 0 for CPU only, if memory is tight.

Endpoint key

A local deployment may omit api_key; its endpoint is loopback only. If you set it, apply reads the value from the environment variable of that name and passes it to the service as LLAMA_API_KEY (llama.cpp) or VLLM_API_KEY (vLLM) through a private file, never the command line. After start, amicu confirms anonymous requests are rejected. Without an api_key, the endpoint is loopback only and that check is skipped. See Secrets.

export LOCAL_API_KEY=$(openssl rand -hex 32)
amicu apply
curl http://127.0.0.1:8000/v1/models -H "Authorization: Bearer $LOCAL_API_KEY"

Stopping by hand

If amicu cannot stop a service, apply prints the command to run:

• Linux: systemctl --user stop <label>
• macOS: launchctl unload ~/Library/LaunchAgents/<label>.plist

Configuration reference

See hardware_profile and deployment for every field.

GPUs and regions

The accelerator is whatever the machine has; amicu does not provision it. llama.cpp offloads layers with n_gpu_layers (Metal on Apple Silicon, CUDA or others per your llama-server build). A native single-process vLLM run needs a supported GPU (NVIDIA CUDA, AMD ROCm, or Intel XPU). There are no regions: the endpoint is on the machine, wherever it is.

Limits

• Loopback only: 127.0.0.1, not configurable.
• On Linux the service is a transient systemd user unit; it does not survive a reboot or logout.
• amicu does not download model weights.
• One service per deployment, labelled io.amicu.llama.<deployment id>.

Troubleshooting

Source

Supervisors: crates/amicu-cli/src/commands/apply.rs. Status of every provider: Providers.