GPU workloads

One key schedules a service on the GPU. The installer already did the hard part.

Using the GPU

app: llm
services:
  vllm:
    image: vllm/vllm-openai:v0.6.4
    args:
      - --model=meta-llama/Llama-3.1-8B-Instruct
      - --port=5000
    port: 5000
    gpu: 1
    volumes:
      hf-cache: /root/.cache/huggingface
    http:
      domain: llm.acme.com
volumes:
  hf-cache:
    size: 40Gi

gpu: 1 runs the container under the NVIDIA runtime with all devices visible. To pin specific devices on a multi-GPU box:

    gpu:
      devices: "0,2"    # exactly these GPUs

That's how you place two model servers on different cards of the same machine: each service pins its own devices.

What the installer set up

When server install detects an NVIDIA device it installs the driver (if missing) and the NVIDIA container toolkit, restarts k3s so its containerd picks up the runtime, registers the nvidia RuntimeClass, and verifies with a CUDA smoke-test pod. Preflight catches the classic traps first: Secure Boot blocking unsigned modules, and the nouveau driver squatting the card.

Checking on it

jekyo server info          # includes nvidia-smi output
jekyo ps llm               # pod status
jekyo logs llm/vllm -f     # model server logs