Published

LocalAI

Self-hosted inference server with an OpenAI-compatible API and a web UI, here on the CPU image running open-weight GGUF models from its gallery. Fits teams that want existing OpenAI clients to talk to a model on their own server.

Resources by load level

What the app itself needs at each level, next to what we recommend for a server that runs only this app.

Level	What it means	App requirements	Recommended
LightEstimate	Model up to 4B params at Q4_K_M (≤2.5 GB file, e.g. gallery qwen3-4b), 8,192-token context, 1 request at a time, one model loaded.	2.5 vCPU · 5 GiB RAM · 3.589 GiB disk	3.12 vCPU · 7.5 GiB RAM · 13.443 GiB disk
MediumEstimate	Model up to 8B params at Q4_K_M (≤5.1 GB file, e.g. gallery qwen3-8b), 8,192-token context, 1 request at a time, one model loaded.	4.5 vCPU · 7.5 GiB RAM · 5.946 GiB disk	5.52 vCPU · 10.5 GiB RAM · 16.153 GiB disk
HeavyEstimate	Model up to 15B params at Q4_K_M (≤9.1 GB file, e.g. gallery qwen3-14b), 8,192-token context, 1 request at a time, one model loaded.	8 vCPU · 11.5 GiB RAM · 9.647 GiB disk	9.72 vCPU · 15.3 GiB RAM · 20.41 GiB disk

Recommended = app requirements + system baseline + safety headroom, under sizing policy 2026.10.0. Options stay at their defaults. “Estimate” means the sources give no number for that case, so it is worked out from them by the method stated under Components. How the numbers are built

Size it in the Stack Sizer

Ready-to-run files

Support level Supported

Compose template from the official sources; not yet verified by a run.

Create docker-compose

Components

What each component needs at each load level, before baseline and headroom.

vCPU / GiB RAM / GiB disk
Component	Light	Medium	Heavy	Sources
LocalAI (model-runtime)	2.5 / 5 / 3.589	4.5 / 7.5 / 5.946	8 / 11.5 / 9.647	[9][10][14][15][16][17][18][21][22][23][24][25][26][27]

How the estimates are worked out

LocalAI · Light: RAM: model 2,497,280,672 B + f16 KV cache 2 × 36 layers × 8 KV heads × 128 × 8,192 ctx × 2 B = 1,207,959,552 B + one idle slot saved in the prompt cache (same size) + LocalAI's overhead 0.04 × model + 0.15 GB = 249,891,227 B; total 4.809 GiB, rounded up to 5. vCPU: one thread per physical core; 8 threads (docs example) at 14.8B × 4.0/14.8 = 2.16, rounded up to 2.5. Disk: image 1.102 GiB (0.285 compressed + 0.818 unpacked) + llama-cpp backend 0.161 + model 2.326 = 3.589.
LocalAI · Medium: RAM: model 5,027,783,904 B + f16 KV cache (36 layers, 8 KV heads, 128, 8,192 ctx) 1,207,959,552 B × 2 (live + one saved slot) + overhead 0.04 × model + 0.15 GB = 351,111,356 B; total 7.259 GiB, rounded up to 7.5. vCPU: 8 threads at 14.8B × 8.2/14.8 = 4.43, rounded up to 4.5. Disk: image 1.102 GiB + backend 0.161 + model 4.683 = 5.946.
LocalAI · Heavy: RAM: model 9,001,753,344 B + f16 KV cache 2 × 40 layers × 8 KV heads × 128 × 8,192 × 2 B = 1,342,177,280 B × 2 (live + one saved slot) + overhead 0.04 × model + 0.15 GB = 510,070,134 B; total 11.359 GiB, rounded up to 11.5. vCPU: 8 threads, the docs example (one thread per physical core). Disk: image 1.102 GiB + backend 0.161 + model 8.384 = 9.647.
extra-model · LocalAI: Disk: Qwen3-8B.Q4_K_M.gguf 5,027,783,904 B = 4.683 GiB (rounded up). No RAM or vCPU added: the template sets LOCALAI_MAX_ACTIVE_BACKENDS=1, so the least recently used model is evicted when another loads.

Options that change resources

extra-model

Keep a second model of up to 5.1 GB (e.g. qwen3-8b at Q4_K_M) installed next to the main one. The template loads one model at a time: requesting the other one unloads the first.

Default: off

When on: LocalAI +0 vCPU · +0 GiB RAM · +4.683 GiB disk [6][22]

Ports

Public ports are meant to be reached from outside the stack, usually through a reverse proxy; internal ones stay on the stack's own network.

Component	Port	Exposure	Purpose	Sources
LocalAI	8080/tcp	Public, through a proxy	OpenAI-compatible API and web UI (HTTP), behind a reverse proxy, API key required	[1][5][6]

Every port in the catalog, with conflicts between apps

User and permissions (UID/GID)

The user each container's process runs as, and the paths it writes to. A bind mount for one of these paths must be writable by that UID:GID.

Component	UID:GID	User	Starts as root	Writes to	Sources
LocalAIDerived	0:0	`root`	No	`/models/backends/configuration/data`	[2][4][9]

LocalAI: The final image stage sets no USER (the image config has none), so entrypoint.sh and local-ai run as root without any step-down. Models go to /models, downloaded backends to /backends, api_keys.json and runtime settings to /configuration, and agents, collections and jobs to /data.

Storage growth

How much each component's data grows per month at each load level, on top of the disk in the profile. A value marked Assumption is ANAROQ's stated assumption, not a figure from the sources; the notes say what it rests on.

GiB per month
Component	Light	Medium	Heavy	Sources
LocalAIAssumption	0	0	0	[4][21][22][23]

What the growth figures rest on

LocalAI: Each level keeps a fixed model and backend, already in the preset disk; /models and /backends grow only when another model or backend is installed. /configuration and /data hold small JSON and SQLite files for API use; agents and collections are not sized here.
LocalAI: Fixed model and backend, already in the preset disk; no growth unless another model is installed or agents and collections are used.

Upgrade and rollback

How to move to a new version safely, and how to go back if it fails.

Back up first

Back up the configuration and data volumes (/configuration, /data) and any model YAML files you edited in /models before every update. [4][19]DerivedLocalAI publishes no upgrade-backup guide; these are the stateful paths the container docs name, and the models docs advise saving YAML configurations.

Upgrade steps

Read the release notes of every version between yours and the target, especially changes to authentication and volumes. [28][29]
Set the new pinned tag localai/localai:vX.Y.Z in the Compose file, then run docker compose pull and docker compose up -d. The four volumes keep models, backends, settings and data. [4][20]
Upgrade the installed backends (e.g. llama-cpp) under Operate → Backends: they are separate OCI images in /backends and are not replaced by the new LocalAI image. [11][12]

Roll back

Pin the previous image tag and run docker compose up -d. If the newer version changed settings or data, restore the /configuration and /data backups taken before the update. [4][8]DerivedNo official rollback guide; this follows from pinned version tags and the documented stateful volumes.

Known breaking changes

In 4.9.0: Routes deny by default when API keys or user auth are set: /version and the URLs of generated audio, images, video and 3D assets now need credentials. [29]
In 4.0.0: AIO images with bundled models are dropped and the HuggingFace backend is removed; persistent data moved to its own path (LOCALAI_DATA_PATH, /data in the container), which needs its own volume. [4][30]

Limits

CPU only: this profile uses the CPU image and no GPU. Generation speed depends on CPU cores and memory speed and drops as the model grows; by default LocalAI uses as many threads as processors, while the docs advise matching physical cores. [4][16][20]
Models above the heavy level (over ~15B parameters or 9.1 GB at Q4_K_M), longer contexts than 8,192 tokens, or more than one parallel slot are not covered: each raises RAM (the KV cache grows linearly with context and slots). [14][17][23]
The llama.cpp prompt cache is on by default and may keep up to 8,192 MiB of saved prompt state in host RAM; the presets count one saved slot. With many distinct prompts, set cache_ram lower (or 0) in the model YAML. [17][18]
The image ships no inference backend: installing the first model downloads the llama-cpp backend from the backend gallery (the floating latest-cpu-llama-cpp tag), so the server needs internet access, and the backend version is not pinned by this file. [11][12]

Security

Without an API key or user authentication LocalAI restricts no request, including model and backend installs. Keep LOCALAI_API_KEY set and serve port 8080 only through an HTTPS reverse proxy; LocalAI does not terminate TLS. [5][7]
A legacy API key grants full admin access with no roles: whoever holds it can install backends (executable OCI images) and change settings. For several people with separate rights, use LocalAI's user authentication (LOCALAI_AUTH), which this template does not set up. [5][12]
The container runs as root (the image sets no USER), and so do the backends it downloads. Keep docker.sock and host paths out of it. [2][9]

Backup

Back up the configuration and data volumes: /configuration holds api_keys.json, runtime_settings.json and external_backends.json; /data holds collections, agent state, tasks, jobs and the user database (database.db) when user auth is on. [4][5]
Model files in /models and backends in /backends can be installed again from the galleries, but keep a copy of model YAML files you wrote or edited. [12][19]