Docker server disk full: find what fills it and free it safely

Which filesystem is full?

Start from the numbers the kernel reports for every mounted filesystem. A full disk shows Use% at or near 100% and almost nothing under Avail; containers then fail with no space left on device, databases stop writing and Docker itself may refuse to start containers.

Docker keeps its data under /var/lib/docker unless its data-root was moved; on most servers that is the root filesystem /. If you are unsure where it is, docker info --format "{{.DockerRootDir}}" prints it, and in every command below you replace /var/lib/docker with that path.

Command

df -h

Answers

• The filesystem that holds Docker's data (often /) is at or near 100%: go to “Is the space in Docker, in the logs, or elsewhere?”
• Only another filesystem is full (for example /boot or /home): go to “The space is outside Docker”
• None is near 100%, yet writes fail with "no space left on device": go to “Are the inodes used up?”

Are the inodes used up?

A filesystem has two limits: bytes and inodes. Every file and directory uses one inode, and when they run out no new file can be created, even with gigabytes free. Millions of tiny files (image layers, build cache, a cache directory of an application, session files) exhaust inodes long before they fill the bytes.

df -i shows the same filesystems counted in inodes; look at IUse% on the filesystem where writes fail.

Command

df -i

Answers

• IUse% is at or near 100%: go to “The filesystem ran out of inodes”
• IUse% is well below 100%: go to “No branch matched: collect what you need to dig further”

Is the space in Docker, in the logs, or elsewhere?

Before deleting anything, find who holds the space. This command sums the three usual suspects: Docker's data directory, containerd's (where the containerd image store keeps image data), and the system logs under /var/log. A directory that does not exist on your server is skipped silently.

Compare the sizes with the Used column that df -h showed for the same filesystem. du reads every file, so on a large directory it can take a minute.

Command

sudo du -sh /var/lib/docker /var/lib/containerd /var/log 2>/dev/null

Answers

• /var/lib/docker (or /var/lib/containerd) holds most of the used space: go to “Where inside Docker's data?”
• /var/log holds most of it: go to “Is it the system journal?”
• Together they hold far less than df shows as used: go to “Find the largest directories”

Find the largest directories

The space is outside Docker and the logs. List the largest directories of the full filesystem, two levels deep; -x keeps du on that filesystem and does not count other mounts under it.

If even the largest directories add up to much less than Used, the space is held by files that were deleted but are still open: du no longer sees them, while the kernel still counts them.

Command

sudo du -xh --max-depth=2 / 2>/dev/null | sort -h | tail -n 15

Answers

• A directory outside Docker and the logs is large: go to “The space is outside Docker”
• The directories add up to much less than df's Used: go to “Deleted files are still held open”

Where inside Docker's data?

Docker's data directory splits by kind: containers holds each container's configuration and its log file, overlay2 holds image layers, the writable layers of containers and the build cache, and volumes holds the data of named and anonymous volumes. The largest one points to the cause.

The command lists the first level of /var/lib/docker, smallest to largest. If the space is under /var/lib/containerd instead, image data and build cache are there.

Command

sudo du -h --max-depth=1 /var/lib/docker | sort -h

Answers

• containers is the largest: go to “Are container logs the cause?”
• overlay2 is the largest, or the space is under /var/lib/containerd: go to “What does Docker itself count?”
• volumes is the largest: go to “Which volumes, and are they used?”

Are container logs the cause?

With the default json-file logging driver, Docker writes everything a container prints to a file next to its configuration, and by default it never rotates or caps that file. One chatty container can write gigabytes a day. docker system df does not count these files, which is why they are easy to miss.

The command runs as root in a shell because only root can read the containers directory; it lists the five largest log files. The directory name in each path is the container's full ID.

Command

sudo sh -c 'du -h /var/lib/docker/containers/*/*-json.log' | sort -h | tail -n 5

Answers

• One or more *-json.log files are large: go to “Container logs are not rotated”
• The log files are small, or there are none (another logging driver): go to “What does Docker itself count?”

What does Docker itself count?

docker system df sums Docker's own objects in four rows: images, containers (their writable layers), local volumes and the build cache. RECLAIMABLE is what is not used by any container right now and could be removed.

Pick the row that holds the most. A large RECLAIMABLE on images or build cache is the easy case; a large Containers row means a container writes into its own filesystem instead of a volume.

Command

docker system df

Answers

• Images, with a large RECLAIMABLE: go to “Old and unused images fill the disk”
• Build Cache: go to “The build cache keeps growing”
• Containers: go to “A container writes into its own filesystem”
• Local Volumes: go to “Which volumes, and are they used?”
• No row is large, or images are large but nothing is reclaimable: go to “No branch matched: collect what you need to dig further”

Which volumes, and are they used?

A volume outlives the container that created it: removing a container, or docker compose down without -v, leaves its volumes in place. Over months, test stacks and old versions leave volumes no container uses, and some of them hold real data.

The verbose form lists every volume with its size and LINKS, the number of containers that use it. LINKS 0 means no container, running or stopped, uses the volume now; it does not mean the data in it is worthless.

Command

docker system df -v

Answers

• Large volumes have LINKS 0 (no container uses them): go to “Volumes no container uses”
• The large volume is used by a container I run: go to “The data of a running service keeps growing”

Is it the system journal?

On systemd servers the biggest tenant of /var/log is usually the journal in /var/log/journal. It keeps every service's output, and by default it may grow to 10% of the filesystem (up to 4 GB per journal), which on a small or already full disk is space you need.

The command prints how much the active and archived journal files take. It only reads.

Command

sudo journalctl --disk-usage

Answers

• The journal takes several gigabytes: go to “The system journal is too large”
• The journal is small; other files in /var/log are large: go to “Log files in /var/log are not rotated or removed”

Container logs are not rotated

The json-file driver keeps every line a container has printed since it was created. The lasting fix is a size cap: max-size and max-file for json-file, or the local driver, which rotates and compresses by default. A cap set in daemon.json applies only to containers created after the change, so existing containers must be recreated.

Emptying a log file frees the space now, and deletes that container's log history: whatever the log held, including the lines that explain a past incident, is gone. Copy the end of it first if you may need it.

What to do

1. Match the ID in the largest log's path to a container name.

   Command

   docker ps -a --no-trunc --format "{{.ID}} {{.Names}}"

2. Read the last lines first, and copy what you may need: the next step erases them.

   Command

   docker logs --tail 200 <container>

3. Empty that container's log file. This deletes its whole log history; the container keeps running and writing to the same file.

   Changes your server

   sudo truncate -s 0 "$(docker inspect --format '{{.LogPath}}' <container>)"

4. Cap the logs for good: set log-driver to local, or json-file with log-opts max-size (for example 10m) and max-file (for example 3), in /etc/docker/daemon.json, or per service under logging: in the Compose file.
5. After changing daemon.json, restart Docker so it reads the file. Every container stops and starts again (those with a restart policy).

   Changes your server

   sudo systemctl restart docker

6. Recreate the containers so the new log settings apply to them. Data in volumes stays; anything a container wrote into its own filesystem is lost.

   Changes your server

   docker compose up -d --force-recreate

7. Check what was freed.

   Command

   df -h

Old and unused images fill the disk

Every docker pull of a new version keeps the old one. A dangling image (<none> in the list) is an old version whose tag moved on; an unused image is one no container, running or stopped, was created from. Updating stacks for months leaves gigabytes of both.

docker image prune removes dangling images only. With -a it removes every image not used by an existing container, including ones you may want again: they are not lost for good, only pulled again on the next start, which needs the registry to be reachable and the tag to still exist.

What to do

1. See the images and their sizes.

   Command

   docker image ls

2. Remove dangling images. It deletes only untagged old versions.

   Changes your server

   docker image prune

3. If that is not enough, remove every image no container uses. Stopped containers keep their images; a stack you took down with docker compose down will pull its images again.

   Changes your server

   docker image prune -a

4. Check what was freed.

   Command

   df -h

The build cache keeps growing

Every docker build and docker compose build keeps its layers and downloaded files in the build cache so the next build is faster. On a server that builds images itself (a CI runner, a deploy that builds on the host), the cache grows with every change, and only its garbage collection limits it.

Removing build cache never touches images, containers or volumes: the next build is slower, because it starts without cache.

What to do

1. See what the cache holds.

   Command

   docker buildx du

2. Remove cache not used in the last week. It deletes build cache only.

   Changes your server

   docker builder prune --filter until=168h

3. Or remove all of it, if the disk is still full.

   Changes your server

   docker builder prune -a

4. Keep it in check from now on with the builder's garbage collection limits in daemon.json (the builder.gc settings).
5. Check what was freed.

   Command

   df -h

A container writes into its own filesystem

Anything a container writes outside its volumes goes into its writable layer under overlay2: uploads, a cache, logs to a file, a database without a volume. That data grows for as long as the container exists, and it is deleted with the container, so it is both a disk problem and a data-loss risk.

The fix is to find the path the application writes to and put it on a volume. Stopped containers that are no longer needed hold their writable layers too.

What to do

1. List containers with the size of their writable layer (SIZE).

   Command

   docker ps -a -s --format "{{.Names}} {{.Size}}"

2. See which files the large one added or changed (A and C).

   Command

   docker diff <container> | head -n 50

3. Mount that path as a volume in the Compose file (volumes: of the service), after copying out anything you need from it with docker cp.
4. Recreate the service with the volume. Its old writable layer, and every file in it, is deleted.

   Changes your server

   docker compose up -d --force-recreate <service>

5. Remove stopped containers you no longer need. It deletes every stopped container and its writable layer, not volumes.

   Changes your server

   docker container prune

6. Check what was freed.

   Command

   df -h

Volumes no container uses

An unused volume can be leftover test data, or the only copy of a database whose stack is down right now. Removing a volume deletes its data for good; no Docker command brings it back. Look inside each one and back up what you are unsure of before removing it.

docker volume prune removes unused anonymous volumes only (since Docker Engine 23); with -a it also removes every unused named volume. Prefer removing the volumes you checked, by name.

What to do

1. List the volumes no container uses.

   Command

   docker volume ls --filter dangling=true

2. See which Compose project created a volume (its labels) and when.

   Command

   docker volume inspect <volume>

3. Look at what it holds.

   Command

   sudo ls -la /var/lib/docker/volumes/<volume>/_data

4. Back up any volume whose data you may need, before removing it.
5. Remove a volume you checked. Its data is deleted for good.

   Changes your server

   docker volume rm <volume>

6. Or remove every unused anonymous volume at once. Containers started without a named volume (a database image with a VOLUME) keep their data in exactly such volumes: check the list above first.

   Changes your server

   docker volume prune

7. Check what was freed.

   Command

   df -h

The data of a running service keeps growing

The space belongs to an application you run: its database, uploads, media, or backups it writes into its own volume. Nothing in Docker should be removed here; the cleanup happens in the application (retention, old backups, a vacuum of the database) or by giving the data a larger disk.

Find which directory inside the volume grows, then use the application's own documented way to clean it. Deleting files from under a running database corrupts it.

What to do

1. See which directory inside the volume is large.

   Command

   sudo du -h --max-depth=2 /var/lib/docker/volumes/<volume>/_data | sort -h | tail -n 10

2. If it is backups or exports the application writes, move them off the server and set how many it keeps.
3. If it is the database itself, use its documented cleanup (for example retention settings or a vacuum), never rm on its files.
4. If the data is simply larger than the disk, plan a larger disk or move Docker's data-root to one; this wizard does not cover growing a disk at a provider.

The system journal is too large

The journal deletes old entries only when it reaches its limit, and the default limit is a share of the filesystem, not a fixed size. Vacuuming removes the oldest archived journal files down to the size you give; the active file is kept, so recent entries stay.

Then set a fixed cap, so the journal cannot fill the disk again: SystemMaxUse= in /etc/systemd/journald.conf, applied after journald restarts.

What to do

1. Delete the oldest archived journal files until the journal takes at most 500 MB. Older system logs are gone after this.

   Changes your server

   sudo journalctl --vacuum-size=500M

2. Cap it: set SystemMaxUse=500M in the [Journal] section of /etc/systemd/journald.conf.
3. Restart journald so it reads the new limit. Logging pauses for a moment.

   Changes your server

   sudo systemctl restart systemd-journald

4. Confirm the size.

   Command

   sudo journalctl --disk-usage

Log files in /var/log are not rotated or removed

Most services that log to files rely on logrotate to rename, compress and eventually delete old files. A service without a logrotate rule, or one that ignores it, grows a single file without end; rotated copies (.1, .gz) pile up when the number kept is high.

Find the large files first. Removing old compressed copies frees space without touching what services write now.

What to do

1. List the largest files and directories under /var/log.

   Command

   sudo du -ah /var/log 2>/dev/null | sort -h | tail -n 15

2. Delete compressed rotated logs older than 30 days. They are gone for good.

   Changes your server

   sudo find /var/log -type f -name "*.gz" -mtime +30 -delete

3. See what logrotate would do, without doing it, to find a file no rule covers.

   Command

   sudo logrotate --debug /etc/logrotate.conf

4. Add a rule for that file in /etc/logrotate.d/ (with size, rotate and compress), or make the service log to the journal instead.
5. Check what was freed.

   Command

   df -h

Deleted files are still held open

Deleting a file removes its name, but the kernel frees its space only when the last process that has it open closes it. A log file deleted by hand while a service still writes to it keeps growing out of sight: df counts it, du cannot find it.

lsof +L1 lists open files that no longer have a name, with the process holding each. Restarting that process, or the container it runs in, releases the space.

What to do

1. List deleted files still open, with their size and process (install lsof if it is missing).

   Command

   sudo lsof -nP +L1

2. If the process is a system service, restart it. The service stops for a moment.

   Changes your server

   sudo systemctl restart <service>

3. If it runs in a container, restart that container.

   Changes your server

   docker restart <container>

4. Next time, empty a log a process still writes to instead of deleting it, or let logrotate handle it.
5. Check what was freed.

   Command

   df -h

The filesystem ran out of inodes

Inodes are freed only by deleting files, so the fix is to find the directory with the most files. On a Docker server it is often overlay2 (many images, or a build cache with many small layers); elsewhere it is a cache, a mail queue or session files of an application.

If the count is in Docker's directory, removing unused images and build cache frees inodes the same way it frees bytes.

What to do

1. Count files per directory, three levels deep, on the full filesystem.

   Command

   sudo du --inodes -x --max-depth=3 / 2>/dev/null | sort -n | tail -n 10

2. If it is under /var/lib/docker: remove dangling images (old untagged versions only).

   Changes your server

   docker image prune

3. And build cache older than a week (build cache only).

   Changes your server

   docker builder prune --filter until=168h

4. If it is elsewhere, clean that directory with the tool of the application that owns it.
5. Check the inodes again.

   Command

   df -i

The space is outside Docker

Docker is not what fills this filesystem: it is a directory of the system or of a user, such as old backups, downloads, a home directory, or old kernels on a small /boot. Pruning Docker objects would delete things without freeing this space.

List the largest directories of that filesystem and remove only what you know is no longer needed; for packages and kernels, use the package manager of your distribution, not rm.

What to do

1. List the largest directories of the full filesystem; replace <mount> with its mount point from df -h.

   Command

   sudo du -xh --max-depth=2 <mount> 2>/dev/null | sort -h | tail -n 15

2. Move old backups and archives off the server before you delete them here.
3. For old kernels and package caches, use your distribution's package manager.

No branch matched: collect what you need to dig further

The answers so far do not point at one place. Collect the full picture before deleting anything: the space and inodes of every filesystem, what Docker counts, where Docker keeps its data, and what the kernel said when writes failed. That is also what a forum or an issue tracker will ask for.

All the commands below only read. docker info prints the server's configuration but no secrets; still look over any output before you share it.

What to do

1. Space and inodes of every filesystem.

   Command

   df -h && df -i

2. Docker's own count, per image, container and volume.

   Command

   docker system df -v

3. Where Docker keeps its data, and its storage and logging drivers.

   Command

   docker info --format "{{.DockerRootDir}} {{.Driver}} {{.LoggingDriver}}"

4. The size of Docker's and the system's directories.

   Command

   sudo du -sh /var/lib/docker /var/lib/containerd /var/log 2>/dev/null

5. Kernel messages of the last hour (filesystem errors, a read-only remount).

   Command

   journalctl -k --since "1 hour ago"