After booting my openSUSE Tumbleweed laptop this morning, I was greeted with the following error messages in the console log:
It looked like the
/var/log btrfs filesystem had run out of free
space, and that systemd could not create new files for its journal.
Thinking this was a run of the mill case of using up all my disk space
– probably because of some huge log file – I reached for
df(1), and printed the filesystem statistics.
What I saw was totally unexpected:
Say what? If only 91% of the filesystem was in use, then why was I hitting out of space errors?
Because, while there was plenty of available data space, I had exhausted the free metadata space.
Reclaim Metadata Space with snapper
To fix this situation, so that I could create new files again, I had
to delete most of the btrfs snapshots for the
The default installation of openSUSE Tumbleweed creates a single btrfs
filesystem for the root filesystem (
/), with btrfs subvolumes for
various other directories. Running out of free metadata space in a
/var/log) means you’ve run out of space in the main
You can query the snapshots of the root filesystem using the
Every time I update packages with
zypper update, two btrfs snapshots
are created. Because openSUSE Tumbleweed is a rolling
release version of
openSUSE, this happens a lot.
To free up some metadata space I deleted every snapshot apart from the
#0) and initial snapshot (snapshot
WARNING: The whole point of creating snapshots is to allow
zypperto rollback to previous states if you encounter issues. That will become impossible if you follow the next step.
After that, it was possible to create files in
/var/lib again, and
systemd was much happier.
If you’re curious to know how I debugged this issue, and how you can do the same for your btrfs filesystems, read on.
Getting Serious with Btrfs Utilities
One of the most surprising things about this whole ordeal was that
btrfs made zero attempts to tell me that I was hitting a metadata
ENOSPC issue; there was no error message of any kind in the console
I think it’s a natural instinct to pull out
df(1) when you hit out
of space issues. But when it showed that I had plenty of free space on
/var/log partition, I was stumped.
It turns out that btrfs has its own utilities for inspecting
partitions; all of them are described in the man page
one I needed was
btrfs fi for short.
Looking at the Metadata line shows that there’s roughly 0.44GiB of metadata space available. So why the error?
On every btrfs filesystem, some space is reserved so that the kernel can always perform critical operations, like deleting files, even when the filesystem is full. The GlobalReserve line above gives the size of this emergency space.
Crucially, the GlobalReserve space is taken from the Metadata pool, so you need to add the used Metadata size and the total GlobalReserve size to calculate how much free metadata space you actually have.
even has this very helpful equation:
In case the filesystem metadata are exhausted,
After I deleted the snapshots I saw a much better picture of filesystem statistics:
A Permanent Fix
The SUSE btrfs developers area aware of this issue. But that bug is assigned to the kernel developers because they were interested in thinking about how to make this scenario less painful at the kernel level.
Which begs the question: Shouldn’t
snapper be deleting old snapshots
so that I don’t run out of disk space and hit this issue again?
snapper can delete snapshots once a certain number have been
it does not provide a way to reduce that number if metadata space
starts to run out on the filesystem.
My only hope for a permanent fix today is to limit
keeping a single pre and a single post snapshot for important and
unimportant updates. That and pray they don’t include too many files and
Again, note that the following suggestion is going to limit how far you can rollback your system, should it stop working properly.
You can make the fix permanent by modifying
/etc/snapper/configs/root with the following settings: