How to get incremental btrfs send/receives to work?

[yorickpeterse]

I have two servers running a bootc Fedora installation: "web" and "deskmini". Both use btrfs. On the "deskmini" I have a script that's supposed to create a btrfs snapshot of "web", then download it to "deskmini". If a previous snapshot exists, it sends an incremental snapshot. This is done using ssh 'btrfs send SNAP' | btrfs receive TARGET and ssh 'btrfs send -p PREV SNAP' | btrfs receive TARGET.

It seems that something in bootc (likely the use of composefs/the exotic mount setup) confuses btrfs. Sending an receiving an initial snapshot works fine and produces the same IDs and all that:

# web
$ btrfs subvolume list /root
ID 256 gen 28125 top level 5 path root
ID 257 gen 28087 top level 256 path ostree/deploy/default/var/lib/shost
ID 258 gen 28079 top level 256 path ostree/deploy/default/var/lib/ssh-container
ID 259 gen 28090 top level 256 path ostree/deploy/default/var/lib/letsencrypt
ID 333 gen 28089 top level 256 path ostree/deploy/default/var/lib/snapshots/letsencrypt.a
ID 334 gen 28090 top level 256 path ostree/deploy/default/var/lib/snapshots/letsencrypt.b

# deskmini
$ btrfs subvolume list /root
ID 256 gen 108744 top level 5 path root
ID 261 gen 108744 top level 256 path ostree/deploy/default/var/lib/grafana
ID 262 gen 108744 top level 256 path ostree/deploy/default/var/lib/omada
ID 263 gen 108744 top level 256 path ostree/deploy/default/var/lib/victoria-metrics
ID 264 gen 92546 top level 256 path ostree/deploy/default/var/lib/mosquitto
ID 265 gen 108211 top level 256 path ostree/deploy/default/var/lib/jellyfin

After the initial send+receive:

# deskmini
$ btrfs subvolume list /root
ID 256 gen 108746 top level 5 path root
ID 261 gen 108745 top level 256 path ostree/deploy/default/var/lib/grafana
ID 262 gen 108746 top level 256 path ostree/deploy/default/var/lib/omada
ID 263 gen 108746 top level 256 path ostree/deploy/default/var/lib/victoria-metrics
ID 264 gen 92546 top level 256 path ostree/deploy/default/var/lib/mosquitto
ID 265 gen 108211 top level 256 path ostree/deploy/default/var/lib/jellyfin
ID 293 gen 108746 top level 256 path ostree/deploy/default/var/lib/backups/web/letsencrypt.a

However an incremental send/receive fails:

ssh web 'btrfs send -p /var/lib/snapshots/letsencrypt.a /var/lib/snapshots/letsencrypt.b' | btrfs receive /var/lib/backups/web 
At subvol /var/lib/snapshots/letsencrypt.b
At snapshot letsencrypt.b
It seems that you have changed your default subvolume or you specify other subvolume to
mount btrfs, try to remount this btrfs filesystem with fs tree, and run btrfs receive again!

Some digging around suggests this happens when setting subvol/subvolid, though the btrfs manual is horribly vague on the exact specifics. Both systems use the same setup, minus "deskmini" using luks:

# web
/dev/sda4 on /sysroot/ostree/deploy/default/var type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root)
/dev/sda4 on /var type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root)

# deskmini
/dev/mapper/luks-4c9af5ab-cf78-4a27-8e0f-3eb5522f554d on /sysroot/ostree/deploy/default/var type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root)
/dev/mapper/luks-4c9af5ab-cf78-4a27-8e0f-3eb5522f554d on /var type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root)

Does anybody know a way around this, short of not using incremental sends and receives? I've messed around with trying to use different paths for the receive target and what not, but nothing seems to work.

[yorickpeterse]

The error message was first introduced in kdave/btrfs-progs@b5a0977, but reading the commit and changes it's not clear exactly what is wrong with the above mount setup.

[yorickpeterse]

To further clarify: the btrfs receive works fine on a regular Fedora 43 workstation setup.

[yorickpeterse]

I patched btrfs-progs as follows:

diff --git a/cmds/receive.c b/cmds/receive.c
index 3e3b989c..daf5d2c8 100644
--- a/cmds/receive.c
+++ b/cmds/receive.c
@@ -370,6 +370,7 @@ static int process_snapshot(const char *path, const u8 *uuid, u64 ctransid,
                                "subvolume or you specify other subvolume to\n"
                                "mount btrfs, try to remount this btrfs filesystem "
                                "with fs tree, and run btrfs receive again!\n");
+               fprintf(stderr, "path: %s\n", parent_subvol->path);
                goto out;
        }
 

When doing a btrfs receive the output is then as follows:

At subvol /var/lib/snapshots/letsencrypt.1777258148
At snapshot letsencrypt.1777258148
It seems that you have changed your default subvolume or you specify other subvolume to
mount btrfs, try to remount this btrfs filesystem with fs tree, and run btrfs receive again!
path: ostree/deploy/default/var/lib/backups/web/letsencrypt.1777257910

This has me slightly more confused: I was expecting it to perhaps resolve to the wrong path and therefor fail to open things, but it seems the path matches that of the parent snapshot?

[yorickpeterse]

I gave https://github.com/rustutils/btrfsutils a shot to see how it would fare. While it seems to be vibe coded (based on the commit messages and high number of commits in such a short time) and thus I didn't have much faith in it, it felt worth trying.

This tool fails with the following:

At subvol /var/lib/snapshots/letsencrypt.1777258148
ERROR: cannot open parent subvolume '/var/root/ostree/deploy/default/var/lib/backups/web//letsencrypt.1777257910': No such file or directory (os error 2)

So it too gets confused, though it's hard to say how much the use of vibe coding is influencing this.

[yorickpeterse]

I created kdave/btrfs-progs#1115 to see what (if anything) can be done on the btrfs-progs side of things.