Use tmpdir within /data for barman-cloud-backup/-restore

[immerrr]

We are having an issue with postgres backups created by barman: they are failing, because barman creates a tar archive in /tmp directory, which only has ~7.5G to work with, so it runs out of space.

# df -h
Filesystem      Size  Used Avail Use% Mounted on
none            7.8G   48M  7.4G   1% /
/dev/vdb        7.8G   48M  7.4G   1% /.fly-upper-layer
shm             2.0G  3.1M  2.0G   1% /dev/shm
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/vdc        147G   20G  121G  14% /data

I've checked barman sources, and it didn't seem to have a way to configure its temp. directories, so I ended up making a wrapper script that creates a temporary directory in a place of your choice, and configured barman-cloud-backup and barman-cloud-restore commands to run through that script.

There should be a better way to handle this, but I wanted to get some early opinions on this.

Also, had to fix the haproxy configs, it seemed to complain about having two server-template clauses with the same name pg

[dangra]

hey @immerrr, I am fine with the wrapper script.

The PR needs a rebase, thanks for the pg17 and haproxy fixes btw.

[kubosuke]

different situation but having same trouble: since the root disk is only 8GB and .fly-upper-layer consumes a lot somehow, we cannot make barman backup

❯ f ssh console -a foobar --machine 2871961ae61438
Connecting to fdaa:2:be18:a7b:73:eb16:7ce8:2... complete
root@2871961ae61438:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
none            7.8G  7.2G  221M  98% /
/dev/vdb        7.8G  7.2G  221M  98% /.fly-upper-layer
shm             2.0G   35M  1.9G   2% /dev/shm
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/vdc         40G  7.8G   30G  22% /data

here's the entire logs of barman-cloud-backup

root@28714d6a6e7d28:/# cat /root/result.log
2026-02-11 11:21:56,801 [32706] INFO: Found credentials in shared credentials file: /data/.aws/credentials
2026-02-11 11:21:57,131 [32706] INFO: Starting backup '20260211T112157'
2026-02-11 11:26:22,719 [32706] INFO: Uploading 'pgdata' directory '/data/postgresql' as 'data.tar'
2026-02-11 11:26:22,933 [2776] INFO: Upload process started (worker 0)
2026-02-11 11:26:22,951 [2778] INFO: Upload process started (worker 1)
2026-02-11 11:26:22,982 [2776] INFO: Found credentials in shared credentials file: /data/.aws/credentials
2026-02-11 11:26:23,016 [2778] INFO: Found credentials in shared credentials file: /data/.aws/credentials
2026-02-11 11:26:23,202 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '1' (worker 1)
2026-02-11 11:26:23,235 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '2' (worker 0)
2026-02-11 11:26:24,084 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '3' (worker 0)
2026-02-11 11:26:24,867 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '4' (worker 1)
2026-02-11 11:26:25,065 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '5' (worker 0)
2026-02-11 11:26:25,310 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '6' (worker 1)
2026-02-11 11:26:25,542 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '7' (worker 0)
2026-02-11 11:26:25,803 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '8' (worker 1)
2026-02-11 11:26:25,950 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '9' (worker 0)
2026-02-11 11:26:26,353 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '10' (worker 1)
2026-02-11 11:26:26,699 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '11' (worker 0)
2026-02-11 11:26:27,095 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '12' (worker 1)
2026-02-11 11:26:28,087 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '13' (worker 0)
2026-02-11 11:26:29,815 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '14' (worker 1)
2026-02-11 11:26:31,967 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '15' (worker 0)
2026-02-11 11:26:34,104 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '16' (worker 1)
2026-02-11 11:26:36,567 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '17' (worker 0)
2026-02-11 11:26:39,000 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '18' (worker 1)
2026-02-11 11:26:41,583 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '19' (worker 0)
2026-02-11 11:26:44,060 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '20' (worker 1)
2026-02-11 11:26:46,715 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '21' (worker 0)
2026-02-11 11:26:49,447 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '22' (worker 1)
2026-02-11 11:26:51,847 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '23' (worker 0)
2026-02-11 11:26:54,338 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '24' (worker 1)
2026-02-11 11:26:57,011 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '25' (worker 0)
2026-02-11 11:26:59,499 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '26' (worker 1)
2026-02-11 11:27:02,071 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '27' (worker 0)
2026-02-11 11:27:04,643 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '28' (worker 1)
2026-02-11 11:27:07,239 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '29' (worker 0)
2026-02-11 11:27:09,811 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '30' (worker 1)
2026-02-11 11:27:12,430 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '31' (worker 0)
2026-02-11 11:27:15,155 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '32' (worker 1)
2026-02-11 11:27:17,715 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '33' (worker 0)
2026-02-11 11:27:20,075 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '34' (worker 1)
2026-02-11 11:27:22,530 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '35' (worker 0)
2026-02-11 11:27:25,431 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '36' (worker 1)
2026-02-11 11:27:27,902 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '37' (worker 0)
2026-02-11 11:27:30,263 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '38' (worker 1)
2026-02-11 11:27:34,304 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '39' (worker 0)
2026-02-11 11:27:36,954 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '40' (worker 0)
2026-02-11 11:27:38,662 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '41' (worker 1)
2026-02-11 11:27:40,938 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '42' (worker 0)
2026-02-11 11:27:43,314 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '43' (worker 1)
2026-02-11 11:27:45,726 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '44' (worker 0)
2026-02-11 11:27:48,262 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '45' (worker 1)
2026-02-11 11:27:50,942 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '46' (worker 0)
2026-02-11 11:27:53,495 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '47' (worker 1)
2026-02-11 11:27:56,174 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '48' (worker 0)
2026-02-11 11:27:58,659 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '49' (worker 1)
2026-02-11 11:28:01,421 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '50' (worker 0)
2026-02-11 11:28:04,306 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '51' (worker 1)
2026-02-11 11:28:07,366 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '52' (worker 0)
2026-02-11 11:28:10,294 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '53' (worker 1)
2026-02-11 11:28:13,467 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '54' (worker 0)
2026-02-11 11:28:16,170 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '55' (worker 1)
2026-02-11 11:28:18,444 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '56' (worker 0)
2026-02-11 11:28:22,754 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '57' (worker 1)
2026-02-11 11:28:27,681 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '58' (worker 0)
2026-02-11 11:28:32,860 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '59' (worker 1)
2026-02-11 11:28:38,581 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '60' (worker 0)
2026-02-11 11:28:44,041 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '61' (worker 1)
2026-02-11 11:28:49,249 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '62' (worker 0)
2026-02-11 11:28:54,469 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '63' (worker 1)
2026-02-11 11:28:59,569 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '64' (worker 0)
2026-02-11 11:29:04,773 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '65' (worker 1)
2026-02-11 11:29:09,954 [2776] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '66' (worker 0)
2026-02-11 11:29:15,129 [2778] INFO: Uploading 'foobar-db/base/20260211T112157/data.tar', part '67' (worker 1)
2026-02-11 11:29:16,652 [32706] ERROR: Backup failed uploading data ([Errno 28] No space left on device)
2026-02-11 11:29:16,653 [32706] INFO: Uploading 'foobar-db/base/20260211T112157/backup.info'
2026-02-11 11:29:16,878 [2776] INFO: Upload process stopped (worker 0)
2026-02-11 11:29:16,885 [2778] INFO: Upload process stopped (worker 1)

the annoying thing is, postgres user tried to re-run barman-cloud-backup over and over again if it failed, and it won't cleanup /tmp files when it died on the way.

so we needed to run barman-cloud-backup with different user like this:

/usr/bin/python3 /usr/bin/barman-cloud-backup --verbose --cloud-provider aws-s3 --endpoint-url https://fly.storage.tigris.dev --profile barman --host foobar-db.internal --user repmgr s3://foobar-db-postgres foobar-db --read-timeout 120 > /root/result2.log 2>&1 &

and kill tmp file periodically, to avoid disk full

while true; do find /tmp -maxdepth 1 -type d -name 'barman-cloud-backup-*' -user postgres -exec rm -rf {} +; sleep 10; done

ps. I found we can change rootfs size; will try this when I encounter this issue again

❯ fly machine update --help | grep root
      --rootfs-persist string       Whether to persist the root
      --rootfs-size int             Root filesystem size in GB. Uses an
                                    overlayfs to allow the root