How do incremental and deduplicated backups work?
After the first backup, BackupWall reads only files that changed, cuts large files into chunks, and uploads only chunks the destination does not already hold. Identical data is stored once, however many points or accounts reference it.
Incremental: how changed files are found
A file watcher tracks which files changed since the last backup point. An Incremental backup looks only at the files the watcher flagged, so it is the fastest. A file whose size and dates are unchanged is never read again.
On kernels 5.9 and newer the watcher uses the Linux fanotify feature. On older kernels (AlmaLinux 8) it falls back to inotify, which covers about 4,000 folders; beyond that BackupWall scans the rest, which is correct but slower. On very small servers (1 CPU or under 1 GB of RAM) BackupWall only compares file details and does not hash.
Full versus Incremental
Both skip a file whose size and dates are unchanged. They differ in how they find the files to look at:
- Incremental only looks at files the watcher flagged.
- Full ignores the watcher and checks every file of the account against what the last backup recorded, so a change the watcher missed is still caught. It does not read all the data again: only a changed or new file is read, and only data not already stored is uploaded. It is slower than Incremental because it walks every file; the cost grows with the number of files, not their size. In PortWall's test, a Full of a 13 GB account with no changes took under 25 seconds.
Run now is always a Full backup, and so is a job with no schedule.
The 30-day backstop
Either type also fully re-reads every file about once every 30 days of backups, a slice of the account each day, so a change that its dates did not show cannot stay out of your backups for long. A weekly job takes about 30 weeks to get through the whole account.
Deduplication
Each file is stored as objects named by a hash of their contents. Large files are cut into content-defined chunks of up to 4 MiB, small files are bundled together, and every object is compressed. An object the destination already holds is not uploaded again. This is why a second point of the same account costs little, why identical files across accounts are stored once, and why a moved or renamed file costs almost nothing.
A change to only a file's mode, owner or timestamp reaches the next point without re-reading the file.
Pack files on remote destinations
On remote destinations (S3 and compatible storage, SFTP, Google Drive and others) small objects are grouped into pack files of about 32 MB instead of one file per object. This makes backups on high-latency links much faster. A local destination does not use packs. Because storage is content-addressed, remote data is not browsable as ordinary files: you need BackupWall, or the read-only bw-extract tool, to read it. See Get your files out without BackupWall.
What the first backup costs
The first backup of an account reads everything. Later backups read only what changed.