Restic Backup Questions

brent

I've gone heavily (for me) through the Restic backup man and web pages, and nowhere are my questions answered by the Restic people. So here we go.

Current backup: rsync to FAT partition on external drive. I don't like this anymore.

Desired: images made by restic to a partition on an external drive.

Here's my questions.

1)If I'm backing up mostly/home stuff, and I backup twice a month, how big should I make my external drive partition? Size is no issue. This external is enormous.

2) When I 'init' to create my server on this partition for the first time....should I format it for restic? If so, how do I format the external partition 'restic-friendly'? Ext4? Fat? Or let Restic do it?
Or when I setup repo on partition does Restic do all this the way it likes to?

If I can answer these questions then I can go forward, and as far as the commands, I think I can take it from there. 'How' to use this is very well documented. Preparing to use it is not. Not explained in youtube videos either since they are all nas or cloud backups demo'd.
Thanks anyone.

WetGeek

brent Not explained in youtube videos either since they are all nas or cloud backups demo'd

I'm not so sure that the NAS examples are totally irrelevant to your needs, and you may be overthinking this a bit. I backup to my Synology NAS, but it's just a small Linux server that serves its contents via both nas and samba, as needed. It's just that simple.

I let the DiskStation415 device partition and format the drives, because I opted for Synology's proprietary RAID technology. But I don't know why Restic wouldn't work just as well with any share that can be mounted and accessed, regardless of its partitioning or formatting. You just need to be able to write to it and read from it.

My fstab just mounts my shares to /mnt, such as /mnt/Backups (and many others, such as /mnt/Downloads, and /mnt/Documents). So in my case, I just set up Restic repositories such as /mnt/Backups/Linux/{device name} for each machine I backup.

In case you no longer have them, here are my macros for Restic backups. Just change things like /home/jerry as needed:

alias backup='restic backup -e "/home/jerry/VirtualBox VMs" /home/jerry -r /mnt/Backups/Linux/hostname'
(not formatted as code, because that screws with the 's in my macro)

The -e option just tells it not to backup any VirtualBox VMs. Those all have shapshots stored, so there's no need to back them up, and they can be sizeable. So you can disregard -e "/home/jerry/VirtualBox VMs", but I left it in my example to show you how to exclude stuff that would be otherwise included.

I didn't bother making a macro for restoring a backup, because I've never needed to restore anything in all the years I've used Restic. And because it's virtually as simple as creating a backup, as you undoubtedly already know from reading the docs. BTW: restic --help is indeed all you need.

Harvey

brent

1)If I'm backing up mostly/home stuff, and I backup twice a month, how big should I make my external drive partition? Size is no issue. This external is enormous.

Personally I just have a folder all restic backups go into on an external drive, instead of using partitions.

The restic backup will grow over time unless you purge older snapshots as every time a file is modified it will retain the old version and add a new version. The more often you purge snapshots the less redundancy there is for restore points.

So for me I just have an entire 2TB drive dedicated to it. Sectioning your drive into multiple partitions only makes for a future headache. Why o why didn't I give it X GB/TB more! 😛

2) When I 'init' to create my server on this partition for the first time....should I format it for restic? If so, how do I format the external partition 'restic-friendly'? Ext4? Fat? Or let Restic do it?
Or when I setup repo on partition does Restic do all this the way it likes to?

You format the partition as you would any other drive, through a disk / partition manager. My advice is to NOT use FAT/NTFS (Windows file systems). Use a Linux file system such as ext4, Windows file systems do not retain file ownership / permissions. So any restore will be painful, requiring adjustment of permissions to make sure individual files can be read properly or rather CAN NOT be read by those that should not have access. Such as with ssh private keys having octal permissions of 600 (Owner gets read/write permissions everyone else gets nothing) instead of 777 (Everyone gets read/write/execute) which is what you get when you restore from a Windows file system.

brent

WetGeek thanks for the great reply. a partition is a partition is a partition that becomes a repo used by restic is exactly what I need to start. I am overthinking. I keep thinking I have to prepare my storage space like one would with a new operating system.

WetGeek But I don't know why Restic wouldn't work just as well with any share that can be mounted and accessed, regardless of its partitioning or formatting.

that's what I'm hoping...since the backups are created .gpg (I think) I have it in the cranium that I have to send to a .gpg-friendly place, whatever that may be.

thanks for the macros (pilfered into my self-tutorial .txt) and the fstab rigs. you are nothing if not dialed in with this. had extra time today: played with dejadup, and borg/vorta then decided I did not want a GUI.
Appreciate your help.

brent

WetGeek Harvey
what should my fstab entry look like?

I have/dev/sdc2 /mnt/backups

calling this partition /mnt/backups is my naming, but how do I link the two?

brent

Harvey You format the partition as you would any other drive, through a disk / partition manager. My advice is to NOT use FAT/NTFS (Windows file systems). Use a Linux file system such as ext4, Windows file systems do not retain file ownership / permissions.

thank you, noted. Don't want to drag windows into this business.
I can do this with gnome-disks then on a 90GB partition.

Harvey So for me I just have an entire 2TB drive dedicated to it. Sectioning your drive into multiple partitions only makes for a future headache. Why o why didn't I give it X GB/TB more!

Because I have stuff on the other partitions and the shell game of moving stuff from one clam to another--I always lose that game. plus lazy🙂.

Harvey The restic backup will grow over time unless you purge older snapshots as every time a file is modified it will retain the old version and add a new version. The more often you purge snapshots the less redundancy there is for restore points.

Eh...am I reading the lit wrong? I read there was recursive 'chunking' of metadata where stuff you deleted will rightfully disappear in existing backups and reverse for newer a la rsync.
It matters little. The way I backup is I only back up what I need to today, not yesterday. So I can pitch most everything old regularly...unless there are best practices about old backups?

you and geek are the holy grail for me on this, thank you. going to make a full run in an hour.

Harvey

brent

Restic will not create NEW backups of files that are cryptographically the exact same file in the same source location as one already in an existing snapshot, however same file in a different source location is still considered unique.

Restic will not delete files from snapshots because those files no longer exists on your system. They will not be tracked in NEW snapshots but if a prior snapshot contained them, they remain. The whole point being if it turns out you really wanted that deleted/older version of that file, you can restore from another "snapshot" in time.

Not even rsync which provides 0 redundancy within the backup itself will delete files no longer present by default. It needs the flag --delete in order to do this. Restic may have a similar flag but I doubt it, doesn't make sense to me for restic to do this given the whole point is maintaining snapshots.

When a snapshot is forgotten from restic it is not actually deleted, you have to prune to reclaim space. I have aliases for this and much more.

My backup aliases

I use aliases so I do not have to remember stupidly long commands. I also have rsync and restic backups because I do not want to trust any one program. I have 2x restic and 1x rsync backup, all on separate drives.

alias rsynchome="rsync -avh --delete --exclude-from='/home/harvey/.backupexclude' /home/harvey /mnt/rsync/Backup/Home"

alias restichome="restic backup /home/harvey --exclude-file='/home/harvey/.backupexclude' -r /mnt/restic/Backup/Home"

Both restic and rsync have the ability to exclude things from being backed up that are listed in a text file. My .backupexclude plain text file contains the following:

.audacity-data
.cache
Desktop
Downloads
.kde
.local
.lyrics
.nv
.pki
repository/full
.steam
.steampath
.steampid
Templates
.vscode
.wget-hsts

Then you have forget and prune:
Every now and then I run these to clean up.

Forget all snapshots except the last 8 (keep as many as you want). Might seem confusing but if something is forgotten it doesn't actually reclaim the space on your drive, for that you need --prune which will delete all unreferenced data (data from the forgotten snapshots).

alias resticprune="restic -r /mnt/restic2TB/Backup/Home forget --keep-last 8 --prune

This makes sure you have a valid restic backup and is recommended to be run after a prune.

alias resticcheck="restic -r /mnt/restic2TB/Backup/Home check

For anyone that reads this the full documentation on restic is much, much more verbose here

brent

Harvey thank you again this is wonderful. I thought --exclude would be followed by a list of ~/folders not a text doc! Glad I read this.

brent

Harvey
in the partition I made a folder, as you suggested, and called it resticrepo.

restic init --repo /dev/sdc2/resticrepo 
enter password for new repository: 
enter password again: 
Fatal: create repository at /dev/sdc2/resticrepo failed: ReadDir: open /dev/sdc2/resticrepo/keys: not a directory

I don't know what /keys is since I never labeled it and how do I make my folder a directory?
Failure at first try. Will keep exploring.

edit: of course cannot find my unique message in the first 50 hits of DDG....this fatal error nowhere to be seen. callin it a night.

brent

Harvey Forget all snapshots except the last 8 (keep as many as you want). Might seem confusing but if something is forgotten it doesn't actually reclaim the space on your drive, for that you need --prune which will delete all unreferenced data (data from the forgotten snapshots).

alias resticprune="restic -r /mnt/restic2TB/Backup/Home forget --keep-last 8 --prune

This makes sure you have a valid restic backup and is recommended to be run after a prune.

alias resticcheck="restic -r /mnt/restic2TB/Backup/Home check

Hi Harvey, I'm doing my third backup since I started this thread/starting using Restic.
By "last 8 snapshots" do you mean the last 8 backups?

The new backup took about 6 minutes and produced a single snapshot, so it had to rely on the TONS of data from the last 2, I assume. In other words tonight's single snapshot and single binary produced are part of/dependent on the dozens of existing binaries from the last 2 backups? In other words, too soon to prune?

thanks for any info and sorry to revive this one.

Harvey

brent
/dev/* is a list of devices on your system. What you want to do when it comes to working with ssd/hdd/sd cards/thumb drives, is mounting /dev/sdc2 assuming that is the correct drive and partition to a valid mount point such as /mnt/blah/ with sudo mount /dev/sdc2 /mnt/blah

Once the drive is mounted you need to reference the mount point that the device can be located at not the device itself, e.g restic init --repo /mnt/blah

brent

Harvey

Harvey is mounting /dev/sdc2 assuming that is the correct drive

yes it is. I mounted this drive simply in gnome-disks.
wasn't that good enough? (real question since I am always learning).
however I will locate the /mnt address --I think I see it in Disks---and add that to my launch command. I understand what you are saying, thanks for the clarity. Will try this and get back.

Harvey

brent wasn't that good enough?

I was working under the assumption you had not mounted the drive at all. If you mounted it through a GUI tool that is fine and you do not have to create a fstab entry if you do not wish to. I don't actually create one for removable drives and mount them through the GUI too.

I had simplified my restic aliases from earlier because I thought it might be easier to follow, auto mounted drives via a gui tend to mount drives in /run/media/username/drive-label/ which for my backup is /run/media/harvey/data/

So the main thing is to make sure you know where the drive is mounted to, if it is truly /mnt/backups/ then you have all you need to make restic work.

restic init --repo /mnt/backups/

brent

Harvey I hope to decare victory after dinner....🙂

brent

WetGeek Harvey

restic init --repo /run/media/brent/backup
enter password for new repository: 
enter password again: 
created restic repository d72a14388c at /run/media/brent/backup

Victory! Thank you both.

WetGeek

brent I hope to decare victory after dinner....

... just keep a good thought! Everything will be fine ... everything will be fine ... everything...

WetGeek

brent Victory! Thank you both.

That looks like a good first step ... creating your repository. Did you actually go on to create a backup there?

brent

My 2 questions revolved around restic startup and they are both answered so SOLVED---thank you kindly.

brent

WetGeek I began my backups. I'm comfortable with the CLI commands and verbiage...backing up in chunks...but I thought there would be some zip to this. 'Very powerful' I read about Restic everywhere I surfed......as fast/slow as any other backup I've used. Identical to grsync for speed if I had to guess.
edit: no complaint. just an observation.

brent

No sir, maybe not tonight. I want to go into my first backup (aside from grsync) with my folders purged of debris and garbage, plus organize, plus I'm compiling an --exclude list as well right now as soon as I decide what I need or don't need. Example: .mozilla? .dmrc? .x-session errors? @Harvey 's exclude list way above is a good baseline but everyone has different debris.

brent

I did my first run by backing up 100GB of media.

So restic busted this 100GBs into 5 small binary files? I saw nothing like that in restic documentation.
What's the binaries for? Restore?
Gotta do some more reading. I'm easily🙂 confused tonight.

edit:
Fatal: unable to save snapshot: write /run/media/brent/backup/data/d5/d59c665502676333eb52eee1c87a3dbff78: no space left on device

--the binaries a reaction to that restic terminal output? if that's true you were right Harvey: you said "don't mess around with partitions" I had 1TB and did not make the partition big enough. Opting for No partitions now.

Still begs the question: the binary is not the "snapshot"? the snapshot is .gpg I had read. I have no .gpg files probably because partition is too small.
I'll be bock🙂