{"id":34,"date":"2017-06-25T20:38:00","date_gmt":"2017-06-25T20:38:00","guid":{"rendered":""},"modified":"2019-03-12T20:10:16","modified_gmt":"2019-03-12T20:10:16","slug":"backup-your-files-with-simple-bash-scripts","status":"publish","type":"post","link":"https:\/\/ntlx.org\/de\/2017\/06\/backup-your-files-with-simple-bash-scripts.html","title":{"rendered":"Backup Your Files with simple Bash Scripts"},"content":{"rendered":"

Ever lost data you stored on a usb drive just because it stopped working and you did not have a backup? How often did you promise yourself to set up a backup system so this will not happen again – just a few days before forgot you wanted to do so? You are not alone – so did I. Until a few months ago, when I decided to store my data on my own NAS, run by a RaspberryPi 3 and OwnCloud, to give me the feeling to have control over where my data is physically stored. On a USB drive below my desk. Without a popup reminding me, my Dropbox is running out of space.<\/p>\n

As hard drives tend to fail, I decided to put a backup system in place so the data is safe as long as only one of the two hard drives stops working. And this was quite easy, so I want to share the simple bash scripts I use to create incremental backups of my data.<\/div>\n
<\/p>\n

The Strategy<\/h2>\n<\/div>\n
First, here is the backup strategy I implemented:<\/div>\n
 – Over 5 Years, I want to keep a backup of the data, as it was in the beginning of that year<\/div>\n
 – Over the last year, I want to keep the first backup of each month<\/div>\n
 – Over the last month, I want to keep the backup of every Monday<\/div>\n
 – Over the last week (7 days), the backups of every day are kept<\/p>\n<\/div>\n
That sounds like a high amount of data to store. But it is not, if you use the rsync argument –link-dest <folder> <\/i>which makes rsync create hard links in the target folder to the folder we pass as an <folder> <\/i>argument, instead of creating actual copies of the source. So, only a bit more space than the actual copy in the beginning is needed for every new backup. That is the data that actually changed – hence the data we want to back up, plus some overhead for folders and the hard links.<\/div>\n
<\/div>\n
Here is the command we can use to create such incremental backups with rsync:<\/p>\n<\/div>\n
\n
\n\n\n\n
\n
1<\/pre>\n<\/td>\n
\n
rsync -a --delete --link-dest ${<\/span>LASTDAYPATH<\/span>}<\/span> ${<\/span>DATADIR<\/span>}<\/span> ${<\/span>TODAYPATH<\/span>}<\/span>
<\/pre>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n
<\/div>\n
This command creates a backup of ${DATADIR}<\/i> to ${TODAYPATH}<\/i> creating links of unchanged data to ${LASTDAYPATH}.<\/i>

<\/i><\/div>\n
\n
The Scripts<\/h2>\n<\/div>\n
Such a command should now be executed every night using a cron job.<\/div>\n
<\/p>\n
\n\n\n\n
\n
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13<\/pre>\n<\/td>\n
\n
#!\/bin\/bash<\/span>

TODAY<\/span>=<\/span>$(<\/span>date +%Y-%m-%d)<\/span>
BACKUPDIR<\/span>=<\/span>\/nas\/backup\/daily\/
SCRIPTDIR<\/span>=<\/span>\/nas\/data\/backup_scripts 
DATADIR<\/span>=<\/span>\/nas\/data\/
LASTDAYPATH<\/span>=<\/span>${<\/span>BACKUPDIR<\/span>}<\/span>\/$(<\/span>ls ${<\/span>BACKUPDIR<\/span>}<\/span> | tail -n 1)<\/span>
TODAYPATH<\/span>=<\/span>${<\/span>BACKUPDIR<\/span>}<\/span>\/${<\/span>TODAY<\/span>}<\/span>
if<\/span> [[<\/span> ! -e ${<\/span>TODAYPATH<\/span>}<\/span> ]]<\/span>; then<\/span>
        <\/span>mkdir -p ${<\/span>TODAYPATH<\/span>}<\/span>
fi              <\/span>
rsync -a --delete --link-dest ${<\/span>LASTDAYPATH<\/span>}<\/span> ${<\/span>DATADIR<\/span>}<\/span> ${<\/span>TODAYPATH<\/span>}<\/span> $@<\/span>
${<\/span>SCRIPTDIR<\/span>}<\/span>\/deleteOldBackups.sh
<\/pre>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
The data<\/b> hard drive is mounted to \/nas\/data<\/i>, the backup<\/b> hard drive is mounted to \/nas\/backup<\/i>. Every day the backup scripts creates a backup of the data drive to the backup drive (in the folder daily – <\/i>which might be a misleading name as we store all the backups in it).<\/p>\n
At the end of the script, we trigger another script deleting all the old backups, which are not needed anymore according to the backup strategy above.
<\/p>\n
\n\n\n\n
\n
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46<\/pre>\n<\/td>\n
\n
#!\/bin\/bash<\/span>

BACKUPDIR<\/span>=<\/span>\/nas\/backup\/daily\/
function <\/span>listYearlyBackups()<\/span> {<\/span>
        for <\/span>i in 0 1 2 3 4 5
                do <\/span>ls ${<\/span>BACKUPDIR<\/span>}<\/span> | egrep \"$(date +%Y -d \"<\/span>${<\/span>i<\/span>}<\/span> year ago\")-[0-9]{2}-[0-9]{2}\"<\/span> | sort -u | head -n 1
        done<\/span>
}<\/span>

function <\/span>listMonthlyBackups()<\/span> {<\/span>
        for <\/span>i in 0 1 2 3 4 5 6 7 8 9 10 11 12
                do <\/span>ls ${<\/span>BACKUPDIR<\/span>}<\/span> | egrep \"$(date +%Y-%m -d \"<\/span>${<\/span>i<\/span>}<\/span> month ago\")-[0-9]{2}\"<\/span> | sort -u | head -n 1
        done<\/span>
}<\/span>

function <\/span>listWeeklyBackups()<\/span> {<\/span>
        for <\/span>i in 0 1 2 3 4
                do <\/span>ls ${<\/span>BACKUPDIR<\/span>}<\/span> | grep \"$(date +%Y-%m-%d -d \"<\/span>last monday -${<\/span>i<\/span>}<\/span> weeks\")\"<\/span>
        done<\/span>
}<\/span>

function <\/span>listDailyBackups()<\/span> {<\/span>
        for <\/span>i in 0 1 2 3 4 5 6
                do <\/span>ls ${<\/span>BACKUPDIR<\/span>}<\/span> | grep \"$(date +%Y-%m-%d -d \"<\/span>-${<\/span>i<\/span>}<\/span> day\")\"<\/span>
        done<\/span>
}<\/span>

function <\/span>getAllBackups()<\/span> {<\/span>
        listYearlyBackups
        listMonthlyBackups
        listWeeklyBackups
        listDailyBackups
}<\/span>

function <\/span>listUniqueBackups()<\/span> {<\/span>
        getAllBackups | sort -u
}<\/span>

function <\/span>listBackupsToDelete()<\/span> {<\/span>
        ls ${<\/span>BACKUPDIR<\/span>}<\/span> | grep -v -e \"$(echo -n $(listUniqueBackups) |sed \"<\/span>s\/ \/\\|<\/span>\/g\")\"<\/span>
}<\/span>

cd<\/span> ${<\/span>BACKUPDIR<\/span>}<\/span>
listBackupsToDelete | while <\/span>read <\/span>file_to_delete; do<\/span>
        <\/span>rm <\/span>${<\/span>file_to_delete<\/span>}<\/span>
done<\/span>
<\/pre>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
The idea of this script is to first list all the backups that should be kept, according to our strategy, and afterwards invert this selection to find out the ones to delete.<\/p>\n
And that’s it! Not much magic in creating incremental backups without needing too much space. My NAS is running these scripts every night since 10 months now, currently backing up 607 Gigabytes. The backups currently take 630 Gigabytes. <\/span><\/span>Fi<\/span>nd the current version of my simple bash scripts in this GitHub repository: https:\/\/github.com\/NautiluX\/backup_scripts<\/a>  <\/div>\n","protected":false},"excerpt":{"rendered":"Ever lost data you stored on a usb drive just because it stopped working and you did not have a backup? How often did you promise yourself to set up a backup system so this will not happen again – just a few days before forgot you wanted to do so? You are not alone… Artikel ansehen<\/a>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[26,15,12,9],"_links":{"self":[{"href":"https:\/\/ntlx.org\/de\/wp-json\/wp\/v2\/posts\/34"}],"collection":[{"href":"https:\/\/ntlx.org\/de\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ntlx.org\/de\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ntlx.org\/de\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ntlx.org\/de\/wp-json\/wp\/v2\/comments?post=34"}],"version-history":[{"count":1,"href":"https:\/\/ntlx.org\/de\/wp-json\/wp\/v2\/posts\/34\/revisions"}],"predecessor-version":[{"id":70,"href":"https:\/\/ntlx.org\/de\/wp-json\/wp\/v2\/posts\/34\/revisions\/70"}],"wp:attachment":[{"href":"https:\/\/ntlx.org\/de\/wp-json\/wp\/v2\/media?parent=34"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ntlx.org\/de\/wp-json\/wp\/v2\/categories?post=34"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ntlx.org\/de\/wp-json\/wp\/v2\/tags?post=34"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}