backing up data to external hard drive

slade17 · #1 24th January 2007, 08:30 PM

I just got an external hard drive today and i was hoping theres software or a script that will back up everything i need to one folder on the hard drive.

essentially what im looking for is a way to designate folders and files to be backed up, then run a program/script to copy all those files to a folder on the external hard drive; in future backups it would copy files that had been created since the last backup, and remove files that had been deleted. the drive is going to be used on multiple computers, so i want to back up to a folder on the drive instead of the drive itself.

also, what files should be backed up? /home, of course (are there any hidden configuration files i should exclude? i had a problem with permissions on a flash drive because i copied some config files; i couldnt delete them even as root. i want to keep .gaim at the least, though) I also want to copy files for reference like /etc/fstab, /etc/yum.conf, /etc/yum.repos.d, /etc/grub.conf, are there any other important files to back up? oh, and where are firefox bookmarks stored?

thanks

Duli · #2 24th January 2007, 08:48 PM

Hello,

I think rsync will do just fine. You should use it in a script (sh) combined with cron.

The script should first of all mount your external hd into a folder. Than it would rsync the files you want to the mounted folder.

slade17 · #3 24th January 2007, 10:32 PM

Quote:

Originally Posted by Duli

Hello,

I think rsync will do just fine. You should use it in a script (sh) combined with cron.

The script should first of all mount your external hd into a folder. Than it would rsync the files you want to the mounted folder.

do you know of a good resource on writing scripts? i've been using linux for a while now but im completely new at scripting.

also, do i actually need to mount the external HD in the script? it automounts under /mount as soon as i plug it in, so it would make sense that the script could just reference the folder.

thanks for the tip about cron

slade17 · #4 24th January 2007, 11:36 PM

also, whats the syntax for rsync?

Code:

[colin@localhost ~]$ rsync /home/colin/rsync/test1 /home/colin/rsync/test2
 skipping directory /home/colin/rsync/test1

#5 24th January 2007, 11:36 PM

I have a script in /etc/cron.daily called rsync.cron. It backs up to an external firewire drive, copying across changes and deletes files from the backup which no longer exist on the source.
The daily cron runs at 4am, so you may wish to adjust this.

The script is:

Code:

#!/bin/bash

if [ -d /mnt/usb/backup ] ; then

rsync -avz --delete /root /mnt/usb/backup/
rsync -avz --delete /sbin /mnt/usb/backup/
rsync -avz --delete /bin /mnt/usb/backup/
rsync -avz --delete /boot /mnt/usb/backup/
rsync -avz --delete /etc /mnt/usb/backup/
rsync -avz --delete /export /mnt/usb/backup/
rsync -avz --delete /home /mnt/usb/backup/
rsync -avz --delete /lib /mnt/usb/backup/
rsync -avz --delete /opt /mnt/usb/backup/
rsync -avz --delete /usr /mnt/usb/backup/
rsync -avz --delete /var /mnt/usb/backup/

fi

/mnt/usb is my mount point. whilst 'backup' is a directory at the root of the external drive. The 'if' statement just checks to make sure that the drive is mounted; I don't wan't to fill up the mount point on my local file system.

paul matthijsse · #6 24th January 2007, 11:47 PM

Quote:

Originally Posted by slade17

do you know of a good resource on writing scripts? i've been using linux for a while now but im completely new at scripting.

Making backups is as easy as coping a dir in your /home/user/ hierarchy to another drive/dir. You can speed up things by using a program like rdiff-backup, that will only copy files that are newer than the ones in the backup archive. But try for yourself in bash with a copy script, or write that in Perl, it's really easy: copy dir-this to dir-that. Use cron to automate that.

Quote:

Originally Posted by slade17

also, do i actually need to mount the external HD in the script? it automounts under /mount as soon as i plug it in, so it would make sense that the script could just reference the folder.

If it automounts, then there's no need to mount it script wise. Just try and see what happens.

Cheers, Paul.

slade17 · #7 25th January 2007, 12:33 AM

Quote:

Originally Posted by pjfg

I have a script in /etc/cron.daily called rsync.cron. It backs up to an external firewire drive, copying across changes and deletes files from the backup which no longer exist on the source.
The daily cron runs at 4am, so you may wish to adjust this.

The script is:

Code:

#!/bin/bash

if [ -d /mnt/usb/backup ] ; then

rsync -avz --delete /root /mnt/usb/backup/
rsync -avz --delete /sbin /mnt/usb/backup/
rsync -avz --delete /bin /mnt/usb/backup/
rsync -avz --delete /boot /mnt/usb/backup/
rsync -avz --delete /etc /mnt/usb/backup/
rsync -avz --delete /export /mnt/usb/backup/
rsync -avz --delete /home /mnt/usb/backup/
rsync -avz --delete /lib /mnt/usb/backup/
rsync -avz --delete /opt /mnt/usb/backup/
rsync -avz --delete /usr /mnt/usb/backup/
rsync -avz --delete /var /mnt/usb/backup/

fi

/mnt/usb is my mount point. whilst 'backup' is a directory at the root of the external drive. The 'if' statement just checks to make sure that the drive is mounted; I don't wan't to fill up the mount point on my local file system.

thanks, it looks like that is going to work. but about the options - what is verbosity, and what does archive mode do (since the file is not stored in an archive format)?

paul - thanks, but my /home folder is over 35 GB; copying that every time i want to back up would be a pain.

*edit*

Code:

[colin@localhost ~]$ rsync -avz --delete /home/colin/rsync /media/WD?Passport/te st
building file list ... done
deleting rsync/test2/test1/bookmarks/bookmarks4.html
deleting rsync/test2/test1/bookmarks/bookmarks3.html
deleting rsync/test2/test1/bookmarks/bookmarks2.html
deleting rsync/test2/test1/bookmarks/bookmarks.html
deleting rsync/test2/test1/bookmarks/
deleting rsync/test2/test1/index.html
deleting rsync/test2/test1/img021.jpg
deleting rsync/test2/test1/grub.conf
deleting rsync/test2/test1/essay_template.odt
deleting rsync/test2/test1/contacts.ods
deleting rsync/test2/test1/chromeblue4pc.jpg
deleting rsync/test2/test1/
rsync/
rsync: chgrp "/media/WD Passport/test/rsync" failed: Operation not permitted (1)
rsync/test1/
rsync: chgrp "/media/WD Passport/test/rsync/test1" failed: Operation not permitt ed (1)
rsync: chgrp "/media/WD Passport/test/rsync/test1/chromeblue4pc.jpg" failed: Ope ration not permitted (1)
rsync: chgrp "/media/WD Passport/test/rsync/test1/contacts.ods" failed: Operatio n not permitted (1)
rsync: chgrp "/media/WD Passport/test/rsync/test1/essay_template.odt" failed: Op eration not permitted (1)
rsync: chgrp "/media/WD Passport/test/rsync/test1/grub.conf" failed: Operation n ot permitted (1)
rsync: chgrp "/media/WD Passport/test/rsync/test1/img021.jpg" failed: Operation not permitted (1)
rsync: chgrp "/media/WD Passport/test/rsync/test1/index.html" failed: Operation not permitted (1)
rsync/test1/bookmarks/
rsync: chgrp "/media/WD Passport/test/rsync/test1/bookmarks" failed: Operation n ot permitted (1)
rsync: chgrp "/media/WD Passport/test/rsync/test1/bookmarks/bookmarks.html" fail ed: Operation not permitted (1)
rsync: chgrp "/media/WD Passport/test/rsync/test1/bookmarks/bookmarks2.html" fai led: Operation not permitted (1)
rsync: chgrp "/media/WD Passport/test/rsync/test1/bookmarks/bookmarks3.html" fai led: Operation not permitted (1)
rsync: chgrp "/media/WD Passport/test/rsync/test1/bookmarks/bookmarks4.html" fai led: Operation not permitted (1)
rsync/test2/
rsync: chgrp "/media/WD Passport/test/rsync/test2" failed: Operation not permitt ed (1)
rsync: chgrp "/media/WD Passport/test/rsync" failed: Operation not permitted (1)
rsync: chgrp "/media/WD Passport/test/rsync/test1" failed: Operation not permitt ed (1)
rsync: chgrp "/media/WD Passport/test/rsync/test1/bookmarks" failed: Operation n ot permitted (1)
rsync: chgrp "/media/WD Passport/test/rsync/test2" failed: Operation not permitt ed (1)

sent 476 bytes  received 104 bytes  1160.00 bytes/sec
total size is 2639379  speedup is 4550.65
rsync error: some files could not be transferred (code 23) at main.c(892) [sende r=2.6.8]

everything copied fine, but what are the errors from?

beissemj · #8 25th January 2007, 01:04 AM

It means you don't have the proper permissions on those files, you probably need to be root.

slade17 · #9 25th January 2007, 01:26 AM

Quote:

Originally Posted by beissemj

It means you don't have the proper permissions on those files, you probably need to be root.

hmm, i should have had proper permissions for all those files. but now that you mention it im going to need root permissions for some of the files i'll want to copy. how do you add root permissions inside a script, just su - and then the password, as you would in a terminal?

slade17 · #10 25th January 2007, 02:51 PM

Quote:

Originally Posted by beissemj

It means you don't have the proper permissions on those files, you probably need to be root.

okay, i ran the above script modified to fit the correct files/devices, after logging in as root. a lot of the data copied but a lot of it failed also: "failed: Operation not permitted (1)" most notably, my entire /home folder failed to copy, with the exception of two text files. what am i doing wrong? i am logged in as root. even the files that copied fine in the test i did outside of a script failed.

Code:

rsync error: some files could not be transferred (code 23) at main.c(892) [sender=2.6.8]

#11 25th January 2007, 03:08 PM

Code:

rsync error: some files could not be transferred (code 23) at main.c(892) [sender=2.6.8]

This usually means that some of the files on the source have been deleted since the list was made (rsync first creates a list of differences between source and target). It is quite normal to see such errors.

As to why some files are failing, the issue seems to be with modifying the permissions hence 'chgrp'.

Can you provide more information:

1. Your script.
2. 'ls -l' on the files that do copy as well as a sample of those that don't. Also 'ls -l /media'
3. Output of the 'mount' command.

slade17 · #12 25th January 2007, 03:27 PM

Quote:

Originally Posted by pjfg

Code:

rsync error: some files could not be transferred (code 23) at main.c(892) [sender=2.6.8]

This usually means that some of the files on the source have been deleted since the list was made (rsync first creates a list of differences between source and target). It is quite normal to see such errors.

As to why some files are failing, the issue seems to be with modifying the permissions hence 'chgrp'.

Can you provide more information:

1. Your script.
2. 'ls -l' on the files that do copy as well as a sample of those that don't. Also 'ls -l /media'
3. Output of the 'mount' command.

Code:

#!/bin/bash

if [ -d /media/WD?Passport/vaio_fedora ] ; then

rsync -avz --delete /home /media/WD?Passport/vaio_fedora
rsync -avz --delete /root /media/WD?Passport/vaio_fedora
rsync -avz --delete /sbin /media/WD?Passport/vaio_fedora
rsync -avz --delete /bin /media/WD?Passport/vaio_fedora
rsync -avz --delete /boot /media/WD?Passport/vaio_fedora
rsync -avz --delete /etc /media/WD?Passport/vaio_fedora
rsync -avz --delete /lib /media/WD?Passport/vaio_fedora
rsync -avz --delete /usr /media/WD?Passport/vaio_fedora
rsync -avz --delete /var /media/WD?Passport/vaio_fedora

fi

directory that copies properly:

Code:

[root@localhost ~]# ls -l /root
total 76
-rw------- 1 root root  1283 Aug 21 20:51 anaconda-ks.cfg
drwxr-xr-x 2 root root  4096 Nov  2 00:52 Desktop
-rw-r--r-- 1 root root 28324 Aug 21 20:51 install.log
-rw-r--r-- 1 root root  4160 Aug 21 20:51 install.log.syslog
-rw-rw-r-- 1 root root  9625 Dec  5 15:01 scsound.log

Code:

[root@localhost ~]# ls -l /home
total 24
drwx------ 48 colin    colin    4096 Jan 25 10:17 colin
drwx------ 12 marjorie marjorie 4096 Aug 23 22:50 marjorie
drwx------  2 rick     rick     4096 Nov 10 19:44 rick

home folders marjorie and rick fail to copy; colin copies with only 2 text files in it.

i can post the output of my home folder, but its fairly messy

Code:

[root@localhost ~]# ls -l /media
total 32
drwxr-xr-x 7 colin root 32768 Dec 31  1969 WD Passport

i dont actually run the mount command, the hard drive automounts as soon as i plug it in.

is there any way to change the device name? the space is annoying me.

#13 25th January 2007, 03:41 PM

Quote:

Originally Posted by slade17

i dont actually run the mount command, the hard drive automounts as soon as i plug it in.

is there any way to change the device name? the space is annoying me.

Actually, the mount command by itslef is helpfull, as it shows what options the volume was mounted as.

To avoid the name with the space, specify a mount point in /etc/fstab along the lines of:

/dev/sda1 /mnt/disk

Provided of course that sda1 is your device and you have created a mount point called disk.

slade17 · #14 25th January 2007, 04:19 PM

Quote:

Originally Posted by pjfg

Actually, the mount command by itslef is helpfull, as it shows what options the volume was mounted as.

To avoid the name with the space, specify a mount point in /etc/fstab along the lines of:

/dev/sda1 /mnt/disk

Provided of course that sda1 is your device and you have created a mount point called disk.

are you saying i should mount it myself instead of letting it automount, so i know what options i use to mount it as? what options should i be using?

and i was talking about actually renaming the device, but i guess that would work too.

wont editing /etc/fstab make the drive automount on boot? since its an external drive and will only be plugged in when i want to back up dataim not sure that that would be the best idea. or would it just force the drive to be mounted at that point when plugged in?

#15 25th January 2007, 04:33 PM

Mounting manually also solves your renaming problem - the auto mounter is just using the volume name as a mount point.

Adding the entry to fstab will mount it if the drive is connected at boot. If you later plug in the drive then just run 'mount /dev/sda1' (or whatever you device is). When you are done 'umount /dev/sda1'. Perhaps you could put those at the beginning and end of your scripts.

For the mount options, I can post what I use later today when I have access to my system again.