Tux

...making Linux just a little more fun!

rsync internals

Amit Saha [amitsaha.in at gmail.com]


Thu, 1 Apr 2010 12:54:33 +0530

Hello TAG:

I have been using 'rsync' off late for backups and helping my Prof. do the same as well.

My prof. has ~6G space left on his hard disk and we are trying to back up everything (~200 G)to a external HDD. As 'rsync' progresses, the disk space on the (source) hard disk gets consumed incrementally and large files (~14G) cannot be copied to the backup hard-disk. rsycn fails saying "No disk space left on device". I wonder why. I can try to look into rsync source to see why this happens. But, why is rsync consuming disk space on the source disk? Its a file copy operation right? It should consume the main memory, not the secondary memory. Or, is it a virtual memory thing?

Thanks for any insights!

Best Regards, Amit

-- Journal: http://amitksaha.wordpress.com ?-blog: http://twitter.com/amitsaha


Top    Back


Kapil Hari Paranjape [kapil at imsc.res.in]


Thu, 1 Apr 2010 14:18:39 +0530

Hello,

On Thu, 01 Apr 2010, Amit Saha wrote:

> Or, is it a virtual memory thing?

If you are using rsync with its default options this should not happen. Are you using a swap file? Check that you are actually writing to the destination. As a final measure you can mount the source disk as read-only in order to take the backup quickly (As it seems to be urgently required!).

Regards,

Kapil. --


Top    Back


Amit Saha [amitsaha.in at gmail.com]


Thu, 1 Apr 2010 14:10:04 +0530

On Thu, Apr 1, 2010 at 2:18 PM, Kapil Hari Paranjape <kapil at imsc.res.in> wrote:

> Hello,
>
> On Thu, 01 Apr 2010, Amit Saha wrote:
>> Or, is it a virtual memory thing?
>
> If you are using rsync with its default options this should not
> happen. Are you using a swap file?

Not sure. Its a MacBook and its not mine, so I don't know much about it whether a swap file is being used or not. I am using 'rsync' default options.

Check that you are actually

> writing to the destination.

Checked.

As a final measure you can mount the

> source disk as read-only in order to take the backup quickly (As it
> seems to be urgently required!).

Hmm..we got a workaround, but not copying those big files, since they were not that important.

Thanks, Amit

>
> Regards,
>
> Kapil.
> --
>
>

-- Journal: http://amitksaha.wordpress.com ?-blog: http://twitter.com/amitsaha


Top    Back


Neil Youngman [ny at youngman.org.uk]


Thu, 1 Apr 2010 09:45:52 +0100

On Thursday 01 April 2010 08:24:33 Amit Saha wrote:

> Hello TAG:
>
> I have been using 'rsync' off late for backups and helping my Prof. do
> the same as well.
>
> My prof. has ~6G space left on his hard disk and we are trying to back
> up everything (~200 G)to a external HDD.  As 'rsync' progresses, the
> disk space on the (source) hard disk gets consumed incrementally and
> large files (~14G) cannot be copied to the backup hard-disk. rsycn
> fails saying "No disk space left on device". I wonder why. I can try
> to look into rsync source to see why this happens. But, why is rsync
> consuming disk space on the source disk? Its a file copy operation
> right? It should consume the main memory, not the secondary memory.
> Or, is it a virtual memory thing?

I assume the disk space is freed up when rsync is stopped?

Some creative hacking with strace should reveal what files rsync is creating. This should give you some pointers to where the disk space is going.

HTH

Neil Youngman


Top    Back


Amit Saha [amitsaha.in at gmail.com]


Thu, 1 Apr 2010 14:17:38 +0530

On Thu, Apr 1, 2010 at 2:15 PM, Neil Youngman <ny at youngman.org.uk> wrote:

> On Thursday 01 April 2010 08:24:33 Amit Saha wrote:
>> Hello TAG:
>>
>> I have been using 'rsync' off late for backups and helping my Prof. do
>> the same as well.
>>
>> My prof. has ~6G space left on his hard disk and we are trying to back
>> up everything (~200 G)to a external HDD. ?As 'rsync' progresses, the
>> disk space on the (source) hard disk gets consumed incrementally and
>> large files (~14G) cannot be copied to the backup hard-disk. rsycn
>> fails saying "No disk space left on device". I wonder why. I can try
>> to look into rsync source to see why this happens. But, why is rsync
>> consuming disk space on the source disk? Its a file copy operation
>> right? It should consume the main memory, not the secondary memory.
>> Or, is it a virtual memory thing?
>
> I assume the disk space is freed up when rsync is stopped?

Yes.

>
> Some creative hacking with strace should reveal what files rsync is creating.
> This should give you some pointers to where the disk space is going.

Hmm..Sure would. Let me see if I can get my hands-on on the laptop again.

Thanks & Cheers, Amit

>
> HTH
>
> Neil Youngman
>                                              
> TAG mailing list
> TAG at lists.linuxgazette.net
> http://lists.linuxgazette.net/listinfo.cgi/tag-linuxgazette.net
>

-- Journal: http://amitksaha.wordpress.com ?-blog: http://twitter.com/amitsaha


Top    Back