rsync, lftp and exclude lists

When it comes to synchronizing stuff with a remote server, most people will use either rsync or ftp for file transfers. I personally prefer rsync, where the upload command would look something like this:

rsync -e ssh -avz --exclude-from 'exclude-list.txt' /path/to/local/dir

I find rsync's --exclude-from option really helpful and much more convenient to use than multiple --exclude switches since you can keep all your filenames and patterns in one place. However, you may not have a possibility to use rsync at all, especially if you're using a cheap shared hosting.

That was my case yesterday, so I searched for an ftp alternative to rsync and stumbled upon lftp in this post. It looked really nice until I found out lftp doesn't support exclude lists, which means you can't make it read exclude patterns from a file. The good news is, lftp does allow you to specify multiple --exclude switches at once, so I modified the command to make use of my exclude-list.txt like this:

lftp -u username,password -e "mirror --only-newer sed 's/^/--exclude /' exclude-list.txt | tr 'n' ' ' --reverse /path/to/local/dir /path/to/public_html"

This way, sed prepends every line with an --exclude switch and tr replaces newlines so in the end we have our lftp mirror command with all necessary excludes in it. Well, a bit later I realized this hosting sucked and changed to a different one with ssh and rsync, but that's a different story.

  • Rolf

    Thanks Martin! Exactly what I wanted to solve :-)

    I like to use a .gitignore file for exclusion, and met with one minor detail: --exclude will interpret the argument as a regex, so a glob pattern like *~ goes wrong. I changed the sed part to:

    sed 's/^/--exclude-glob /' .gitignore

  • Pingback: Deploying a website with lftp -

  • There isn't any time for cut-and-dried monotony. There's time for work. And time for love. That leaves no other time!
    Carpe per diem - seize the check.

  • Dan

    I use lftp with -f because I hold my script in a txt file. I want to only include the .tar.gz files in the transfer. Is this right ?

    mirror --reverse --delete --only-newer sed 's/^/--include-glob /' .tar.gz /local /remote

  • Maybe you should look into bash extended globs.

    shopt -s extglob
    ls !(filetoexclude|anotherfile)

    Just use the built in globbing with whatever transport system you like (lftp/rsync)

  • Henrik

    You can use rsync even if you only have FTP access, if you first mount the FTP directory you want locally.
    Once such example using curlftpfs:

    $ curlftpfs live_mount
    $ rsync -rltpzv --exclude-from 'exclude-list.txt' live_mount staging/

    I keep credentials in ~/.netrc which gets picked up by curlftpfs and where only my user can read them.
    It's been working very well for me and transfers reasonably fast, although it does take a while to get past the "sending incremental file list" so you may thing nothing is happening, especially if lots of files have changed.