rsync, lftp and exclude lists

When it comes to synchronizing stuff with a remote server, most people will use either rsync or ftp for file transfers. I personally prefer rsync, where the upload command would look something like this:

rsync -e ssh -avz --exclude-from 'exclude-list.txt' /path/to/local/dir

I find rsync's --exclude-from option really helpful and much more convenient to use than multiple --exclude switches since you can keep all your filenames and patterns in one place. However, you may not have a possibility to use rsync at all, especially if you're using a cheap shared hosting.

That was my case yesterday, so I searched for an ftp alternative to rsync and stumbled upon lftp in this post. It looked really nice until I found out lftp doesn't support exclude lists, which means you can't make it read exclude patterns from a file. The good news is, lftp does allow you to specify multiple --exclude switches at once, so I modified the command to make use of my exclude-list.txt like this:

lftp -u username,password -e "mirror --only-newer sed 's/^/--exclude /' exclude-list.txt | tr 'n' ' ' --reverse /path/to/local/dir /path/to/public_html"

This way, sed prepends every line with an --exclude switch and tr replaces newlines so in the end we have our lftp mirror command with all necessary excludes in it. Well, a bit later I realized this hosting sucked and changed to a different one with ssh and rsync, but that's a different story.

About the authorMartin Boze is the Founder of MultiMerch Marketplace. I also love skydiving, motorcycles and space. And other cool things.

7 replies
  1. Rolf
    Rolf says:

    Thanks Martin! Exactly what I wanted to solve :-)

    I like to use a .gitignore file for exclusion, and met with one minor detail: --exclude will interpret the argument as a regex, so a glob pattern like *~ goes wrong. I changed the sed part to:

    sed 's/^/--exclude-glob /' .gitignore

  2. Dan
    Dan says:

    I use lftp with -f because I hold my script in a txt file. I want to only include the .tar.gz files in the transfer. Is this right ?

    mirror --reverse --delete --only-newer sed 's/^/--include-glob /' .tar.gz /local /remote

  3. Josh
    Josh says:

    Maybe you should look into bash extended globs.

    shopt -s extglob
    ls !(filetoexclude|anotherfile)

    Just use the built in globbing with whatever transport system you like (lftp/rsync)

  4. Henrik
    Henrik says:

    You can use rsync even if you only have FTP access, if you first mount the FTP directory you want locally.
    Once such example using curlftpfs:

    $ curlftpfs live_mount
    $ rsync -rltpzv --exclude-from 'exclude-list.txt' live_mount staging/

    I keep credentials in ~/.netrc which gets picked up by curlftpfs and where only my user can read them.
    It's been working very well for me and transfers reasonably fast, although it does take a while to get past the "sending incremental file list" so you may thing nothing is happening, especially if lots of files have changed.


Trackbacks & Pingbacks

  1. [...] files and directories too, but curiously has no option to read a list of exclusions from a file. Martin Boze wrote how he fixed that, by using sed and [...]

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published.