michael schurter

Rsync is an excellent file transfer utility thats especially well suited for backing up files over the Internet because it only transfers the data that has changed. A friend asked me how to set it up, so I thought I’d post what I sent him here.

Goal: Backup a directory from computer Zim to computer Ark

Details:

• Both Zim and Ark are subdomains of example.com
• The user on Ark which receives the backup files is named backupuser
• The user on Zim with access to the files you want to backup is named steve

Prerequisites:

• ssh installed on both hosts
• rsync installed on both hosts

1. Login to Zim via ssh:

   ssh steve@zim.example.com

2. Generate a ssh key pair using:

   ssh-keygen -t rsa
   <press enter when prompted where to save the key>
   <press enter twice when asked for a passphrase>

3. To use the key to login to Ark remotely without manually entering a password you need to copy the public key from Zim to Ark using:

   ssh-copy-id -i .ssh/id_rsa.pub backupuser@ark.example.com

   If you don’t have ssh-copy-id on your system, get a new system. If thats not possible you can download the script with:

   wget -O ssh-copy-id http://cvsweb.mindrot.org/index.cgi/~checkout~/openssh/contrib/ssh-copy-id?rev=1.6;content-type=text%2Fplain && chmod +x ssh-copy-id

   Then retry the above command only you’ll need to prepend a “./”:

   ./ssh-copy-id -i .ssh/id_rsa.pub backupuser@ark.example.com

4. Verify the key copied properly by attempting to login to Ark. You should not be prompted for a password:

   ssh backupuser@ark.example.com

5. Logout of Ark. The key is setup, so you’re now ready to rsync files without having to manually enter a password.
6. Test rsync by choosing a small file to backup and using:

   rsync -tP /some/small/testfile backupuser@ark.example.com:/tmp

   A nice little progress bar should be displayed as the file is transferred. Confirm that “testfile” is now in /tmp on Ark.

7. You’re finally ready to do a real rsync like:

   rsync -t /directory/to/backup/* backupuser@ark.example.com:/existing/backup/directory

   Note: There are several useful options for rsync. Check man rsync to find out more.

   • -p — preserve permissions (useful for backups, use -E if you only care about the executable bit)
   • -r — recursively backup directories.
   • -z — compressed uncompressed files
   • And just FYI: -t tells rsync to use the last modified timestamp to determine whether or not to transfer files. It makes rsync a lot faster at determining whether or not files have changed.
8. To schedule the backup to take place nightly at 1:13 AM edit your crontab using crontab -e and insert the following line:

   13 1 * * * rsync -qt /directory/to/backup/* backupuser@ark.example.com:/existing/backup/directory

Caveats:

• These instructions will push files from Zim to Ark. There’s no reason why Ark couldn’t pull files from Zim. In fact, this is often more secure if Zim is a web server with a larger attack surface than Ark. Mea culpa.
• If the IP address of Ark is dynamic, use a service like dyndns.com. Otherwise SSH will give you errors.
• Major security warning: If someone breaks into Zim, they can also delete all of your backups on Ark. Never ever ever use the root user for backups on Ark. You can use the root user on Zim to send the backups, but its best to have a special backup user setup on Ark to receive the backup.

At the CherryPy BOF[1] at Pycon in Chicago we discussed how people are using CherryPy in lots of different ways. We’re afraid people aren’t grokking how flexible CherryPy is just by briefly glancing at the web site and thought having some interesting case studies would be useful.

The wiki has a good start. However, its a bit out of date, and it’d be nice to have some great Case Studies as well as a separate list of Projects using CherryPy.

If your company deploys CherryPy or you’ve used CherryPy for a client, I’d love to hear the story. Here’s a basic format you could follow:

1. Who is using CherryPy: (your name)
2. Where are you using Cherrypy?
   (Company or client name. You can of course remain anonymous)
3. What do you do with CherryPy?
   (What’s your project: a traditional public web site? internal web services?)
4. How is CherryPy deployed?
   (Behind modproxy/modpython/modwsgi, on a cluster behind a load balancer, or is it your frontend HTTP server?)
5. How long have you used CherryPy?
   (The “when” question)
6. Why did you choose CherryPy?
   (While “because its the best” is obviously the answer, a little more detail would be nice. )

Feel free to include screenshots, links, code snippets, diagrams, etc, and e-mail me!

With 3.1 nearing release, it’d be nice if we could launch an updated version of the web site at the same time, so your help is greatly appreciated!

I also posted this over at the cherrypy-user mailing list.

[1]

if excuse not in ['at funeral', 'giving birth', 'saving planet']:
	raise Exception('%s is no reason to miss the CherryPy BOF' % excuse)

If you’re hacking Django and get this…

ImproperlyConfigured: Error while importing URLconf 'proj.app.urls': 'tuple' object is not callable

…you’re probably missing a comma in your URL configuration as Rajesh Dhawan pointed out.

Django pros can move along, I know you never make typos.

Last night Dennis Gilmore, Build Engineer for OLPC, spoke at Bradley University. Since I live 3 blocks from Bradley and have been following OLPC off and on since the beginning, I couldn’t pass this up (even though I was the token old guy in a classroom mostly full of college undergrads).

Dennis had brought 3 XOs with him for the students to play with. He gave a good introductory presentation on OLPC, XO, and Sugar. Not much new for anyone who has been following the project, but I did learn that the XO runs each activity’s process in a different uid/gid. The students asked lots of good questions and were obviously excited about OLPC.

Read on if you care to hear my personal experience with an XO…
(more…)

The little recipe I posted for a dedicated Linux firewall with a CherryPy powered administrative interface finally has a permanent home over at Google Code.

Many thanks to Kyle Waremburg for creating the project page and helping me develop firewall-admin! I hope other people find it useful.

Today I deployed my first Django application for a client. Its yet-another-blog, so I’ll refrain from posting the code and cluttering up the django-*blog* namespace on Google Code. Before you roll your eyes and complain about why I didn’t use an existing solution, I think I have 2 somewhat valid reasons:

1. The client actually needed a subset of the features most blogs offer, so I wouldn’t really have anything to contribute back to an existing project.
2. Blogs are one of the simplest content driven web applications in existence. Wikis are just a bit simpler perhaps. At any rate, creating a blog app is an excellent way to learn a framework.

Python Deployment Decisions

In the past I’ve used CherryPy as my framework and a simple mod_proxy configuration to run the applications behind Apache. Django considers its built-in web server a development tool only, so I figured it was time to explore the myriad of Python web app deployment alternatives: mod_python, FastCGI, modwsgi. I’m sure there are many more, but I’d say those are the big 3.

I had tried to deploy Python web applications on DreamHost using FastCGI before and entered the hell that is deploying Python web apps on shared hosts. So FastCGI wasn’t my first choice this time.

I had also tried mod_python for deploying CherryPy apps on my Linode before and for whatever reason just found mod_proxy to be much easier to setup and manage.

I was kind of eager to try out modwsgi because its been getting a lot of attention lately, so I downloaded the source and compiled it on my Debian Etch server.

Deploying a Django App via modwsgi

modwsgi was quite easy to setup as long as you follow the instructions in their wiki for Django integration. I was hit by bug #3762, but the modwsgi documentation got me through it. (For what its worth the attached wsgi.patch also worked, but I don’t really want to run a patched version of Django.)

One big problem I ran into was sqlite3 gave me OperationalError: unable to open database file whenever I did anything that would write to the database. My database file was owned by www-data (the Apache process owner) and had the permissions 664.

I switched to PostgreSQL, ran syncdb, and everything worked beautifully.

My wsgi script file /srv/spam/eggs/eggs.wsgi:

import os, sys
sys.path.append('/srv/spam')
sys.path.append('/srv/spam/eggs')
os.environ['DJANGO_SETTINGS_MODULE'] = 'eggs.wsgi_settings'
 
import django.core.handlers.wsgi
 
_application = django.core.handlers.wsgi.WSGIHandler()
 
def application(environ, start_response):
    environ['PATH_INFO'] = environ['SCRIPT_NAME'] + environ['PATH_INFO']
    return _application(environ, start_response)

Note I use wsgi_settings instead of my usual settings file. wsgi_settings just imports my main settings file and changes some to their production values.

My Django application actually drops into the /blog/ and /accounts/ folders under a VirtualHost otherwise occupied by static files and some PHP scripts. modwsgi made this easy by putting this in my existing VirtualHost:

WSGIScriptAliasMatch /(blog|accounts)/.* /srv/spam/eggs/eggs.wsgi
 
# A simple Alias directive handles my static files
Alias /static/ /srv/spam/eggs/static/

Bottom Line

I highly recommend using modwsgi for deploying Python web applications. sqlite3 may work for you. In my case its probably best I use PostgreSQL for a number of reasons.