Small update on my toy tc-rest project: I switched to using WebOb for creating HTTP Request and Response objects. Cleaned up the code a bit, but a real dispatcher is what’s needed to really remove the cruft.
I’m anxious to extend the API and add features, but I have no clue when I’ll have time to touch it again. In the mean time I’ve pushed tc-rest to bitbucket.org if you want to take a look.
Have you ever wondered how hard it would be to tack a RESTful HTTP interface on top of a fast key/value database like TokyoCabinet?
Probably not, but I did: tc-rest.tar.gz
Components:
Getting TokyoCabinet+pytc to work inside a virtualenv was a bit tricky, so check out my run.sh script if you’re having trouble getting it to start.
Once you get it started, load okapi in your browser:
http://localhost:8080/static/okapi.html
And then create a database by doing a POST like:
http://localhost:8080/foo/
And finally store/get keys and values using GET and POST requests like:
http://localhost:8080/foo/bar/
http://localhost:8080/foo/baz/
Doing a GET request to a database URL lists keys.
At any rate, I had big dreams for building a system where you would store JSON, specify indexes on certain keys, and the server would maintain those indexes for you by creating ad hoc TokyoCabinet databases.
Instead I ended up wasting most of my time learning how to write a low-level WSGI app. I should have just used CherryPy or Django from the beginning, but I had never written a pure WSGI app before. It was a good lesson even if it meant not getting some of my features implemented.
I’ll probably keep playing with this idea, but the next version will probably be based on some existing framework. Parsing environ['PATH_INFO'] and running start_response(...) manually gets old fast.
fapws3 is pretty neat, but had lots of annoying rough edges. I had to manually create a README file because its setup.py expects one to exist. Then I had to manually allow DELETE HTTP methods in fapws/base.py, otherwise it would return an HTML error message for me! That was a bit shocking since I was working under the assumption fapws3 is just a low-level HTTP/WSGI server.
Update
I did an introductory talk on Tokyo Cabinet, Tokyo Tyrant, pytc, and PyTyrant at the Portland Python User Group meeting last night.
It was definitely just an introductory talk, so if you’re already familiar with these projects it probably won’t be very interesting.
Errata: My slides state PyTyrant 1.1.11 doesn’t work with Tokyo Tyrant 1.1.17 — this is wrong. I don’t know what I was doing yesterday, but today all my tests work on Tokyo Tyrant 1.1.17.
Recently I deployed a Django project on an OSX server. I foolishly thought this would be as easy as on Linux until I ran into the mess that is x86_64 Apache + mod_wsgi* + Django + psycopg2 + i386 PostgreSQL. After wasting far too much time googling and recompiling various bits trying to get everything happy, I followed Eric Florenzano’s post on deploying Django using CherryPy’s** wsgiserver.
Here’s my lightly modified version of Eric’s script:
import wsgiserver import sys import os import django.core.handlers.wsgi if __name__ == "__main__": # Setup paths - a bit hackish, but works for me. # Assumes an absolute path is stored in <project>.local_settings.ROOT sys.path.append(os.path.realpath(os.path.dirname(__file__))) from foo.local_settings import ROOT sys.path.append(ROOT) # Startup Django os.environ['DJANGO_SETTINGS_MODULE'] = 'foo.settings' server = wsgiserver.CherryPyWSGIServer( ('0.0.0.0', 8888), # Use '127.0.0.1' to only bind to the localhost django.core.handlers.wsgi.WSGIHandler() ) try: server.start() except KeyboardInterrupt: print 'Stopping' server.stop()
I also went with the latest stable version of CherryPy’s wsgiserver instead of checking out trunk like Eric’s post suggested.
Then I just enabled mod_proxy in Apache and setup the following VirtualHost:
<Proxy *> Order allow,deny Allow from all </Proxy> <Location "/"> ProxyPass http://127.0.0.1:8888/ ProxyPassReverse http://127.0.0.1:8888/ </Location>
If you’re cool you’ll write some sort of system specific script to launch your web app on boot. In a pinch, you can always use a crontab:
@reboot /usr/bin/python /path/to/app.py &
YMMV 
* To mod_wsgi’s credit, it took about 10 seconds to compile, generated a Universal binary, and in general Just Worked.
** I’m already a CherryPy fan thanks to dowski, so it wasn’t a hard decision.
Apache was probably the first Linux application I learned how to configure. However, I’ve gotten a bit frustrated with it recently…
The Problem
A memory leak. Apache is eating up memory so quickly that I need to restart it every couple days or risk my entire server grinding to a halt as it starts swapping wildly. I’ve poured over log files and pmap output, but I still can’t figure out where the problem lies. Curse you monolithic in-process architecture!
Actually I know what my problem is, I’m running a mess of modules:
ssl – 2 certificates on 2 portsphp5 – blerg, who doesn’t have to run this?suphp – I suspect this is my problem, but I can’t prove it. A client’s 3rd party web application requires it, but I think its easily replaceable with FastCGI.wsgi – No complaints. Python apps are out-of-process thankfully.proxy – Again no complaints. Can’t imagine how this module could cause any problems except it does proxy some large (multi-megabyte, not huge) POSTs at times. I can’t imagine a memory leak could slip into this module without a lot of people noticing.Solution A: Apache+FastCGI
I love the idea of putting each web application in its own process and letting Apache just act as an HTTP router. FastCGI seems to have all the features I need, and I’m not really worried about the CPU overhead incurred by IPC.
However, there are 2 competing FastCGI modules for Apache, and I have no idea what to choose. Anecdotally the official mod_fastcgi is buggy and fastcgi.com is a spam infested wasteland. However, I’ve found no authoritative source saying: “fastcgi is dead, long live fcgid!” (Lame excuse, I know.)
Solution B: Lighttpd
I know Lighty is the darling of Rails sites, but whenever I stop by its site I’m greeted with a list of recently fixed security bugs, and now it seems as though they’re rewriting the core!
I’m sure Lighty is a high quality intelligently engineered project, but it seems to be the definition of immature. Not necessarily bad (in fact it usually means its progressing quickly!), but perhaps not as reliable as good old workhorses like Apache.
Solution C: Cherokee
I’ve been following Cherokee for some time now and running it locally on my workstation. I love the web interface. I’m usually a very anti-webmin, pro-vim kind of guy, but I’m sick of editing Apache’s config files. I do it about once a month and therefore it always takes lots of double-checking the docs. I don’t know why, but its configuration has just never felt natural to me.
However, the lead Cherokee developer’s bravado is by the most off-putting aspect of the project. He mocks modwsgi and posts simplistic benchmarks showing Cherokee to be the fastest web server, but meanwhile Cherokee churns out numerous bug patch releases in-between feature releases and has yet to reach 1.0 status.
It seems like an excellent project technically, but I’m afraid there will be negative consequences for the lead developers hubris. (I’m not meaning to insult the guy. He’s probably a far better hacker than I’ll ever be. Self-promotion just makes me uncomfortable.)
Solution D: nginx
I don’t know much about nginx except that it works. Basically all I’ve heard about it is:
While “working” is definitely my primary objective, nginx seems a bit bare bones for me. I just don’t think I’m the target demographic. I’d kind of like for my web server to handle spawning and kill of FastCGI processes.
nginx feels like git to me. Those who know it: use it and love it. Those who don’t: stand in fear and awe of its unbridled power.
…or maybe its just a nice simple barebones HTTP server…
Conclusions?
I think Solution A: FastCGI is the most sensible. Apache has always served me well, and the memory leak is most likely due to that shoddy suphp module.
Moving my web applications to FastCGI is also the best way to prepare to move to one of these 2nd generation web servers.
However, I’m getting kind of sick of Apache, and the ambiguousness of which FastCGI solution to choose is fairly annoying.
So dear lazyweb, for your everyday web developer consultant looking to run a bunch of PHP and Python web applications, what HTTP server stack should I use? (Debian Lenny packages are a plus.)
Gnome notifications popup in the lower left corner of your desktop by default* which constantly annoys me. I usually have a terminal open in the lower left corner, and having my work covered by notifications is quite annoying.
Luckily the fix is easy:
apps > notification-daemonpopup_location to be something less annoying. I prefer top_right.~$ killall notification-daemon ~$ /usr/lib/notification-daemon/notification-daemon &
Note that the notifications will actually show up over the top of your panel which seems a bit strange. However, I’d rather the notification covered the panel than take up any more precious application space than is necessary.
Luckily you can easily test notifications if you have Python and python-notify installed:
>>> import pynotify >>> pynotify.init('foo') True >>> pynotify.Notification('foo', 'bar').show() True
* at least on Gnome 2.22.3 on Debian Sid with notification-daemon 0.3.7-1+b1
** aka gconf-editor from the gconf-editor package which should be installed with Gnome.
