California Dreams

I was testing a Python web application that was using FastCGI deployment on Dreamhost, when I found myself looking at a souped up Python Traceback in my browser. At first I couldn’t understand why that was happening. As far as I knew I was running with full production settings and as such I would have expected a terse internal server error message.

Looking at the HTML source of the error page I discovered reference to cgitb. But as far as my code was concerned I did not set that. I tried specifically disabling that in my script but that made no difference. In a momentary act of desperation I did a find for all cgitb.py files under my account and made the cgitb.enable() function do nothing. Yet I was still seeing the tracebacks.

After a bit of scratching my head and throwing different words at Google it occurred to me to take a look at the fcgi.py script. Oops. The [WSGI]Server class has an error() method whose docstring states that it “May and should be overridden”. No $%^, the default just plasters all the dirty little secrets for the world to see! I’d like to see something like Debug[WSGI]Server that pretty prints the error, and leave the [WSGI]Server the production class. The naming would make it clear that you should not be using the debug version in production. As it is now, I wonder how many people actually read all the way towards the bottom of the 1331 line file to discover this gem.

I also added a warning to the Dreamhost documentation regarding Python FastCGI.

I solved the locked out Trac admin problem by creating a second admin using the command line trac-admin tool, and then logged in as that admin to delete the broken admin account (it is weird that admin can not edit accounts in Trac, only delete them). I then recreated the original admin, and as I had set up outgoing email even that worked. After I confirmed everything was working I nuked the temporary admin.

I was able to get a pretty good improvement in performance by symlinking the trac/htdocs/common into my site’s chrome/common. Initial wait was still long, but all pages I tried seemed to be served under a second, compared to some taking over 20 seconds before this operation. Usable for me only although the initial wait was long, but would probably not be usable with a group of people hammering the server.

Next I looked at FastCGI. The documentation claims it should be as simple as changing index.cgi references in .htaccess to index.fcgi, which in turn differs from index.cgi only by calling trac.fcgi instead of trac.cgi. trac.fcgi seems to be invoking a variant of Allan Saddi’s fcgi module. But try as I might, I was only getting 500 internal errors with the log not being very helpful:

[Wed Jul 16 21:55:31 2008] [error] [client ...] FastCGI: comm with (dynamic) server ".../index.fcgi" aborted: (first read) idle timeout (60 sec)
[Wed Jul 16 21:55:31 2008] [error] [client ...] FastCGI: incomplete headers (0 bytes) received from server ".../index.fcgi"

I also didn’t see any *.fcgi processes, like I see with my horsetrailratings.com site.

I thought maybe this was an instance where I would need to file a support ticket to see if raising Apache2 Softlimit would help, but suddenly FastCGI started working. As far as I know, I didn’t touch anything that should have made a change in this, so I am still quite clueless to what happened.

The final point about the Trac installation which I was happy to note was that my paths are pretty out of the box (no index.fcgi in the path), so I don’t need to resort to any of the path hacks mentioned in various guides on how to install Trac on Dreamhost.

I am working on launching a little project, and thought about using Trac to manage the development. This is the first time I am setting up Trac, and given that people seem to be having lots of problems installing Trac on Dreamhost I figured this would be a painful experience.

First I created the svn.example.com and trac.example.com sites, and I think I actually made the installation of Trac simpler because of this. Another option would have been example.com/trac, which I believe would make some of the later steps harder. I also used the Dreamhost Panel to create the Subversion repository at svn.example.com/svn.

First of all, there is a nice script for making Trac install nearly painless on Dreamhost. Unfortunately the script is hardcoded to use the system Python 2.3, and at least one of the plugins the script installs is not compatible with 2.3. I found this out by installing, then changing the admin preferences by adding an email address which completely broke the installation: any URL would just result in a Python stack trace.

Which reminds me that I find it really strange that Trac is giving the full stack traces in the web UI; that is a security problem. Pylons for example does that only in the development mode by default, and in production mode the default puts errors only in the log (and it can also email the error to administrator if email is set up). After brief search I have not seen a way to stop the stack traces from appearing in the web UI.

I then figured I’d install Trac the hard way, by hand. How hard could it be, right?

First I needed to set up some prerequisites as was described in the official Trac installation guide (some of these at least might be installed automatically by setuptools, but I installed what I wanted in advance). I had installed Python 2.5 earlier, so that was no problem. SWIG was already high enough version so I didn’t need to do anything about that. I figured I wouldn’t need Clearsilver or SQLite. First I installed setuptools, then it was trivial to install python-mysql (mysql was already on the system), genshi, docutils, pygments and pytz. Naturally Apache was already provided by Dreamhost as well, but even though Subversion was provided I also needed to build one myself to get the Python bindings for 2.5. To make managing the repository easier, I chose to install the exact same Subversion as was the system wide version; that way I could use the Dreamhost Panel to manage the repository and give Trac access to it as well. Of course, if Dreamhost ever updates its version I will be in trouble… The build and installation were a breeze with good instructions. Finally easy_install Trac went through without a hitch. I also installed Account Manager plugin which turned out to be a mistake.

Then came a pretty hard part; how to create a Trac site. I adapted pieces of the easy Trac install script into my own setup, and managed to use trac-admin to create my site. I was also able to serve it with tracd, but this is good only for testing. Next I tried to figure out how to serve Trac as CGI. I was stopped cold by not finding the cgi-bin directory that was mentioned in the instructions. After a bit of fumbling and doing web searches I figured out how to use trac-admin deploy command (trac-admin /path/to/trac/env deploy /path/to/dump/cgi-bin-etc).

Next step was putting in the index.cgi and .htaccess files. I started from what the script install used, but it turned out all I really needed for index.cgi was:

#!/bin/bash
export TRAC_ENV="/path/to/trac/env"
export PATH=/path/to/python2.5/bin:$PATH
exec /path/to/cgi-bin/trac.cgi

The cgi file naturally needs to be executable.

You will also need to make trac.cgi executable, and change the hash bang line to /usr/bin/env python - the default will pick up the old system Python. Note that trac.cgi will output stack trace to the web in case of error, so you should take out the line that does that and leave just the line that prints into log.

The last struggle was with .htaccess, which turned out like so (you may want to modify this to allow the Dreamhost stats gathering of your site):

DirectoryIndex index.cgi
Options ExecCGI FollowSymLinks
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ index.cgi/$1 [L]
</IfModule>

Oh, lest I forget, I also created the admin user, admins group, and the default password for the admin using the procedure described in the easy Dreamhost installation script.

After this I was able to browse the Trac site fine. I logged in as administrator and changed the password. Then I made the mistake of putting in an email address for the administrator. Apparently (?) due to the Account Manager plugin the admin account was made limited until email verification was completed. The problem was that I had not set up outgoing email yet. And then I made another error by clearing the admin email, which resulted in getting a stack trace on every page while logged in.

I’ve since set up outgoing email, which proved simple by using a GMail account. But the admin account is still hosed, and I am too tired to figure that out right now.

Yesterday I was puzzled to note my site horsetrailratings.com was reporting internal server error. Looking at the server logs did not help. Running the FastCGI script by hand gave a mysterious traceback that ended with:

ImportError: /home/.../lib/python2.4/lib-dynload/collections.so: undefined symbol: _PyArg_NoKeywords

Experimenting further it seemed just importing many stdlib modules resulted in the same thing, for example import threading failed with that. It seems others have run into the same problem with their virtual python installations. I filed a support request at Dreamhost, but they responded saying Python is working just fine. And it is, it is just that virtual python installations broke.

I decided I would use this as an excuse to try and get Python 2.5 running on Dreamhost and migrate my application to that. It turned out to be easy with these instructions. I had tried compiling Python on Dreamhost before, but had always ended up with a binary that did not work. The working incantation is: ./configure --prefix=$HOME/opt/ --enable-unicode=ucs4

I guess I will need to set some kind of monitoring system for my web app so that I will know about outages sooner rather than later…

I updated horsetrailratings.com today… I mean yesterday.

I made one painful decision: users need to create an account before they can add trails. Another option would have been to create a system where changes need to be moderated. The main reason I decided to do user accounts instead is because most web applications will actually need user accounts at some point and I wanted to learn to deal with those sooner rather than later. I looked at AuthKit, but the documentation regarding the Pylons integration seems to be a little misleading or incomplete based on the comments on the Pylons wiki. That is what I would probably choose for a production system, still. But given how easy it is to bolt user management into standalone Pylons I decided to do that first (and learn a bit more about Pylons in the process). All the hard work went into making the login forms, and the part that I was originally most concerned about - tracking the logged in status - was trivial with Pylons session object. Add a little decorator to ensure users are logged in for certain controller actions and you are good to go.

Most visible in the new version is of course much better UI, although some work still remains. It also seems to work reasonably well in other browsers, although I noticed a small gap in IE7 under the menu buttons and the banner lower border on which they are supposed to rest. The add trail page is perhaps the most polished, with WYSIWYG HTML editors and stars you can click to rate the trail.

After deploying on Dreamhost I noticed one regression. The FastCGI script shows in the URLs even though I put in the fix for broken routes. This worked in the 0.1 version so I am not sure what is going on yet. (Update: Found a typo that had slipped in, all good now. Need more tests…)

So finally, after a few false starts, I have a Pylons-based application running on Dreamhost.

My experience was that the only way to get a working Python installation was to use Virtual Python, as described in the Python on Dreamhost wiki page. The Python 2.5.1 that I compiled myself was crashing left and right, and rather than try and figure that one out I used Virtual Python with python2.4.

Another piece of the puzzle is that you need to use FastCGI on Dreamhost, because Dreamhost seems to kill long running processes like Python. Also, you need to go with the “obsolete” fcgi.py rather than Flup, because Flup seems to be too slow for Dreamhost. The FastCGI script does not need to be named dispatch.fcgi - I have used other scripts with .fcgi suffix successfully, all chmod 755. Using differently named FastCGI script has the benefit that you can easily kill just the FastCGI for a specific application of yours. This post provided a usable FastCGI script for Pylons-based application. The Pylons documentation includes other samples, like this, but I have not yet deployed a real application so just noting that for reference.

In my previous post I described how I kind of got TurboGears running on Dreamhost. The problem was that it died after a few seconds. Crawling the wiki and other sources, it seems like there is a process that will kill the Python process, unless it happens to be named dispatch.fcgi. So I scrapped that plan (rm -fr my.domain.name) and decided to start again.

Looking at the Python FastCGI page I first easy_installed Flup (as described at the end of the page), created .htaccess file to redirected everything to dispatch.fcgi like instructed on that page, and made the dispatch.fcgi like in the Flup section of that page, chmod 755 dispatch.fcgi, waited a little, and tried browsing to my site. After a long time, it finally returned server error. Not good. But since there is so little code now, it is easy to debug. I put just a little exception handler around the whole script, and made it write to a log file any exceptions. And sure enough, there was error! Now we were getting somewhere. The error was about being unable to import flup. Hmm, so it looks like my Python environment wasn’t working through FastCGI as I expected.

So I did an alternative approach by using fcgi.py (wget http://svn.saddi.com/py-lib/trunk/fcgi.py into my.domain.name). Then I edited the dispatch.fcgi to match the dispatch.fcgi like in the beginning of the Python FastCGI wiki page on Dreamhost. Waited a while. Browsed to my site. Waiting, waiting, waiting…. IT WORKED!

So, I think I’ll either do my own Python 2.5 installation or virtual Python setup. Both approaches are described on the Python on Dreamhost wiki page.

I think I am also going to forget TurboGears and move to Pylons, since it seems Pylons is the future.

Update: See my Pylons on Dreamhost post for the final details.

I have been reading comparisons about TurboGears and Django and the other Python web frameworks. There doesn’t seem to be a clear winner in the documents I’ve read, but there are quite a few comparisons I have yet to read. But my initial impression is that with Django you can get an application up quickly, while TG would give you more power to tweak things. And that TG reuses 3rd party packages, while Django project creates similar packages for itself. This latter point made me start my experimentation with TG, although that commentary was from 2006 so maybe things have changed.

I read the documentation on how to set up TurboGears on Dreamhost, and it all seems dated and overly complicated. I plan on updating those instructions, but here’s what I did to get the Wiki20 sample running:


mkdir -p ~/bin
mkdir -p ~/python2.4/site-packages

cat >>.bash_profile
alias python=python2.4
export PYTHONPATH=/home/<username>/python2.4/site-packages
export PATH=/home/<username>/bin

source .bash_profile

wget http://www.turbogears.org/download/tgsetup.py
python tgsetup.py –install-dir=/home/<username>/python2.4/site-packages –script-dir=/home/<username>/bin

cd my.domain.name
tg-admin quickstart

mv Wiki-20/* .

# edit .htaccess
# edit dev.cfg

python start-wiki20.py

#at this point I run into the following error:
You are required to install SQLObject but appear not to have done so.
Please run your projects setup.py or run `easy_install SQLObject`.
#end error

easy_install –install-dir=/home/<username>/python2.4/site-packages –script-dir=/home/<username>/bin “SQLObject>=0.7.1dev-r1860,<=0.7.99″

python start-wiki20.py

#at this point it complained the port was taken, so I experimented until it found an unused port
#browsed over to my.domain.name which showed the app running
# but alas! it died immediately after serving that one page:
2008-01-31 21:14:00,909 cherrypy.msg INFO ENGINE: SystemExit raised: shutting down autoreloader
2008-01-31 21:14:00,910 cherrypy.msg INFO HTTP: HTTP Server shut down
2008-01-31 21:14:00,914 cherrypy.msg INFO ENGINE: CherryPy shut down

Update: TG seems to actually go down soon after start, regardless of whether it serves any pages or not.

Update 2: See my second attempt at just using Python on Dreamhost, followed by the final bits for working Pylons on Dreamhost.