Archive for the ‘Python’ Category.

M2Crypto 0.20.1 Fixes Regression in httpslib.ProxyHTTPSConnection

Miloslav Trmac noticed a regression in httpslib.ProxyHTTPSConnection and provided a fix for it, and I’ve just tagged and uploaded the new 0.20.1 version to PyPI. See the 0.20 release announcement for general information about the 0.20 release series.

I’ve also been getting a few requests for help on building M2Crypto on Fedora Core -based systems, so I’ve added some info to the FAQ. Basically there is a fedora_setup.sh wrapper script in the source tarball that you can use instead of the plain setup.py. There are more details about the fedora_setup.sh script, if you are interested.

M2Crypto 0.20

I am pleased to announce M2Crypto 0.20, which has been in development for over nine months. It fixes over 30 bugs, and fixes and new functionality was contributed by more than ten people. Hooah!

The CHANGES file lists the full list of changes, but personally I am most pleased by M2Crypto having reached the magical 80% unit test (code) coverage. Besides that technicality, there are Python 2.6 fixes, threading fixes, added support for RSASSA-PSS signing and verifying, certificates with large serial numbers, and more.

Download from pypi.

Or use easy_install (may not work on all systems): easy_install M2Crypto

M2Crypto is the most complete Python wrapper for OpenSSL featuring RSA, DSA, DH, HMACs, message digests, symmetric ciphers (including AES); SSL functionality to implement clients and servers; HTTPS extensions to Python’s httplib, urllib, and xmlrpclib; unforgeable HMAC’ing AuthCookies for web session management; FTP/TLS client and server; S/MIME; ZServerSSL: A HTTPS server for Zope and ZSmime: An S/MIME messenger for Zope. M2Crypto can also be used to provide SSL for Twisted. Smartcards supported through the Engine API.

M2Crypto 0.20 Beta Cycle Begins

Better late than never… I am announcing the first beta of M2Crypto 0.20 release. M2Crypto is the most complete Python wrapper for OpenSSL.

The 0.20 release has been in development for about nine months. About 30 bugs and new features have been implemented by more than ten people. Unit tests now cover 80% of the code base. Tinderbox is used to automatically test changes on various flavors of Ubuntu, Fedora Core, Redhat and Cygwin. We could use more Tinderbox clients, so please drop me a line if you have some spare machine cycles available.

The release include some fairly significant changes, including tricky ones in threading and so forth. See the CHANGES file for list of changes. Please test your applications, and go file bugs on any issues you notice. I’ll wait for feedback for a week, spin the next beta and so forth until there are no more release blockers found within a beta period.

There are a few issues I feel bad about that did not make the first beta. If you can help create fixes for these, I’d be willing to consider including the fixes in 0.20 if the changes don’t look too scary. Here is my wish list:

Download from pypi.

Or use easy_install (may not work on all systems): easy_install M2Crypto

Remember your in operator

After you have worked with Python for a while you develop a sense that tingles when you are doing something that feels stupid, or wrong, on unaesthetic. I have found that more often than not someone before me apparently felt the same, and there is a more elegant, or Pythonic way of doing it. One of the simplest cases I find is with the in operator. Consider the following code:

if a == 1 or a == 7 or a == 15:
    do_some_cool_stuff()

This might be a personal pet peeve of mine, but that just feels tedious and somewhat prone to errors. Luckily there is a neater way of doing it using the in operator:

if a in (1, 7, 15):
    do_some_cool_stuff()

There, much less typing, and it looks nice, too!

Performance wise there isn’t that big a difference, at least on 32-bit Linux with CPython. Here are some comparisons with the help of ipython and timeit:

In [1]: a=16
 
In [2]: timeit a == 1 or a == 7 or a == 15
10000000 loops, best of 3: 167 ns per loop
 
In [3]: timeit a in (1, 7, 15)
10000000 loops, best of 3: 155 ns per loop
 
In [4]: timeit a in [1, 7, 15]
10000000 loops, best of 3: 157 ns per loop
 
In [5]: a=1
 
In [6]: timeit a == 1 or a == 7 or a == 15
10000000 loops, best of 3: 72.4 ns per loop
 
In [7]: timeit a in (1, 7, 15)
10000000 loops, best of 3: 83.4 ns per loop
 
In [9]: timeit a in [1, 7, 15]
10000000 loops, best of 3: 83.5 ns per loop

Office Resource Locator Ported to Django

After going through the Django tutorial, I decided to try how Django felt with a (small) real life application. I decided to port Solu (Self-service Office Resource Locator) – an application I originally wrote in Werkzeug.

Django-Solu gained two new features compared to the original: an administrator’s interface, and support for multiple maps. Administrators can add, delete and modify maps (as well as other resources), and when you create a resource you can select the map to put it on. There is no installable package, which is a regression from the original, due to the reasons described below.

At some level I was surprised how similar Django was to Werkzeug. For example, I was able to lift some unit tests and make just minor tweaks like changing a property name to get them to work in Django. There were also some very big differences, most notably in the database layer (compared to SQLAlchemy of which I have prior experience) and templates (compared to Mako Templates).

I can see how I can get stuff done in Django, but I was left puzzled with some pieces. Django seems to make a big deal about applications being self contained, so that (theoretically) you could create an app and anyone could download and install your app in their Django environment. Except this seems to fail a little bit with imports and templates, and spectacularly when dealing with static resources. Imports can be fixed by going through some hoops and modifying PYTHONPATH (which feels like a hack to me), templates by moving them inside the app directory (unlike what the tutorial instructs you to do) and calculating site root etc. in settings.py file (which isn’t actually one of the “standard” settings so you can’t rely on it), but I haven’t figured out any decent solutions for static resources. The problem with static resources is that I don’t know how to point to static resources from templates. Currently I just made these absolute paths, which may need to be edited based on deployment situation. Maybe there is a simple solution I am overlooking, since I can’t imagine Django being so popular with this kind of limitation.

The templates themselves kind of force you to not put any code in the templates, but this has in turn resulted in some bizarre constructs to work around the no code limitations. For example, the template language still allows you to do simple arithmetic with tortured syntax: my_variable|add:"-10" adds -10 to my_variable before printing it out.

The database layer was simple to work with, at least in my simple use case, although there was one point where I was stumped for a while. When searching for resources, I wanted to order the search results in certain way. But there does not seem to be a way to do that in the query syntax. Instead, you need to add a Meta class as a child of each model class and add ordering class attribute. I don’t see how this would work if you wanted to change ordering on the fly or if you have joins and want to order by attributes from different tables. It was also a little difficult to find this information about ordering.

All in all I spent 9-10 hours porting the application, which was roughly what I expected. I spent another 2 hours deploying a live demo application on Dreamhost, which was somewhat more involved than I expected.

Django First Impressions: The Tutorial

I finally started looking at Django, after mostly working with Pylons with a little side project in Werkzeug. I spent a couple of hours going through the Django tutorial, and wrote down my impressions while they were still fresh in my mind. The comments are a mix about Django and the tutorial and the Django website.

  • First of all I was surprised that the installation instructions did not mention you can run easy_install django to get going, even though egg files were mentioned.
  • Main tutorial is for SVN “release”. Personally I don’t consider it a release if it is just an svn revision! You need to notice to click a link to go to the latest actual release tutorial. These seem backwards to me; I’d expect the main version to be for the release, with a link to go to the SVN version.
  • Python config file makes me feel uneasy.
  • Absolute path requirement for config file options seems to make deployment a hassle.
  • Settings file location needs to be specified in an environment variable, which feels a little brittle and hacky.
  • Database API feels hacky with double underscore constructs, but on the other hand model creation seemed simpler than with SQLAlchemy.
  • The tutorial flowed smoothly until it was time to change admin templates. I was expecting a manage.py command to write a default admin template, but instead had to manually copy a file from the Django installation.
  • Admin stuff looks good, but why does the tutorial then break it?
  • There does not seem to be an interactive debugger in browser. Pylons and Werkzeug have this and I love it.
  • Development server does not restart automatically when new files are added. I can’t remember if Pylons/Werkzeug had this problem.
  • The term app isn’t properly explained when first used.
  • View term usage different from what I am used to.
  • I find myself preferring Django’s urlconf over routes at this point, seems clearer.
  • After getting used to Mako templates that allow you to run python, Django templates looks limiting and with some arcane constructs (witness forloop.counter).
  • I was surprised there was no sight of the popular “20 minute wiki” guide. Although a little web search shows showmedo video with this topic. After Pylons documentation I was also expecting some links to screencasts, but didn’t notice any. Again, doing some web searches lists several, but for some reason none prominently listed from the Django site (from the pages that I visited).

I was also left with the impression that I could replace pieces of the framework I don’t like with something else, for example the templates. I don’t know how difficult this is in practice, or how many components are meant to be replaceable. But at least the idea is there, which makes someone with Pylons background breath a little easier.

On to coding my first Django app, then.

P♥THON

A few weeks ago I noticed a car at a gas station just as I was pulling out that had one of the best license plate texts I have seen in a while. I wish I had had more time; I would have wanted to ask the driver if the plate really meant what I thought it meant.

Today I saw the same car, and had a bit more time to snap a picture. Unfortunately the driver was nowhere to be seen, so it will still remain somewhat of a mystery. However, given the text around the plate I am pretty sure that the owner of this cool plate likes the Python programming language:

California P♥THON License Plate

California P♥THON License Plate

Importing fabric into your own scripts

“Fabric is a simple pythonic remote deployment tool” states the Fabric website. But fabric can offer more (or could, if certain assumptions were fixed). Fabric offers simple function calls to run commands locally, over the network, upload and download files and so on. You can of course program all this yourself using various stdlib and 3rd party libraries, but the simplicity of the fabric API feels nice, like this example shows (using fabric 0.0.9 or earlier):

import fabric
 
fabric.local('echo hello')
 
fabric.set(fab_hosts=['host-a'])
fabric.put('myfile.txt', 'myfile.txt')
fabric.run('uname -a')

I had made a script that used fabric like this. It worked fine when I tested against a single host. But when I tried to connect to different hosts in my script, and doing different things on each host, I run into a problem. It turns out that fabric cached the connection to the first host, and later on all my operations were executed against the first host. Luckily I found a simple workaround for that. After each fabric.set() do this:

fabric.CONNECTIONS = []

It may be that fabric caches other fab_* parameters as well, but my use case worked so I haven’t looked further into this yet.

Unfortunately fabric 0.1.0 breaks this import pattern. The developers are aware of the issue, though, so it may be that later versions of fabric could be used like this as well. But for the time being you should mark the dependency at 0.0.9 or 0.0.8.

There are some additional warts when importing fabric into your script. While it is great that fabric notices when a command fails, it will actually exit rather than raise an exception. Probably not an ideal situation for most scripts, so it would be nice if this was configurable. Another feature I would really like added was timeouts for all of the commands. Running external commands that can be killed after a timeout is difficult in Python.