California Dreams

Better late than never… I am announcing the first beta of M2Crypto 0.20 release. M2Crypto is the most complete Python wrapper for OpenSSL.

The 0.20 release has been in development for about nine months. About 30 bugs and new features have been implemented by more than ten people. Unit tests now cover 80% of the code base. Tinderbox is used to automatically test changes on various flavors of Ubuntu, Fedora Core, Redhat and Cygwin. We could use more Tinderbox clients, so please drop me a line if you have some spare machine cycles available.

The release include some fairly significant changes, including tricky ones in threading and so forth. See the CHANGES file for list of changes. Please test your applications, and go file bugs on any issues you notice. I’ll wait for feedback for a week, spin the next beta and so forth until there are no more release blockers found within a beta period.

There are a few issues I feel bad about that did not make the first beta. If you can help create fixes for these, I’d be willing to consider including the fixes in 0.20 if the changes don’t look too scary. Here is my wish list:

• Should work with OpenSSL 1.0.0
• A regression dealing with RSA keys
• Problems with Unicode URLs
• verify_cert not implemented
• CRL bindings

Download from pypi.

Or use easy_install (may not work on all systems): easy_install M2Crypto

I happened to read a post on how to detect if someone has changed files on your webserver to serve nebulous scripts and what not. The idea in the post was to compute hashes of the files on your server and then compare periodically that the hashes match. This works against some simple attacks where all the attacker does is modify some of your content files. It won’t work if the attacker has also gotten access to the script doing the checks, or the hashes, and so forth.

As I was reading the post I realized that if you use a version control system like Subversion to publish changes to your site (the live site is a checked out copy), you get this automatically. All you need to do additionally is to set up a cronjob to run svn status (or equivalent if using some other version control system) on the server. svn status does not print anything if there are no changes, but for any added, deleted or changed files or directories it will print one line of output. Of course, this only works for files, not for content coming from databases.

Ever since M2Crypto got support for Elliptic Curves (EC) cryptography, it has been somewhat difficult to build M2Crypto on systems where OpenSSL has been built without EC support. Notably distributions based on Fedora Core, which besides Fedora Core include of course Redhat and CentOS.

Disabled EC support alone wouldn’t be an issue, since normally opensslconf.h defines OPENSSL_NO_EC, but those same systems have also changed OpenSSL build so that instead of opensslconf.h you need to include processor architecture dependent file. And if that wasn’t enough this isn’t actually what you are supposed to be doing, and you will hit a compiler error to notify you of that. The final step in the recipe is to tell SWIG to treat errors as warnings. The distributions make build changes in their own versions of M2Crypto, but unless you know the recipe, you are going to have a hard time building M2Crypto yourself. Miloslav Trmač showed me what kind of changes were needed, and I made a new setup wrapper fedora_setup.sh which should help you get off the ground with M2Crypto if you are having a hard time building it. Let me know if you run into any problems with the setup wrapper.

I am sure the distros did not make these changes just to make it harder to build systems based on OpenSSL, although it sure does feel like that at times. Maybe someone can shed light on why these changes were made in the first place.

I have been working on an installer system that needs to ask the user a bunch of questions, including passwords, and store the answers in an .ini so that if the user runs the installer again, they will get prefilled answers. After the first installation it should be possible to run the installer in a batch mode since all the answers were recorded. But being security conscious, I did not like all those passwords being stored in plain text.

Luckily for me, I had already implemented almost exactly what I needed in Chandler, namely the password manager feature of Chandler which encrypts account passwords with a master password. I just needed to rip out the Chandler-specific features to get a general purpose encryption and decryption library, which defaults to 256 bit AES (Rijndael) in CBC mode and derives a key from the master password using PBKDF2 algorithm. The encrypted data serializes to just text, which can appear in .ini file. The serialized form does not follow any standards. With a few extra lines of code it can also act as a utility to encrypt and decrypt files. Thus, m2secret was born. It is built using the M2Crypto library.

Here’s how you could use it:

import m2secret
 
# Encrypt
secret = m2secret.Secret()
secret.encrypt('my data', 'my master password')
serialized = secret.serialize()
 
# Decrypt
secret = m2secret.Secret()
secret.deserialize(serialized)
data = secret.decrypt('my master password')
 
assert data == 'my data'

The command line utility is called m2secret.

Installation should be as easy as easy_install m2secret.

Recently I was admonishing my colleagues for embedding configuration options into Python code. I thought it was commonly accepted that all configuration options should be in text files that are not code. Typically in Python world this seems to be something like INI files, which are parsed to get to the values. Nowadays JSON could also be an option. After polling the opinions of some other colleagues, friends and others, it seems the situation is not as clear as I thought.

I think I have personally embedded configuration options in one case in production code, and felt dirty afterwards. But maybe I have just been living in the past, thinking the thoughts of a C/C++ programmer…

Some of the first things that come to my mind when thinking why configuration options should be in regular text files:

• Text (INI format especially) is easier to edit than (Python) code
• It is easier to reload configuration options from a regular text file
• Code files should be read only
• If you make a mistake in a code file, you will get an error that only programmers will understand
• Loading configuration options from a regular text file offers better security

The counter arguments to those could include:

• It is actually simpler to write complex structures in Python than INI, and you can use all of Python’s power
• It can be pretty easy to reload Python modules as well (although you have to be careful with imports). Often there is no need to reload options.
• Of course the code file would be made writable for only a brief period of time
• If the target audience of the program are programmers, it makes perfect sense to show errors only programmers could understand. And you could even catch some errors and give nicer messages to end users.
• The configuration file can be edited only by the super users (or the legitimate intended users of the program, and the program can not grant them more power than they would otherwise have)

I can think of a bunch more reasons but they all seem to follow pretty much the same pattern, with same pattern of counter arguments.

So if the intended users of the application were software engineers, systems administrators or others capable of deciphering Python tracebacks, and the program couldn’t grant the user more access to the system than they would otherwise have, I guess I don’t have a strong case against config-in-code.

But in most use cases I think the security implications are actually a pretty strong reason why config should be separate from code. You can lock down code as tightly as your system allows, and you can be as strict about parsing and loading values from text config file as you like. This doesn’t eliminate all security issues from config files, but it adds a layer of protection.

I have set up a Tinderbox2 instance for M2Crypto. If you are unfamiliar with Tinderbox, it is a tool to do continuous builds and tests, similar to Buildbot.

Now I am looking for people who would be able to run buildslaves. Ideally computers that are online 24/7, able to do a build + test of M2Crypto once an hour, but even once a day would be useful. The computer should have the ability to pull sources from svn and email the build & test log. The build and test cycle takes less than a minute on my laptop, so we are talking minimal resource usage.

Secure Sockets Layer (SSL) support in Python stdlib took a big leap forward with the introduction of the ssl module in Python 2.6.

Servers

The ssl module provides the means to build safe SSL servers just by using a stock installation of Python 2.6. There are still some pitfalls as usual, but it can be done by limiting compatibility.

The main problem with the ssl module in 2.6 in server use is that it is not possible to disable SSL version 2. This is important since SSL version 2 is insecure. And unfortuantely the most compatible mode selection with OpenSSL (on which Python’s SSL support is based) is to select the SSLv23 mode which on the server is compatible with all SSL and TLS versions supported by OpenSSL. OpenSSL does provide a way to select the SSLv23 mode and selectively turn off any SSL version, but this was not implemented in Python 2.6. It is available in at least M2Crypto, which is a 3rd party Python wrapper for OpenSSL that I maintain.

The 2.6 implementation will also not let you choose ciphers, so you are at the mercy of the protocol version you choose and the options that OpenSSL was compiled with. M2Crypto defaults to strong ciphers, and lets the application developers set any ciphers they wish.

With those important caveats, the server side SSL implementation looks reasonable, assuming you remember to provide the keyfile and certfile arguments to ssl.wrap_socket(). Especially if you can forgo some compatibility and limit your server to just TLSv1. (Transport Layer Security or TLS for short is the newer SSL.)

I wrote earlier how you can use M2Crypto to extract CA certificates from NSS project’s certdata.txt. You can also get free SSL server certificates that work in many (but not all) browsers from StartCom, for example. You still need to pay to cover all the mainstream browsers. In some use cases you may also be able to run your own CA and issue your own certificates but it is not for the faint of heart.

Here are sample servers written using stock Python 2.6 with SSL support compiled in, and M2Crypto 0.19:

Python 2.6 server:

import socket, ssl
bindsocket = socket.socket()
bindsocket.bind(('', 8000))
bindsocket.listen(5)
newsocket, fromaddr = bindsocket.accept()
c = ssl.wrap_socket(newsocket, server_side=True, certfile="servercert.pem",
                    keyfile="serverkey.pem", ssl_version=ssl.PROTOCOL_SSLv3)
print c.read()
c.write('200 OK\r\n\r\n')
c.close()
bindsocket.close()

M2Crypto 0.19.1 server:

from M2Crypto import SSL
ctx = SSL.Context('sslv3')
ctx.load_cert(certfile="servercert.pem", keyfile="serverkey.pem")
bindsocket = SSL.Connection(ctx)
bindsocket.bind(('', 8000))
bindsocket.listen(5)
c, fromaddr = bindsocket.accept()
print c.read()
c.write('200 OK\r\n\r\n')
c.close()
bindsocket.close()

The situation is not so good for the client side, although things have definitely improved.

Clients

The same SSLv2 issue and cipher issue is true for the client side as well. As a client developer, besides needing to specify ca_certs pointing to valid CA certificates for ssl.wrap_socket(), you will also need to remember to pass in cert_reqs=ssl.CERT_REQUIRED.

In addition, the client side needs to check that the certificate hostname (in the subjectAltName or commonName) matches the hostname that the client tried to connect to. If you don’t, you won’t know who you are talking to and your connection is subject to a man-in-the-middle attack. I was not able to convince Bill Janssen that this post connection check callback should be provided by the ssl module. I think there is value in doing a hostname check by default, and letting application developers override it if necessary. As it stands, you must code this check in the application code; you can get the peer certificate as a dictionary with getpeercert() method from the wrapped socket. Again, M2Crypto does this hostname check by default, but it can be overridden by the application if necessary. You might want to take a look at the M2Crypto checker code, although I am not convinced it is 100% correct per specifications (but it should err on the side of safety).

Below is a comparison of Python 2.6 and M2Crypto 0.19.1 clients.

Python 2.6 client:

from socket import socket
import ssl
s = socket()
c = ssl.wrap_socket(s, cert_reqs=ssl.CERT_REQUIRED,
                    ssl_version=ssl.PROTOCOL_SSLv3, ca_certs='ca.pem')
c.connect(('www.google.com', 443))
# naive and incomplete check to see if cert matches host
cert = c.getpeercert()
if not cert or ('commonName', u'www.google.com') not in cert['subject'][4]:
    raise Exception('Danger!')
c.write('GET / \n')
c.close()

M2Crypto 0.19.1 client:

from M2Crypto import SSL
ctx = SSL.Context('sslv3')
# If you comment out the next 2 lines, the connection won't be secure
ctx.set_verify(SSL.verify_peer | SSL.verify_fail_if_no_peer_cert, depth=9)
if ctx.load_verify_locations('ca.pem') != 1: raise Exception('No CA certs')
c = SSL.Connection(ctx)
c.connect(('www.google.com', 443)) # automatically checks cert matches host
c.send('GET / \n')
c.close()

Certificate Revocations

Another aspect that affects especially clients, but also servers that have been set up to require peer certificates, is the issue of certificate revocation. The two standard ways of doing this are the Certificate Revocation Lists (CRL) and Online Certificate Status Protocol (OCSP), but the ssl module in Python 2.6 provides no support for these. Granted, they are not easy to deal with in any of the 3rd party libraries either, but at least they provide some APIs to deal with them. Python is not unique here, though, because many (most?) modern applications still do not do any revocation checks by default (if at all).

Other Libraries

Besides the stdlib’s ssl module and M2Crypto, there are other SSL options for the Python developers out there. You might want to take a look at pyOpenSSL, which is again being maintained. Or if you want a pure Python version which can use various crypto libraries for performance improvements, you might want to check out TLS Lite (although it hasn’t been updated in a while). An exiting new package you may want to pay attention to is python-nss, which is a Python wrapper for the Network Security Services (NSS) libraries from the Mozilla project. If there are other maintained Python SSL packages out there I would be interested in hearing about them.

Update: In my opinion the essential resource for anyone working with OpenSSL-based software is Network Security with OpenSSL, by Viega et al., ISBN 059600270X.

M2Crypto 0.19 shipped with some code that expected OpenSSL with EC support. But since EC can be disabled, and at least Fedora Core ships with that disabled, things didn’t work so well in that case. Thanks to Miloslav Trmac who turned in a quick fix, we now have M2Crypto 0.19.1 that fixes this issue. No other major issues have surfaced in the week since 0.19 release, and since there have been at least 153 downloads according to PyPI, things look pretty stable.

Incidentally, if you were able to build 0.19, you should not need to get 0.19.1.