California Dreams

I was testing a Python web application that was using FastCGI deployment on Dreamhost, when I found myself looking at a souped up Python Traceback in my browser. At first I couldn’t understand why that was happening. As far as I knew I was running with full production settings and as such I would have expected a terse internal server error message.

Looking at the HTML source of the error page I discovered reference to cgitb. But as far as my code was concerned I did not set that. I tried specifically disabling that in my script but that made no difference. In a momentary act of desperation I did a find for all cgitb.py files under my account and made the cgitb.enable() function do nothing. Yet I was still seeing the tracebacks.

After a bit of scratching my head and throwing different words at Google it occurred to me to take a look at the fcgi.py script. Oops. The [WSGI]Server class has an error() method whose docstring states that it “May and should be overridden”. No $%^, the default just plasters all the dirty little secrets for the world to see! I’d like to see something like Debug[WSGI]Server that pretty prints the error, and leave the [WSGI]Server the production class. The naming would make it clear that you should not be using the debug version in production. As it is now, I wonder how many people actually read all the way towards the bottom of the 1331 line file to discover this gem.

I also added a warning to the Dreamhost documentation regarding Python FastCGI.

I just pushed out the first beta for the M2Crypto 0.19 release. The plan is to release 0.19 as quickly as possible following Python 2.6 release.

The road to 0.19 has been surprisingly long, and I didn’t intend it that way. While I was taking a break after 0.18.2 release, I found out it was time to find a new job. With the job search, and later ramping up with the new job, there just wasn’t much time and energy left to put in M2Crypto. But I have settled in with the changes, and it is high time to roll out the bug fixes and new features in M2Crypto that many people have worked hard for.

In my opinion the 0.19 release highlights are as follows:

• Python 2.6 support
• Fixed SSL deadlocks caused by GIL handling changes done in 0.18
• Wrappers for OpenSSL ENGINE_* functions, which enable smart card usage
• Wrappers for OpenSSL OBJ_* functions, making it easier to deal with X.509 certificates
• Fixed crash that prevented encryption using public key from X.509
• Fixed several functions and methods that failed silently or with wrong errors
• Switched to writing private keys in more secure manner

You might want to take a look at the full change log as well.

I have done most of my development on a 64-bit Ubuntu Linux machine except for the last week or so since that machine died. Over the weekend and this week I have tested on 32 bit Ubuntu Linux and Cygwin. The Python versions I have covered are 2.4.x, 2.5.x and 2.6 release candidates. OpenSSL versions were late 0.9.8 series (0.9.8g or so). SWIG 1.3.33 or thereabouts. I would especially appreciate it if someone could test on Mac, and using native Windows Python. Also tests using 0.9.7 series OpenSSL and SWIG version < 1.3.30 would be a big help.

You can grab the sources from the M2Crypto homepage, or just do easy_install M2Crypto.

Extended Validation Certificates (EV) were invented to fix the mess of what became of SSL in the race to provide ever cheaper certificates. Historically browsers were displaying just a lock icon for any certificate that was issued by a trusted certificate authority (CA), and since there was no standard levels or verification, the least validated and cheapest (sometimes even free) domain validated certificates (DV) won a big chunk of the market. While DV certificates are fine for hobby forums and the like, you would really want more from a bank or an ecommerce site. The trouble is, you can’t easily tell if the certificate is DV or if more checks have been done. With EV there are standardized guidelines for the minimum level of checking that is needed, and browser vendors are on the board by displaying different UI for EV sites. The expectation is that EV certificates will give a high assurance that the the entity you think you are talking to really is that entity.

Dan Kaminsky’s recent DNS vulnerability find highlights the fact that DNS is still not secure. It is possible to spoof DNS and get a DV certificate for any domain, and then use another DNS spoof to redirect traffic to a site containing attack code. Now the problem arises when sites using EV certificates mix content from sites with DV certificates. There are some high profile sites doing this, by embedding Google Analytics scripts, or advertisements. So sites mixing DV content on EV page are actually vulnerable to DNS hacks still!

There is currently discussion going on in the Mozilla security forums on how to fix this. One way would be to state that an EV site can only load content from sites that are controlled by the same entity as the main EV site. After all, the idea with EV is that you can be really sure who you are dealing with, but if you have content coming from multiple sources it is no longer so clear. Another option could be to require EV site to load content only from other EV sites, regardless of who controls the other sites. You’d naturally need to tell the user who all the parties are they are talking to, but this will quickly result in a messy UI. And at least I would like to know which entity controls what part of the page I am looking at, but this would be a hard problem to solve with dynamic content.

Since I started experimenting with online advertisements myself, I started noticing a worrying trend of practices that are vulnerable to malicious activities. Security holes are of course not limited to online advertising software and services, but the very nature of it makes it potentially more harmful than run-of-the-mill security vulnerabilities.

First issue is of course that there is obviously money involved, which will get the criminal elements involved right away. The potential victims are often individuals hoping for quick and easy extra income using point and click solutions without the know-how to vet the security of the solutions. And at least in the United States, to be able to receive income, individuals will need to give their name, address, phone number, social security number and potentially other tax documentation to the entity through which they get the advertisements to publish on their sites. All that personal information just waiting to be captured and used for identity theft. It is also the user’s expectation that since all this personal data and money is involved, the software and service providers would be especially careful about security, but unfortunately this does not always seem to be the case. The users cannot be expected to be enforcing security either; the dancing pigs problem is well known.

In my experience the smaller and less experienced software and service providers are on the whole more likely to suffer from security vulnerabilities, but the big players are not immune either.

Here are three examples of issues I have noticed:

1. A marketing plugin for Wordpress provides two ways to fetch the ad information. The one they recommend uses PHP to fetch PHP code from their host over http and execute it without any checks. If an attacker can inject their own code, the attacker’s code can run with the web server’s permissions, reading login and password information from the Wordpress installation at a minimum. Depending on webserver configuration, it might also have access to other areas of the file system. This is especially worrying attack now after Kaminsky’s DNS vulnerability has been disclosed, which makes the code injection easy. The fix for this issue would be to provide PHP code that would only fetch ad parameters that would be validated before being used, and nothing received over the network would ever be executed. The connections could also be protected with SSL, which would be an even better guarantee that no bogus data was returned.
2. A Google AdSense competitor requires their users to login and submit all their personal information over unsecured connection!
3. Many Google AdSense account holders will use the same Google account to login to their Google Analytics pages. (In fact, they will probably use the same account for GMail and every other Google service that requires an account.) However, if you go to https://www.google.com/analytics, you will be redirected to the unsecured login page, giving malicious parties a way to get the users account information and access to victims AdSense information and other services. The workaround is to login through https://www.google.com/analytics/home. This issue has been discovered before by other people and reported to Google.

I have some recommendation for the online publishers to make it safer to operate in the advertisement business:

1. If possible, form a company. Company information is public anyway, so there is not much harm in divulging this to others. This isn’t such an appealing prospect for people starting in the publisher business because of the extra paperwork and in some cases significant amount of money and time that is required to start and run a company.
2. Make sure that you are giving and viewing personal information only through a secure connection. If you can’t see a link to a secure area, try changing http to https and hitting enter. In some cases you will be lucky and get the secure version of the page. If that does not work, email the service provider and ask if they have a secure version. Even if they don’t, this will give them incentive to work on implementing the secure version. You could also use other means of sending your information to the company, like calling or sending a letter by courier for example. But keep in mind that they might add your information on your behalf to the database that serves the data unsecured over the network. There is also quite a bit of competition in the ad space, so look at the competitors as well.
3. When given a choice of ad injection method, as a rule of thumb HTML markup is safer than doing anything on the server side, so opt for the HTML version (this would typically be a Javascript tag). If server side is the only option, you would do well to either check the code yourself or ask someone else to go through the code looking for security vulnerabilities. This is a big topic in itself, and there are several books written on the topic, but one easy thing to check for is the use of “eval” or “exec” which are available in most languages. Any use of such a thing is a warning sign. Basically the code should be checking all the data that it reads over the network as potentially dangerous and do sanity checks on it before using it. A sort of a corollary of this advice is to avoid using pugins to manage ads, but do this by hand by inserting the ad markup in the desired locations. Of course this is not possible if you maintain large sites.

I am thinking of releasing the next M2Crypto version quickly after the Python 2.6 release. This means I will not be making any major changes, and this will be a pretty small release compared to the other M2Crypto releases I have done. The next version will include a handful of important fixes, including some regression fixes and one 2.6 compatibility issue.

There is still a little time to get some minor changes in, so anyone who is on notice for unit tests and so on, this is your final wakeup call if you want your changes in the next version.

Even though it is well known and easy to understand why it is a bad idea to serve login pages unencrypted, many organizations still do so. They claim that all is well since the page actually submits to a secure URL. Of course the problem is that the insecure page can be spoofed, so there really are no guarantees at all about that secure submit URL. There is even an unprotected login hall of shame page listing some notable examples.

For quite a long time ago I actually stumbled into a funny feature present on many such unprotected login pages. If you just hit submit (or login, or whatever the button says), you will often be redirected to the SSL-protected version of the login page. The first one that comes to mind where this works is LinkedIn. The way I found this is that I knew I had configured Firefox to remember my login for a site, yet when I went back Firefox did not fill in the login information. Looking in the password manager I noticed the login information was saved, but for the SSL version. I just decided to hit submit without password to see what would happen, and was glad to notice I ended up on the SSL-version of the page where Firefox promptly filled in the login information.

I recently bought Effective C++, 3rd Edition by Scott Meyers, and have enjoyed reading it. I had read the previous edition of this book, as well as the More Effective C++, but things have changed considerably in the whole C++ landscape, which has changed many of the best practices described in the previous editions.

While reading Effective C++ I also realized that my C++ language knowledge is probably still higher than my Python language knowledge, even though I haven’t used C++ much in the past four years. Before I went to work for OSAF, I had somewhat irrationally thought that C++ would be the language that I would use primarily for the rest of my life. It is a relatively difficult language to master, and experts in the language can expect to command respectable salaries. So it is no wonder I made a big investment in learning C++ well. I started in the University of Jyvaskyla with C++ programming class, then read C++ Primer by Stanley Lippman, followed by the C++ Programming Language by Bjarne Stroustrup. When I joined Citec I needed to program with MFC, but unfortunately I don’t remember the title of the great book I read back then. I then followed with the Effective C++ series and Thinking in C++ (free ebook) by Bruce Eckel, which I didn’t actually find that useful after the Effective C++ series. Around this time I also found Design Patterns by the GoF, which I had originally heard of when taking object oriented programming class at the University of Kent at Canterbury. I also read Object-Oriented Analysis and Design by Grady Booch when at Kent. When I started on the Mozilla code I got my hands on Effective COM by Don Box.

After joining Netscape (R.I.P.) I got pretty interested in security, and quickly gobbled Building Secure Software by John Viega and Gary McGraw and Software Security by Gary McGraw. After joining OSAF I read Secure Programming Cookbook for C and C++ by Viega and Messier, Writing Secure Code by Howard and LeBlanc and 19 Deadly Sins of Software Security by Howard, LeBlanc and McGraw. There are probably other books I am forgetting, but all the aforementioned books give an extremely good understanding of the C++ language, its pitfalls and effective usage. There is quite a bit of repetition in the security books, but all of them offer something new and interesting, and some repetition is actually good for you…

When I compare this to my Python reading it is pretty pitiful. I started with Learning Python by Mark Lutz and David Ascher, then read Python in a Nutshell by Alex Martelli (which I didn’t like much) and then Python Cookbook. So why did I not read more Python books? When I first started at OSAF I had to quickly ramp up with the language, but I also had to ramp up with cryptography and refresh on some mathematics, so I got a little burned out reading work-related books. There also aren’t that many books on Python, compared to C++. And finally the language itself is simpler, meaning you don’t feel the need to continuously learn about the language just to be productive and proficient with it. I would really like to find a book dedicated to security problems concentrating on Python, as well as some equivalent to Effective C++ series for Python (the Python Cookbook comes closest).

But I think I have finally found the joy of reading about computer programming languages again, so I am starting to look at improving my Python knowledge as well. There seem to be reasonably good selection of books on web application development with Python, so I believe I will start there. If you have any recommendations on Python books, please leave a comment.

When it rains… Just got some reports that people were unable to build M2Crypto 0.18.1 on Red Hat Linux and Debian Linux (unstable) due to double typedef of Py_ssize_t. This turned out to be due to insufficient #if guard around the typedef. I fixed that to follow PEP 353 and it built fine on Debian, so I released 0.18.2.