California Dreams

Android Market handles sales through Google Checkout. I haven’t tried selling anything else online before, but what this setup provides for me as the seller leaves a lot to be desired. One issue you will have trouble with is getting the data needed to file taxes.

Google provides a Google Checkout Notification History API that lets you programmatically query sales data. For my purposes the API requests are really simple: just post a small XML document with the date range I am interested in, get back XML documents that contain my data. If there is more data that fits in a single response, look for an element that specifies the token for the next page and keep pulling until you get all data.

Below is a really simple Python script that uses M2Crypto to handle the SSL parts for the connection (needed since Python doesn’t do secure SSL out of the box). You will also need to grab certificates. You should save the script as gnotif.py, save the certificates as cacert.pem and create gnotif.ini as described in the script below all in the same directory. When you execute it, it will ask for start and end date (in YYYY-MM-DD format) and then fetch all the data, saving them in response-N.xml files, where N is a number.

#!/usr/bin/env python
# Script to query Google Checkout Notification History
# http://code.google.com/apis/checkout/developer/Google_Checkout_XML_API_Notification_History_API.html
 
# Supporting file gnotif.ini:
#[gnotif]
# merchant_id = YOUR_MERCHANT_ID_HERE
# merchant_key = YOUR_MERCHANT_KEY_HERE
 
import base64
import re
from ConfigParser import ConfigParser
 
from M2Crypto import SSL, httpslib
 
ENVIRONMENT = "https://checkout.google.com/api/checkout/v2/reports/Merchant/"
XML = """\
<notification-history-request xmlns="http://checkout.google.com/schema/2">
%(query)s
</notification-history-request>
"""
 
config = ConfigParser()
config.read('gnotif.ini')
MERCHANT_ID = config.get('gnotif', 'merchant_id')
MERCHANT_KEY = config.get('gnotif', 'merchant_key')
 
rawstr = r"""<next-page-token>(.*)</next-page-token>"""
compile_obj = re.compile(rawstr, re.MULTILINE)
 
auth = base64.encodestring('%s:%s' % (MERCHANT_ID, MERCHANT_KEY))[:-1]
 
ctx = SSL.Context('sslv3')
# If you comment out the next 2 lines, the connection won't be secure
ctx.set_verify(SSL.verify_peer | SSL.verify_fail_if_no_peer_cert, depth=9)
if ctx.load_verify_locations('cacert.pem') != 1: raise Exception('No CA certs')
 
start = raw_input('Start date: ')
end = raw_input('End date: ')
 
data = XML % {'query': """<start-time>%(start)s</start-time>
<end-time>%(end)s</end-time>""" % {'start': start, 'end': end}}
 
i = 0
 
while True:
    c = httpslib.HTTPSConnection(host='checkout.google.com', port=443, ssl_context=ctx)
    c.request('POST', ENVIRONMENT + MERCHANT_ID, data,
             {'content-type': 'application/xml; charset=UTF-8',
              'accept': 'application/xml; charset=UTF-8',
              'authorization': 'Basic ' + auth})
 
    r = c.getresponse()
 
    f=open('response-%d.xml' % i, 'w')
    result = r.read()
    f.write(result)
    f.close()
 
    print i, r.status
 
    c.close()
 
    match_obj = compile_obj.search(result)
    if match_obj:
        i += 1
        data = XML % {'query': """<next-page-token>%s</next-page-token>""" % match_obj.group(1)}
    else:
        break

As you take a look at the data you will probably notice that you are only getting the sale price information, but no information about the fees that Google is deducting. Officially it is a flat 30%, but I have found out a number of my sales have the fee as 5%. So we need to get this information somehow. Luckily you can toggle a checkbox in your Google Checkout Merchant Settings. Unfortunately there is a bug, and the transaction fee shows as $0 for Android Market sales. I have reported this to Google, and they acknowledged it, but there is no ETA on when this will be fixed.

I also haven’t found any way to programmatically query when and how much did Google Checkout actually pay me. (I can get this info from my bank, but it would be nice to query for that with the Checkout API as well.)

Last but certainly not least, working with the monster XML files returned from Google Checkout API is a real pain. If someone has a script to turn those into a format that could be imported into a spreadsheet or database that would be nice…

Many companies run services in the Amazon cloud infrastructure, so it makes an attractive target for criminals as well. You need to make sure that you really are talking to the right Amazon servers when you use the cloud services.

boto seems to have emerged as the winner in the scramble to develop Python libraries to deal with Amazon Web Services (AWS). By default, boto will use the stdlib httplib.HTTPSConnection. This is a problem, because the stdlib does not provide secure SSL out of the box. However, boto designers have made it easy to plug in alternative SSL implementations that conform to the httplib.HTTPSConnection interface. M2Crypto provides this in httpslib.HTTPSConnection.

The first step is to get CA certificates that we can use to verify that the Amazon servers we will be talking to have valid certificates issued by trusted certificate authorities.

Amazon offers various services that boto provides access to, so the exact details vary a little bit (namely what connection class to instantiate). I’ll use the SimpleDB as an example, because the first 25 machine hours per month are free so it makes a great test system (you still need to sign up for AWS and provide credit card information).

#!/usr/bin/env python
 
import sys
 
from M2Crypto import httpslib, SSL
from boto.sdb.connection import SDBConnection
 
def https_connection_factory(host, port=None, strict=0, **ssl):
    """HTTPS connection factory that creates secure connections
    using M2Crypto."""
    ctx = SSL.Context('tlsv1')
    ctx.set_verify(SSL.verify_peer | SSL.verify_fail_if_no_peer_cert, depth=9)
    if ctx.load_verify_locations('cacert.pem') != 1:
        raise Exception('No CA certs')
    return httpslib.HTTPSConnection(host, port=port, strict=strict,
                                    ssl_context=ctx)
 
def create_connection(aws_access_key_id, aws_secret_access_key):
    """Create SimpleDB connection."""
    conn = SDBConnection(aws_access_key_id=aws_access_key_id,
                         aws_secret_access_key=aws_secret_access_key,
                         https_connection_factory=(https_connection_factory, ()))
    return conn
 
if __name__ == '__main__':
    # Sample usage
    if len(sys.argv) != 3:
        sys.exit('Usage: %s aws_access_key_id aws_secret_access_key' % sys.argv[0])
 
    conn = create_connection(*sys.argv[1:])
    domain = conn.create_domain('mytest')
    try:
        item, key, value = 'item1', 'key1', 'value1'    
        domain.put_attributes(item, {key: value})
        assert value == domain.get_attributes(item)[key]
    finally:
        conn.delete_domain(domain)
 
    print 'Usage:', conn.get_usage()

The sample application takes your AWS access key and secret access key as parameters, and it assumes cacert.pem file containing the CA certificates is in the same directory. Typically running that application shows that it uses less than 0.006 secondshours of Amazon computing facilities so you could run this application over 15 million4500 times a month without charge.

Update:I mixed up units, which Mocky pointed out; fixed above.

M2Crypto has been claiming support for OpenSSL 0.9.7 but it actually turned out I wasn’t testing with quite that old OpenSSL version. Recently M2Crypto got support for RSA PSS stuff, but it turns out this was added in OpenSSL 0.9.7h, and you could not build/run M2Crypto against an older OpenSSL version. Arguably you should not use those old OpenSSL versions, but apparently there are people who can’t help it. And since M2Crypto claims support all the way back to 0.9.7 it made sense to make it so.

The M2Crypto trunk and 0.20.2 now omit the RSA PSS stuff if you have too old OpenSSL. Additionally, to prevent this kind of error from happening in the future, I added “minreq” (for Minimum Requirements) Tinderbox client that builds and tests M2Crypto trunk using Python 2.3, OpenSSL 0.9.7 and SWIG 1.3.28 (the current minimum requirements) on Ubuntu 8.04.

Miloslav Trmac noticed a regression in httpslib.ProxyHTTPSConnection and provided a fix for it, and I’ve just tagged and uploaded the new 0.20.1 version to PyPI. See the 0.20 release announcement for general information about the 0.20 release series.

I’ve also been getting a few requests for help on building M2Crypto on Fedora Core -based systems, so I’ve added some info to the FAQ. Basically there is a fedora_setup.sh wrapper script in the source tarball that you can use instead of the plain setup.py. There are more details about the fedora_setup.sh script, if you are interested.

I am pleased to announce M2Crypto 0.20, which has been in development for over nine months. It fixes over 30 bugs, and fixes and new functionality was contributed by more than ten people. Hooah!

The CHANGES file lists the full list of changes, but personally I am most pleased by M2Crypto having reached the magical 80% unit test (code) coverage. Besides that technicality, there are Python 2.6 fixes, threading fixes, added support for RSASSA-PSS signing and verifying, certificates with large serial numbers, and more.

Download from pypi.

Or use easy_install (may not work on all systems): easy_install M2Crypto

M2Crypto is the most complete Python wrapper for OpenSSL featuring RSA, DSA, DH, HMACs, message digests, symmetric ciphers (including AES); SSL functionality to implement clients and servers; HTTPS extensions to Python’s httplib, urllib, and xmlrpclib; unforgeable HMAC’ing AuthCookies for web session management; FTP/TLS client and server; S/MIME; ZServerSSL: A HTTPS server for Zope and ZSmime: An S/MIME messenger for Zope. M2Crypto can also be used to provide SSL for Twisted. Smartcards supported through the Engine API.

Better late than never… I am announcing the first beta of M2Crypto 0.20 release. M2Crypto is the most complete Python wrapper for OpenSSL.

The 0.20 release has been in development for about nine months. About 30 bugs and new features have been implemented by more than ten people. Unit tests now cover 80% of the code base. Tinderbox is used to automatically test changes on various flavors of Ubuntu, Fedora Core, Redhat and Cygwin. We could use more Tinderbox clients, so please drop me a line if you have some spare machine cycles available.

The release include some fairly significant changes, including tricky ones in threading and so forth. See the CHANGES file for list of changes. Please test your applications, and go file bugs on any issues you notice. I’ll wait for feedback for a week, spin the next beta and so forth until there are no more release blockers found within a beta period.

There are a few issues I feel bad about that did not make the first beta. If you can help create fixes for these, I’d be willing to consider including the fixes in 0.20 if the changes don’t look too scary. Here is my wish list:

• Should work with OpenSSL 1.0.0
• A regression dealing with RSA keys
• Problems with Unicode URLs
• verify_cert not implemented
• CRL bindings

Download from pypi.

Or use easy_install (may not work on all systems): easy_install M2Crypto

Ever since M2Crypto got support for Elliptic Curves (EC) cryptography, it has been somewhat difficult to build M2Crypto on systems where OpenSSL has been built without EC support. Notably distributions based on Fedora Core, which besides Fedora Core include of course Redhat and CentOS.

Disabled EC support alone wouldn’t be an issue, since normally opensslconf.h defines OPENSSL_NO_EC, but those same systems have also changed OpenSSL build so that instead of opensslconf.h you need to include processor architecture dependent file. And if that wasn’t enough this isn’t actually what you are supposed to be doing, and you will hit a compiler error to notify you of that. The final step in the recipe is to tell SWIG to treat errors as warnings. The distributions make build changes in their own versions of M2Crypto, but unless you know the recipe, you are going to have a hard time building M2Crypto yourself. Miloslav Trmač showed me what kind of changes were needed, and I made a new setup wrapper fedora_setup.sh which should help you get off the ground with M2Crypto if you are having a hard time building it. Let me know if you run into any problems with the setup wrapper.

I am sure the distros did not make these changes just to make it harder to build systems based on OpenSSL, although it sure does feel like that at times. Maybe someone can shed light on why these changes were made in the first place.

I have been working on an installer system that needs to ask the user a bunch of questions, including passwords, and store the answers in an .ini so that if the user runs the installer again, they will get prefilled answers. After the first installation it should be possible to run the installer in a batch mode since all the answers were recorded. But being security conscious, I did not like all those passwords being stored in plain text.

Luckily for me, I had already implemented almost exactly what I needed in Chandler, namely the password manager feature of Chandler which encrypts account passwords with a master password. I just needed to rip out the Chandler-specific features to get a general purpose encryption and decryption library, which defaults to 256 bit AES (Rijndael) in CBC mode and derives a key from the master password using PBKDF2 algorithm. The encrypted data serializes to just text, which can appear in .ini file. The serialized form does not follow any standards. With a few extra lines of code it can also act as a utility to encrypt and decrypt files. Thus, m2secret was born. It is built using the M2Crypto library.

Here’s how you could use it:

import m2secret
 
# Encrypt
secret = m2secret.Secret()
secret.encrypt('my data', 'my master password')
serialized = secret.serialize()
 
# Decrypt
secret = m2secret.Secret()
secret.deserialize(serialized)
data = secret.decrypt('my master password')
 
assert data == 'my data'

The command line utility is called m2secret.

Installation should be as easy as easy_install m2secret.