Python - Opening URLs in a webbrowser

The other day at work I needed to test a web page that I had created. The page was a form that would allow certain logged in users to suggest changes to profile information that we have on a group of people. The form data was pulled from a database, and the page knew whose information to pull based on a URL parameter called "mem" - which contained the numeric primary key of the directory member's record in the database. There were several places where the page could crash, based on what was coming our of the database, as the original creator didn't conform to a standard way of representing a "NULL" value. Some fields held NULL, some 0, some an empty string - and some columns could be any of those 3, but all meant the same thing. To test the page, I would simply change the "mem" parameter to pull up someone else's data - this would help me find any other bugs that I may have overlooked. The page would crash, but in our testing environment the error message (detailing the error and the line number where it occurred) is embedded within the page. I wanted to be completely thorough with this testing, but there are several hundred members of this directory, and trying every single possibility by hand would have taken a VERY long time.

Setting up a proper testing environment with unit tests and everything would be ideal - but where I work that sadly isn't standard practice, and we don't have a good way of doing that. So, I improvised by writing a quick script in Python to test every single possibility for me. First, I grabbed a list of all the primary keys of the members in the database, and then set up a URL opener using a HTTPCookieProcessor, so I could get into the protected page. I looped through the list, assigning the "mem" parameter to the primary keys in the list, and checked for a HTTPError, which would happen only if the page crashed (HTTPError is thrown when there is a 501 Server Error).

Since there was a server error, the URL opener wouldn't let me download the page to process the error message - so I used the Python webbrowser module to simply open the page up in Firefox for me - that way I could look at the page myself and examine the error. Since the page I was testing is in a protected area of the site, I needed to first open up Firefox and log in - Python can't pass cookie data to Firefox. After that, all I needed to do was run the script. All the pages that had problems would open as new tabs in my currently running Firefox instance.

Here is the script (obviously changed since I can't be giving out login credentials):

import urllib2, urllib
import webbrowser
testMemberIds = []

opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())
urllib2.install_opener(opener)

loginParams = urllib.urlencode( {
'login' : '',
'password' : '',
} )
opener.open( 'http://',  loginParams)

for memNum in testMemberIds:
   print "Trying {0}...".format(memNum),
   try:
       opener.open("?mem={0}".format(memNum))
   except urllib2.HTTPError, e:
       print "Server error! - Member: {0}".format(memNum)
       webbrowser.open("?mem={0}".format(memNum))

This is a pretty hackish way of testing a site, but it got the job done for me. It would have taken hours to test every possible primary key in the list by hand, but this script only took a a few minutes to throw together, and I found all the errors that could possibly come up (with the data currently in the database that is) and fixed - all in less than 20 minutes.

This is the first time I used the webbrowser module to do anything truly helpful to me, but it got me thinking about some other uses for it - possibly opening up separate tabs for items that match certain criteria on eBay or CraigsList. Just a thought - might be something to play with later on.