Thursday, August 20, 2009

Google App Engine - Cookie Handling with URL Fetch

I started working on creating a web based solution (on Google App Engine, using the Python API of course) to send mass SMS messages through your Google Voice account this week, but ran into a couple of problems right off the bat during some initial testing. The problem was that I could not log into my Google Account - the response always included a message that my "browser" didn't have cookies enabled. I was confused by this since I was using the exact method and code that I use in my Google Voice Command Line Script, which works perfectly:

self.opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())

Doing this will allow you to use the newly created "opener" object to open any URLs, and any Cookie data that is sent from the server is saved and resubmitted in the headers of each additional request. So, if you are visiting a site that requires authorization, you can send your credentials to the login page, and each subsequent request made with that opener will contain the Cookie info. This lets you access the protected areas/privileges of the site (such as sending SMS messages or making calls with your GV account) without much effort at all - it really is a nice feature of the language.

Well, I set this up in my app, but no luck. So I looked around for a bit and found out that the urllib2, urllib and httplib libraries perform requests using Google's URL fetch service (read more here). This isn't terrible, but the important thing is, is that the "urlfetch" service does NOT handle Cookies, even if you use a HTTPCookieProcessor, or a CookieJar (read more about that here).

This entry isn't a lesson on what Cookie's really are and how they work, but you should know that Cookie information is sent back from the server, and sent to the server in the headers portion of the request/response. Although the urlfetch service does not handle Cookie automatically, it does give you full access to process the header information received from a server, and also what information to send in the headers when making a request to the server. So, I built a separate class that I could use to open up URL's that would handle all Cookie information for me, as well as handle any redirects. The Google Account login system uses redirects when logging in - they forward you to a bunch of different sites to make sure you are actually logged in and that you have the right Cookie info.

Here is the class:

import urllib, urllib2, Cookie
from google.appengine.api import urlfetch

class URLOpener:
def __init__(self):
self.cookie = Cookie.SimpleCookie()

def open(self, url, data = None):
if data is None:
method = urlfetch.GET
method = urlfetch.POST

while url is not None:
response = urlfetch.fetch(url=url,
data = None # Next request will be a get, so no need to send the data again.
method = urlfetch.GET
self.cookie.load(response.headers.get('set-cookie', '')) # Load the cookies from the response
url = response.headers.get('location')

return response

def _getHeaders(self, cookie):
headers = {
'Host' : '',
'User-Agent' : 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv: Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729)',
'Cookie' : self._makeCookieHeader(cookie)
return headers

def _makeCookieHeader(self, cookie):
cookieHeader = ""
for value in cookie.values():
cookieHeader += "%s=%s; " % (value.key, value.value)
return cookieHeader

The class is simple, but works for my purposes. The class really has only one method that you should worry about: "open". Lets walk through that method: First, it checks to see if you are posting any data, and if not it makes it a GET request. Then, it starts into it's main loop. The first loop sends the data (if there is any) and some basic header information in the request, then saves the Cookie info it received in the response, then changes the request method to GET. It checks the headers for the "location" value to see if it needs to be redirected - if it does, it keeps going, saving and sending the received Cookie information along the way. Once it is done, it returns the response of the final location. Once you return the response, you can access the content by calling ".content" on the returned value.

Currently, it doesn't support GET requests with data, but mainly because I didn't need that for this project. It would be trivial to implement, however.

Here is an example on how to log in to your Google Voice account, then parse out the ever-so-important "_rnr_se" value using the URLOpener class:

opener = URLOpener()

loginParams = urllib.urlencode({
'Email' : email,
'Passwd' : password,
'continue' : ''
}) '', loginParams)


match ='name="_rnr_se".*?value="(.*?)"', googleVoiceHomePage)

_rnr_se =

Hope this helps!

Monday, August 17, 2009

Python - Boggle Solver

I have been a long time fan of the game Boggle. Last year in one of my CS classes we were required to create a program in Java that would find all the words on any given Boggle board - a very fun project, and one that I found useful for a number of reasons. The other day I was messing around with my old Java implementation, and thought it would be good practice to convert it to Python. During the process, I learned a few new tricks about Python and also some other solutions to problems that may come up in other places.

The first useful thing I learned how to do was to create a class that acted as if it had two __init__ functions, which of course Python cannot handle. There are a couple of ways to do it, but my favorite implementation uses the "@classmethod" decorator. The "@classmethod" decorator makes the method static (not like in Java or C++ - use @staticmethod for that) - so within the method you can initialize a new object, set the instance variables then return the newly created object once you are done. Here is a code snippet from my Word class where I used this:

class Word:
def __init__(self):
self.letter_sequence = ""
self.used_board_coordinates = set()

def new(cls, row, column):
word = cls()
word.used_board_coordinates.add((row, column))
return word

def new_from_word(cls, word):
new_word = cls()
new_word.letter_sequence += word.letter_sequence
return new_word

One of the constructors (called "new") accepts two parameters - the originating coordinates. It is called like so:

word =, column)

The second takes one parameter - another word. It creates a new word, and then copies over the data it needs. You call it much like the first:

wordCopy = Word.new_from_word(word)

Another problem I had to solve was prefix look-up. I have a large dictionary, and while searching for words on the board I only wanted to pursue directions that could potentially lead to actual words - so I needed to be able to quickly see if a letter sequence is a valid word prefix. I considered implementing a Trie or something similar, or using a modified Binary Search like I did in my Java program. I didn't really like either solution, so I came up with a solution that can take up a lot of memory, but runs extremely quickly - a simple set(). For each word loaded from my dictionary, I run a for loop that adds incrementing amounts of the word into my prefix set.

for index in xrange(len(word.strip()) + 1):

So, when I receive the word "python" from my dictionary, I add "p", "py", "pyt", "pyth"...etc to my prefix set. I was first afraid at how long it would take to do this, since the dictionary I have has 170,000+ words in it. It takes less than 1 second, and the look-up times are top-notch.

I also found a use for a generator - it could have been done without it, but I like how it cleans up the code. I use one in the BoggleSolver class - it helps me loop through all of the valid moves (coordinates) for a given word. The rules are this for Boggle - you can only use each letter (coordinate) once when constructing a word, and you can only move to adjacent spaces that are on the board. Here an example from my code:

for row, column in self._get_valid_coodinates_for_word(word, row, column):
if(self.dictionary.contains_prefix(word.letter_sequence + self.board[row][column])):
self._find_words(Word.new_from_word(word), row, column)

def _get_valid_coodinates_for_word(self, word, row, column):
for r in range(row - 1, row + 2):
for c in range(column - 1, column + 2):
if r >= 0 and r < self.board.side_length and c >= 0 and c < self.board.side_length:
if ((r, c) not in word.used_board_coordinates):
yield r, c

There are a lot of "for"s and "if" statements there, I know - but I thought it made it more readable, and it doesn't affect the performance. So, to get the coordinates for the word, simply pass the word and the current coordinates to the _get_valid_coordinates_for_word generator, and you can iterate over the returned row, column values. Pretty slick, if you ask me.

The actual problem that this script solves may not be too interesting to others, but some of the problems it solves within itself might, so I am posting the source. If you want to use it, it is ready to be used at the command line. Type "python" then a space separated list of the letters on your board. The board must be a square - 3x3, 4x4, 5x5...etc for it to word (it checks).


python a d e g n u p t e m l p w e f t

You can download the source here.
Be sure to grab the dictionary here (Place it in the same folder as the script).

Here is the source if you wish to just view it:

import sys

class BoggleSolver:
def __init__(self, letter_list):
self.dictionary = Dictionary("C:\\Dropbox\\Programming Projects\\Python\\PyBoggle\\dictionary.txt")
self.board = Board(letter_list)
self.min_length = 4
self.found_words = set()

# Find all words starting from each coordinate position
for row in xrange(self.board.side_length):
for column in xrange(self.board.side_length):
self._find_words(, column), row, column)

def _find_words(self, word, row, column):
word.add_letter(self.board[row][column], row, column)

if (self._can_add_word(word)):

for row, column in self._get_valid_coodinates_for_word(word, row, column):
if(self.dictionary.contains_prefix(word.letter_sequence + self.board[row][column])):
self._find_words(Word.new_from_word(word), row, column)

def _can_add_word(self, word):
return len(word) >= self.min_length and self.dictionary.contains_word(word.letter_sequence)

def _get_valid_coodinates_for_word(self, word, row, column):
for r in range(row - 1, row + 2):
for c in range(column - 1, column + 2):
if r >= 0 and r < self.board.side_length and c >= 0 and c < self.board.side_length:
if ((r, c) not in word.used_board_coordinates):
yield r, c

class Board:
def __init__(self, letter_list):
self.side_length = len(letter_list) ** .5
if (self.side_length != int(self.side_length)):
raise Exception("Board must have equal sides! (4x4, 5x5...)")
self.side_length = int(self.side_length)

self.board = []

index = 0
for row in xrange(self.side_length):
for column in xrange(self.side_length):
index += 1

def __getitem__(self, row):
return self.board[row]

class Word:
def __init__(self):
self.letter_sequence = ""
self.used_board_coordinates = set()

def new(cls, row, column):
word = cls()
word.used_board_coordinates.add((row, column))
return word

def new_from_word(cls, word):
new_word = cls()
new_word.letter_sequence += word.letter_sequence
return new_word

def add_letter(self, letter, row, column):
self.letter_sequence += letter
self.used_board_coordinates.add((row, column))

def __str__(self):
return self.letter_sequence

def __len__(self):
return len(self.letter_sequence)

class Dictionary:
def __init__(self, dictionary_file):
self.words = set()
self.prefixes = set()
word_file = open(dictionary_file, "r")

for word in word_file.readlines():
for index in xrange(len(word.strip()) + 1):

def contains_word(self, word):
return word in self.words

def contains_prefix(self, prefix):
return prefix in self.prefixes

if __name__ == "__main__":
boggleSolver = BoggleSolver(sys.argv[1:])
words = boggleSolver.found_words
print words


Saturday, August 8, 2009

Python + Google Voice. Mass SMS and Iterative Calling at the Command Line

EDIT (Jan 2011): Google changed the way contacts are downloaded. Many of the scripts below will break. Use the updated version here.

EDIT (Oct 3 2009): Google changed a few things on their login page which broke my old scripts. I have updated the scripts in this post, including those that you can download. If you have been receiving "Could not log in with provided credentials" errors, these new scripts should fix that problem.

You can download a ready-to-use-no-Python-installation-required Windows executable version of the program
here. Just download, unzip, find the GVMassContact.exe file and double click to use. This version's source is identical to those of scripts located below.

Click here to navigate to the downloadable Python scripts.

I have already written twice about Google Voice (here and here), but the scripts in this installment are an improvement over both of the scripts provided the other posts.

I wanted to create an interactive command line script that would allow me to either send an SMS message or call everyone in one of my Google Contacts Groups. Google Voice does not yet let you send mass text messages, or call people in an iterative fashion, so I came up with a script to let me do just that.

Instead of using Googles gdata Python API to access my Google Contacts, I decided to use Pythons "csv" module to parse an exported csv file downloaded from my GMail account. Doing this lets you run the script without needing to download any other libraries, assuming you are using a version of Python that has all the features used in this script (I use 2.6.2 currently).

The first thing I did was create a "" module with several helpful classes. These allow you to:
  1. Log in
  2. Gather all Google Contacts into separate groups
  3. Selectively narrow down the contacts in a group
  4. Gather the phone numbers that you have entered
  5. Send SMS messages
  6. Place Calls
I am by no means a seasoned Python developer, but what I have created works well for my purposes. I tried to make the classes as loosely coupled with the UI as possible in case I want to put a GUI around it sometime, but for now I am doing everything at the command line (which I often prefer).

The GoogleVoiceLogin class will allow you to log in, get the "opener" which lets you keep your log in cookie data for subsequent requests, and gives you access to the _rnr_se value (the "key") that is used when sending SMS messages or placing a call. Most of the other classes in the "gvoice" module will accept the opener and key as parameters to their constructors.

I created a sample script that uses the classes in my "gvoice" module to:
  1. Prompt the user for his/her Google Account credentials
  2. Allow the user to select a Group from his/her Google Contacts Groups.
  3. Allow the user to narrow down the contact list of the Group (nothing permanent is done to the Contacts)
  4. Choose whether to send an SMS message or call everyone in the list
  5. If calling, it allows you to choose a number from your phone list, or enter a custom number (like the Google Voice site does when placing calls)
The script that I wrote assumes I always want to use the contact's mobile phone (the first one) to send SMS messages and place calls, but that is easy to change.

There are two files needed for this to work - here is the main driver program (

from gvoice import *
import getpass
import sys
import re
import os

# Function used to create a separator
def separator():
return '-' * 25

def get_numeric_input(prompt):
return int(raw_input(prompt))

# Function to clear the screen
def clear_screen():
if == "posix":
# *nix systems
elif in ("nt", "dos", "ce"):
# Windows

# Main method to be run
def main():
# Log in
print "Please enter your Google Account credentials"
email = raw_input("User name: ")
password = getpass.getpass("Password: ")

gv = GoogleVoiceLogin(email, password)
if not gv.logged_in:
print "Could not log in with provided credentials"
print "Login successful!"

# Use the ContactLoader to download Google Contacts
contact_loader = ContactLoader(gv.opener)

# Use the ContactSelector to select the group and
# final list of contacts to contact
contact_selector = ContactSelector(contact_loader.contacts_by_group_list)

group_list = contact_selector.get_group_list()
selected_group = None
while selected_group not in range(1, len(group_list)+1):
print "Your Google Groups"
print separator()
for group_item in group_list:
print "{0}: {1}".format(group_item[0], group_item[1])
print separator()
selected_group = get_numeric_input("Enter the index of the group to select: ")

# Now that a group is selected, narrow down the list of people in the group
removing = True
while removing:
print "Contact List"
print separator()
for contact_item in contact_selector.get_contacts_list():
print "{0}: {1}".format(contact_item[0], contact_item[1])
print separator()
input_list = raw_input("Enter a list of the indexes (coma, space or otherwise delimeted)\nof those contacts you do not wish to contact this session.\nPress enter when finished: ")
if input_list != '':
contacts_to_remove_list = [int( for match in re.finditer(r"(\d+)", input_list)]
removing = False

# Print final list
print "Final List:"
print separator()
for contact_item in contact_selector.get_contacts_list():
print "{0}".format(contact_item[1])
selected_option = None
while selected_option not in [1, 2]:
print separator()
print "Options: "
print "1: Send Text"
print "2: Call"
print separator()
selected_option = get_numeric_input("Select which action to take: ")

# Send texts to all people in contact list
if (selected_option == 1):
print separator()
text_sender = TextSender(gv.opener, gv.key)
text = raw_input("Enter text message. Press enter when finished: ")
text_sender.text = text
for contact in contact_selector.get_contacts_list():
number = contact[1].mobile
if number == '':
print "{0} does not have a mobile number".format(contact[1])
print "Sending message to {0} at {1}...".format(contact[1], contact[1].mobile),
if text_sender.response:
print "Success!"
print "Failed!!"

# Call all people in contact list
elif (selected_option == 2):
print separator()
number_dialer = NumberDialer(gv.opener, gv.key)

number_retriever = NumberRetriever(gv.opener)
phone_number_items = number_retriever.get_phone_numbers()

# Get the forwarding number
forwarding_number_input = None
while forwarding_number_input not in range(1, len(phone_number_items) + 2):
print "Select forwarding number"
print separator()
for phone_number_item in phone_number_items:
print "{0}: {1}".format(phone_number_item[0], phone_number_item[1][0])
print "{0}: {1}".format(len(phone_number_items) + 1, "Other")
print separator()
forwarding_number_input = get_numeric_input("Choose from your previously entered numbers, or select \"Other\": ")

if forwarding_number_input in range(1, len(phone_number_items) + 1):
forwarding_number = phone_number_items[forwarding_number_input - 1][1]
forwarding_number = ''
while not re.match(r"\(?\b[0-9]{3}\)?[-. ]?[0-9]{3}[-. ]?[0-9]{4}\b\Z", forwarding_number):
forwarding_number = raw_input("Enter the forwarding number to dial: ")
number_dialer.forwarding_number = forwarding_number

print separator()
# Loop through and make the calls
for contact in contact_selector.get_contacts_list():
number = contact[1].mobile
if number == '':
print "{0} does not have a mobile number".format(contact[1])
input = None
while input not in ['', 'n','N', 'q', 'Q'] :
input = raw_input("Press enter to call {0} at {1} ('n' to skip, 'q' to quit): ".format(contact[1], contact[1].mobile))
if input == '':
print "Calling {0}....".format(contact[1]),
if number_dialer.response:
print "Success!"
print "Failed!!"
elif input .upper() == 'N':
elif input.upper() == 'Q':
print "Call chain aborted."

if __name__ == "__main__":

And here is the other required script (must be named ""):

import csv
import sys
import re
import urllib
import urllib2

class GoogleVoiceLogin:
def __init__(self, email, password):
# Set up our opener
self.opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())

# Define URLs
self.loing_page_url = ''
self.authenticate_url = ''
self.gv_home_page_url = ''

# Load sign in page
login_page_contents =

# Find GALX value
galx_match_obj ='name="GALX"\s*value="([^"]+)"', login_page_contents, re.IGNORECASE)

galx_value = if is not None else ''

# Set up login credentials
login_params = urllib.urlencode( {
'Email' : email,
'Passwd' : password,
'continue' : '',
'GALX': galx_value

# Login, login_params)

# Open GV home page
gv_home_page_contents =

# Fine _rnr_se value
key ='name="_rnr_se".*?value="(.*?)"', gv_home_page_contents)

if not key:
self.logged_in = False
self.logged_in = True
self.key =

class ContactLoader():
def __init__(self, opener):
self.opener = opener
self.contacts_csv_url = ""
self.contacts_csv_url += "?groupToExport=^Mine&exportType=ALL&out=OUTLOOK_CSV"

# Load ALL Google Contacts into csv dictionary
self.contacts = csv.DictReader(

# Create dictionary to store contacts and groups in an easier format
self.contact_group = {}
# Assigned each person to a group that we can get at later
for row in self.contacts:
if row['First Name'] != '':
for category in row['Categories'].split(';'):
if category == '':
category = 'Ungrouped'
if category not in self.contact_group:
self.contact_group[category] = [Contact(row)]

# Load contacts into a list of tuples...
# [(1, ('group_name', [contact_list])), (2, ('group_name', [contact_list]))]
self.contacts_by_group_list = [(id + 1, group_contact_item)
for id, group_contact_item in enumerate(self.contact_group.items())]

class Contact():
def __init__(self,contact_detail):
self.first_name = contact_detail['First Name'].strip()
self.last_name = contact_detail['Last Name'].strip() = contact_detail['Mobile Phone'].strip() = contact_detail['E-mail Address'].strip()

def __str__(self):
return self.first_name + ' ' + self.last_name

# Class to assist in selected contacts by groups
class ContactSelector():
def __init__(self, contacts_by_group_list):
self.contacts_by_group_list = contacts_by_group_list
self.contact_list = None

def get_group_list(self):
return [(item[0], item[1][0]) for item in self.contacts_by_group_list]

def set_selected_group(self, group_id):
self.contact_list = self.contacts_by_group_list[group_id - 1][1][1]

# Return the contact list so far
def get_contacts_list(self):
return [(id + 1, contact) for id, contact in enumerate(self.contact_list)]

# Accept a list of indexes to remove from the current contact list
# Assumes 1 based list being passed in
def remove_from_contact_list(self, contacts_to_remove_list):
if self.contact_list == None:
for id in contacts_to_remove_list:
if id in range(0, len(self.contact_list)+1):
self.contact_list[id - 1] = None
self.contact_list = [contact for contact in self.contact_list if contact is not None]

class NumberRetriever():
def __init__(self, opener):
self.opener = opener
self.phone_numbers_url = ''
phone_numbers_page_content =

# Build list of all numbers and their aliases
self.phone_number_items = [(,
for match
in re.finditer('"name":"([^"]+)","phoneNumber":"([^"]+)"',

def get_phone_numbers(self):
return [(id + 1, (phone_number_item))
for id, phone_number_item
in enumerate(self.phone_number_items)]

class TextSender():
def __init__(self, opener, key):
self.opener = opener
self.key = key
self.sms_url = ''
self.text = ''

def send_text(self, phone_number):
sms_params = urllib.urlencode({
'_rnr_se': self.key,
'phoneNumber': phone_number,
'text': self.text
# Send the text, display status message
self.response =, sms_params).read()

class NumberDialer():
def __init__(self, opener, key):
self.opener = opener
self.key = key
self.call_url = ''
self.forwarding_number = None

def place_call(self, number):
call_params = urllib.urlencode({
'outgoingNumber' : number,
'forwarding_number' : self.forwarding_number,
'subscriberNumber' : 'undefined',
'remember' : '0',
'_rnr_se': self.key

# Send the text, display status message
self.response =, call_params).read()

For the "" script to work, you need to have the "" file located in the same folder.

If you would rather not copy and paste, you can download the files here:

If you don't care about having the files separated, and would rather have to deal with just one script that you can keep on your desktop, you can download this file:
Here is a new, ready-to-use-no-Python-installation-required Windows executable version. Just download, unzip, find GVMassContact.exe in the GVMassContact folder, double click and go.

The only difference between the "" script and the others is that all of the classes contained in the "" file are located at the top of the file.

If you download the "" script, you can keep it on your desktop and simply double click it to get the interactive command line session to come up.


Easier Python script execution in Widows

While this post does not contain a sample script as all previous posts do, it can make creating and executing scripts easier if you are working on a Windows machine. There are two things that I like to set up to accomplish this:

1) Make widows execute scripts at the command line without needing to type "python" before the name of the script, or the letters ".py" after the name of the script.
2) Put the folder that contains all of my commonly used scripts on the system Path.

To accomplish the first step, do this:
Control Panel -> System -> Advanced System Settings -> Environments Variables
In the new window that shows up, scroll down in the bottom list to find PATHEXT. Double click on it and add ";.PY" to the end.

Once you are done with that, double click on the "Path". Append ";[FULL PATH TO YOUR SCRIPT FOLDER]" to what is already there.


If you have a script in your folder that you just added to your Path called "" you can now simply type "helloWorld" and it will be run. It isn't that much better than "python", but I prefer this method.

Wednesday, August 5, 2009

Python - Drag and Drop Zip

I often need to quickly zip a folder or file on my desktop to email off to someone. I have 7-zip installed, and it is a great program, but wanted to come up with something even easier to use as well as try something new.

I created a Python script that sits on my desktop called "Zip". To create a zip archive of a file or folder, I simply drag and drop the file onto the "Zip" script and a new .zip file will show up wherever the script is located. It works just fine, and does not require any non-standard Python modules. Tested on on Windows, Python 2.6.2.

Here is the script source:

import os, sys
import zipfile

# Make sure a folder or directory was specified
if len(sys.argv) == 1:

# Assign name of file or folder to resusable variable
resource = sys.argv[1]

# Create name of new zip file, based on original folder or file name
zipFileName = os.path.splitext(resource)[0] + '.zip'

# Create zip file
zipFile = zipfile.ZipFile(zipFileName, "w")

# Function to create the archive name
# Otherwise the zip folder contains many, unnecessary sub folders
def getArchiveName(resource, root, file):
if root == resource:
return (os.sep).join([resource, file])
return (os.sep).join([resource, root, file])

# Write file(s) to the zipFile
print "Creating zip archive..."
if os.path.isdir(resource):
for root, dirs, files in os.walk(resource):
for file in files:
zipFile.write((os.sep).join([root, file]),
getArchiveName(os.path.basename(resource), os.path.basename(root), file),
zipFile.write(resource, os.path.basename(resource), zipfile.ZIP_DEFLATED)

# Close file when finished

You can also download the script here:

The unzip version is coming soon.