Friday, October 9, 2009

Python - Custom Google Voice API - Installable Module

Google Voice is a great service, but it lacks a couple of important things:
  1. An official API from Google
  2. A way to send mass SMS messages
I have spend some time creating my own unofficial API that is, I hope, easy to use and extend. I am by no means a seasoned Python developer, but I know enough to get what I want done. Many of the posts on my blog that interact with Google Voice depend on a "gvoice.py" script that I wrote, which contains several helpful classes. These class allow you to perform some basic actions, such as:

  1. Logging in to your Google Voice Account
  2. Sending a text message
  3. Placing a phone call

The script also allows you to:
  1. Download your Google Contacts into memory (Used mainly to get contact information to call or send a message)
  2. Programmatically select from you Contact Groups which people to contact
  3. Retrieve the numbers you have set up to work with your Google Voice account

Currently the script has only been tested using Python 2.6 (Python 3 might work, but I am not sure).

You can download the module with installation instructions below. Or, just scroll down to the bottom of the post to see the source in it's entirety.


Here are some of the posts where I show examples on how to use this module:

And the complete source:


"""
gvoice.py

Created by: Scott Hillman

http://www.everydayscripting.blogspot.com

This module comes as is an with no warranty.
You are free to use, modify and distribute this
code however you wish, but I ask that if you post
it anywhere, you at least make reference to me and
my blog where you acquired it.
"""

import csv
import sys
import re
import urllib
import urllib2

class GoogleVoiceLogin:
"""
Class that attempts to log in the Google Voice using the provided
credentials.

If either no password or email is provided, the user will be
prompted for them.

Once instantiated, you can check to see the status of the log in
request by accessing the "logged_in" attribute

The primary usage of a GoogleVoiceLogin object is to be passed
in to other constructors, such as the TextSender, or NumberDialer
"""

def __init__(self, email = None, password = None):
"""
Given the email and password values, this method will attempt to log
in to Google Voice. The "response" attribute can be checked to
see if the login was a success or not.

If the login was successful, the "opener" and "key" attributes will
be available to use when creating other objects.

To use an this object with the other classes in this module, simply
pass it in to the constructor. (ie text_sender = TextSender(gv_login))
"""

if email is None:
email = raw_input("Please enter your Google Account username: ")
if password is None:
import getpass
password = getpass.getpass("Please enter your Google Account password: ")

# Set up our opener
self.opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())
urllib2.install_opener(self.opener)

# Define URLs
self.login_page_url = 'https://www.google.com/accounts/ServiceLogin'
self.authenticate_url = 'https://www.google.com/accounts/ServiceLoginAuth'
self.gv_home_page_url = 'https://www.google.com/voice/#inbox'

# Load sign in page
login_page_contents = self.opener.open(self.login_page_url).read()

# Find GALX value
galx_match_obj = re.search(r'name="GALX"\s*value="([^"]+)"', login_page_contents, re.IGNORECASE)

galx_value = galx_match_obj.group(1) if galx_match_obj.group(1) is not None else ''

# Set up login credentials
login_params = urllib.urlencode({
'Email' : email,
'Passwd' : password,
'continue' : 'https://www.google.com/voice/account/signin',
'GALX': galx_value
})

# Login
self.opener.open(self.authenticate_url, login_params)

# Open GV home page
gv_home_page_contents = self.opener.open(self.gv_home_page_url).read()

# Fine _rnr_se value
key = re.search('name="_rnr_se".*?value="(.*?)"', gv_home_page_contents)

if not key:
self.logged_in = False
else:
self.logged_in = True
self.key = key.group(1)

class ContactLoader():
"""
This class is used to download and organize a csv file
of Google Contacts.It is often used in conjunction with
the ContactSelector class.

Example:

contact_loader = ContactLoader(gv_login)
contact_selector = ContactSelector(contact_loader)
"""
def __init__(self, gv_login):
"""
Pass in a GoogleVoiceLogin object, and the persons Google Contacts
Will be downloaded and organized into a structure called
"contacts_by_group_list"
which is organized in form:

[(1, ('group_name', [contact_list])), (2, ('group_name', [contact_list]))]

Which allows for easy access to any group.
"""
self.opener = gv_login.opener
self.contacts_csv_url = "http://mail.google.com/mail/contacts/data/export"
self.contacts_csv_url += "?groupToExport=^Mine&exportType=ALL&out=OUTLOOK_CSV"

# Load ALL Google Contacts into csv dictionary
self.contacts = csv.DictReader(self.opener.open(self.contacts_csv_url))

# Create dictionary to store contacts and groups in an easier format
self.contact_group = {}
# Assigned each person to a group that we can get at later
for row in self.contacts:
if row['First Name'] != '':
for category in row['Categories'].split(';'):
if category == '':
category = 'Ungrouped'
if category not in self.contact_group:
self.contact_group[category] = [Contact(row)]
else:
self.contact_group[category].append(Contact(row))

# Load contacts into a list of tuples...
# [(1, ('group_name', [contact_list])), (2, ('group_name', [contact_list]))]
self.contacts_by_group_list = [(id + 1, group_contact_item)
for id, group_contact_item in enumerate(self.contact_group.items())]

class Contact():
"""
Simple class to contain information on each Google Contact person.

Only stores information on:
First Name
Last Name
Mobile Number
Email address
"""
def __init__(self, contact_detail):
"""
Extract data from the given contact_detail

The following attributes are available:

first_name
last_name
mobile
email
"""
self.first_name = contact_detail['First Name'].strip()
self.last_name = contact_detail['Last Name'].strip()
self.mobile = contact_detail['Mobile Phone'].strip()
self.email = contact_detail['E-mail Address'].strip()

def __str__(self):
return self.first_name + ' ' + self.last_name

# Class to assist in selected contacts by groups
class ContactSelector():
"""
Class with helps select contacts after using the ContactLoader
object to download them.

Provides methods to:
1) Display the list of groups (get_group_list())
2) Set the selected group to work with (set_selected_group(group_id))
3) Get the contacts from the working list (get_contacts_list())
4) Remove names from the working list(remove_from_contact_list(contacts_to_remove_list))
"""
def __init__(self, contact_loader):
"""
Initialize the object - a ContactLoader object is expected here
"""
self.contacts_by_group_list = contact_loader.contacts_by_group_list
self.contact_list = None

def get_group_list(self):
"""
Extract a list of all the groups.
List is in the form:
[(1, 'Group Name'), (2, 'Groups Name'), ...]
"""
return [(item[0], item[1][0]) for item in self.contacts_by_group_list]

def set_selected_group(self, group_id):
"""
Select the group to work with.

This method will make the working contact_list contain all the
contacts from the selected group.
"""
self.contact_list = self.contacts_by_group_list[group_id - 1][1][1]

# Return the contact list so far
def get_contacts_list(self):
"""
Return a list of all the contacts, and assign them each a number

List is in the form:
[(1, Contact), (2, Contact), ...]
"""
return [(id + 1, contact) for id, contact in enumerate(self.contact_list)]

def remove_from_contact_list(self, contacts_to_remove_list):
"""
Accepts a one based list of ids, indicating which contacts
to remove from the list.

List needs to be a list in ints:
[3, 6, 7]
"""
if self.contact_list is None:
return
for id in contacts_to_remove_list:
if id in range(0, len(self.contact_list) + 1):
self.contact_list[id - 1] = None
self.contact_list = [contact for contact in self.contact_list if contact is not None]

class NumberRetriever():
"""
Class that will allow you to retrieve all stored phone numbers and their aliases
"""

def __init__(self, gv_login):
"""
Pass in the GoogleVoiceLogin object, this class will then
download all the numbers and aliases of the persons GV Account
"""
self.opener = gv_login.opener
self.phone_numbers_url = 'https://www.google.com/voice/settings/tab/phones'
phone_numbers_page_content = self.opener.open(self.phone_numbers_url).read()

# Build list of all numbers and their aliases
self.phone_number_items = [(match.group(1), match.group(2))
for match
in re.finditer('"name":"([^"]+)","phoneNumber":"([^"]+)"',
phone_numbers_page_content)]

def get_phone_numbers(self):
"""
Return the list of phone numbers in the form:
[(1, number), (2, number)...]
"""
return [(id + 1, (phone_number_item))
for id, phone_number_item
in enumerate(self.phone_number_items)]

class TextSender():
"""
Class used to send text messages.

Example usage:

gv_login = GoogleVoiceLogin('username', 'password')
text_sender = TextSender(gv_login)
text_sender.text = "This is an example"
text_sender.send_text('555-555-5555')

if text_sender.response:
print "Success!"
else:
print "Fail!"
"""
def __init__(self, gv_login):
"""
Pass in a GoogleVoiceLogin object, set the text message
and then call send_text
"""
self.opener = gv_login.opener
self.key = gv_login.key
self.sms_url = 'https://www.google.com/voice/sms/send/'
self.text = ''

def send_text(self, phone_number):
"""
Sends a text message containing self.text to phone_number
"""
sms_params = urllib.urlencode({
'_rnr_se': self.key,
'phoneNumber': phone_number,
'text': self.text
})
# Send the text, display status message
self.response = "true" in self.opener.open(self.sms_url, sms_params).read()

class NumberDialer():
"""
Class used to make phone calls.

Example usage:

gv_login = GoogleVoiceLogin('username', 'password')
number_dialer = NumberDialer(gv_login)
number_dialer.forwarding_number = 'number-to-call-you-at'

number_dialer.place_call('number-to-call')

if number_dialer.response:
print "Success!"
else:
print "Fail!"
"""
def __init__(self, gv_login):
self.opener = gv_login.opener
self.key = gv_login.key
self.call_url = 'https://www.google.com/voice/call/connect/'
self.forwarding_number = None

def place_call(self, number):
"""
Pass in a GoogleVoiceLogin object, set the forwarding_number
and then call place_call('number-to-call')
"""
call_params = urllib.urlencode({
'outgoingNumber' : number,
'forwardingNumber' : self.forwarding_number,
'subscriberNumber' : 'undefined',
'remember' : '0',
'_rnr_se': self.key
})

# Send the text, display status message
self.response = self.opener.open(self.call_url, call_params).read()

Thursday, October 8, 2009

Python - Google Voice from the command line (CLI SMS and Calling)

I have written about Google Voice a few times already, but the script that gets the most attention is the one that allows you to mass text/call people in your Google Contacts Groups, which you can find here.

Many people do not need such a large script, since they simply want to be able to use something to send one text message at a time through Google Voice at the command line. Even though my scripts were written with mass contacting in mind, you can still use my gvoice module to easily contact just one person.

To use this script, you will need to download my gvoice module. There are instructions about installing this module here, as well as some documentation on it.

Once you have the gvoice module installed, creating a command line script to text someone is trivial:


import gvoice
import sys

if len(sys.argv) < 3:
print "You must provide both a number and a message!"
exit()

gv_login = gvoice.GoogleVoiceLogin('username', 'password')
text_sender = gvoice.TextSender(gv_login)

text_sender.text = sys.argv[2]
text_sender.send_text(sys.argv[1])

if text_sender.response:
print 'Success! Message sent!'
else:
print 'Failure! Message not sent!'


Notice in this script I am explicitly setting the username and password. If you leave out those parameters, you will be prompted for them at run time.

You can now call the script (which I am naming"smser.py") like so:

python smser.py "555-555-5555" "This was sent from the command line!"

If you want to make a call to someone, it is just as easy:


import gvoice
import sys

if len(sys.argv) < 3:
print "You must provide both a forwarding number and number to dial!"
exit()

gv_login = gvoice.GoogleVoiceLogin('username', 'password')
number_dialer = gvoice.NumberDialer(gv_login)
number_dialer.forwarding_number = sys.argv[2]
number_dialer.place_call(sys.argv[1])

if number_dialer.response:
print 'Success! You should hear the phone ringing shortly...'
else:
print 'Call failed!'


You would use this script (which I am naming "caller.py") much the same way:

python caller.py "number-to-dial" "number-to-call-you-at"


Just remember the order of the numbers there, and you should be good to go.

Why would this CLI be helpful to you if you have the nice GUI of Google Voice? I can think of at least two good reasons:

1) It is faster to do at the command line (If you know the numbers!)
2) You can automate the sending of a text message. One reader shared with me the idea of executing something at the command line that would take a long time, and chaining on the the end of it "smser.py 'my-number' 'Finished!'", so he could walk away from his computer and know when the command had finished up. (Thanks Shane!) You could also set up cron jobs to send out a message on a given day and time (Another friend used this to wish himself Happy Birthday)

If you are on a Windows machine, you can add the folder where you download these scripts to your PATH environment variable so you can access them anywhere. If you want to go one step further and make it so you don't even have to type in "python" before the file name or ".py" after the file name when executing a Python script, follow the instructions here.

If you are in Linux, you should be able to create a symbolic link in /usr/bin if you want to achieve the same thing.

As usual, here is a zip file containing the source code of the examples shown above, plus the examples shown in the respective files. I also threw in there the installable gvoice module and the standalone gvoice module with some brief instructions on how to install either.

gv_cli_examples.zip

Let me know if you have any questions or problems using these scripts.

Friday, October 2, 2009

Python - Fixes to the Google Login script

Just recently Google changed some things behind the scenes to their login page. More specifically, on the login page there is now a hidden form element called "GALX", the value of which must be passed along with the Username and Password when signing in - other wise, the sign in fails. This affected a few of my scripts, most notably my Google Voice Mass Contact script which a few visitors let me know about.

I modified the scripts that are contained in that post to work with the new system. So, if you have been having trouble with the mass contact script, go to that post and re-download the scripts.

Here is the complete login script, including how to get the _rnr_se value after logging in.


import urllib
import urllib2
import getpass
import re

email = raw_input("Enter your Google username: ")
password = getpass.getpass("Enter your password: ")

opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())
urllib2.install_opener(opener)

# Define URLs
loing_page_url = 'https://www.google.com/accounts/ServiceLogin'
authenticate_url = 'https://www.google.com/accounts/ServiceLoginAuth'
gv_home_page_url = 'https://www.google.com/voice/#inbox'

# Load sign in page
login_page_contents = opener.open(loing_page_url).read()

# Find GALX value
galx_match_obj = re.search(r'name="GALX"\s*value="([^"]+)"', login_page_contents, re.IGNORECASE)

galx_value = galx_match_obj.group(1) if galx_match_obj.group(1) is not None else ''

# Set up login credentials
login_params = urllib.urlencode( {
'Email' : email,
'Passwd' : password,
'continue' : 'https://www.google.com/voice/account/signin',
'GALX': galx_value
})

# Login
opener.open(authenticate_url, login_params)

# Open GV home page
gv_home_page_contents = opener.open(gv_home_page_url).read()

# Fine _rnr_se value
key = re.search('name="_rnr_se".*?value="(.*?)"', gv_home_page_contents)

if not key:
logged_in = False
print 'Failed!'
else:
logged_in = True
key = key.group(1)
print 'Success!'

Friday, September 18, 2009

Python - Back to Basics: Accessing Online Content

Many of the different scripts that I write in Python deal with accessing content online. I thought that it might be a good idea to go over some of the different methods Python provides, and give some examples of their every day uses.

Most of the functionality you may ever want to access online content can be found in one of two modules (and often you need to use both) : urllib and urllib2.

First situation: You want to download the content of some page, perhaps to scrape the page for information for later processing. You want the content to be stored in a single variable (a string) so that you can do some regular expression matching on it, or some other form of parsing. A few simple lines:


import urllib2

page_contents = urllib2.urlopen("http://www.google.com").read()

print page_contents


Notice how on the second line I have the ".read()" method call. If you leave this off, you will have just the socket file object, which you can later call ".read()" on to get the contents. I like to do it all in one line however. Once you do this, it is a regular string and you can do whatever you want to/with it.

Now, lets say that you don't want to just download the page contents - you want to download an actual file, like an image, video, Power Point Presentation, PDF or anything else. Also, just a few lines:


# Download the Python image from http://python.org
import urllib

# File will be called "Python-Logo.gif",and will be contained in the folder
# where the script was executed
urllib.urlretrieve("http://python.org/images/python-logo.gif", "Python-Logo.gif")


You can use the "urlretrive" method to download pretty much any file you want - you just need to know it's URL and what you want to call it when you download it. Keep in mind that if you don't specify anything but the name of the file, it will be downloaded to the folder where you run the script. You can put things like "C:\images\Python-Logo.gif" in there if you want it to go somewhere else.

Using what we have gone over in the first two example, we can write some useful scripts. W can get the page source, find all of the PDF files linked to on it, and download them all to a folder. I do this very thing for one of the classes that I am currently taking. Here is that script:


import urllib
import re
import os

# Specifiy where I want to download my files
download_folder = "C:\DropBox\My Dropbox\School\CS479\Slides\\"

# Download page contents
# Note: I am using urllib, not urllib2, but only because urlretrieve does not exist in urllib2
# Both have this method, and do pretty much the same thing (for this kind of usage, that is)
page_contents = urllib.urlopen("https://cswiki.cs.byu.edu/cs479/index.php/Lecture_slides").read()

# Use regular expression to find all pdf files on site.
# match.group(1) will contain the link to the file
# match.group(2) will contain the name of the file
for match in re.finditer(r'<a href="(.*?\.pdf)"[^>]+>([^<]+)', page_contents):
file_url = match.group(1)
# Remove any characters that files cannot contain
file_name = re.sub(r'(\\|/|:|\*|\?|"|<|>|\|)', "", match.group(2))
# Check and see if I have already downloaded the file or not
if not os.path.exists(download_folder + file_name + '.pdf'):
print "Downloading new file {0}...".format(file_name)
urllib.urlretrieve(file_url, download_folder + file_name + '.pdf')


Lets make things more interesting - lets say that you want to get the content of a page, but you must be logged in to get at it. You can handle this just fine as well. Lets say that you want to get into your Google Voice account and scrape the page for the ever so important "_rnr_se" value. (I know that I have shown this a few times before, but many people still wonder how to do this, and it is a good, practical example).

Here are the steps we need to do to make this happen:
  1. Create an "opener" with the urllib2 module, through which all of our requests will be made. The opener will be created with a HTTPCookieProcessor that will handle all the Cookies from request to request (This allows us to stay "logged in").
  2. Install the opener, so whenever we make a request, our opener that we created will be used (and any Cookie data that has been received by previous requests will be sent along and updated when necessary)
  3. Prepare our login credentials, and URL encode them.
  4. Post them to the login page
  5. Do whatever we need once we are logged in.
This might seem like a lot, but it really isn't and it is very simple to do. Here is the script for that:



import urllib, urllib2
import re
from getpass import getpass

email = raw_input("Enter your Gmail username: ")
password = getpass("Enter your password: ")

opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())
urllib2.install_opener(opener)

# Set up login credentials for Google Accounts
# The 'continue' param redirects us to the Google Voice
# homepage, and gives us necessary cookie info
login_params = urllib.urlencode( {
'Email' : email,
'Passwd' : password,
'continue' : 'https://www.google.com/voice/account/signin'
})

# Perform the login. Cookie info sent back will be saved, so we remain logged in
# for future requests when using the opener.
# Once we log in, we will be redirected to a page that contains the _rnr_se value on it.
gv_home_page_contents = opener.open( 'https://www.google.com/accounts/ServiceLoginAuth', login_params).read()

# Go through the home page and grab the value for the hidden
# form field "_rnr_se", which must be included when sending texts and dealing with calls
match = re.search(r"'_rnr_se':\s*'([^']+)'", gv_home_page_contents)

if not match:
logged_in = False
else:
logged_in = True
_rnr_se = match.group(1)

if logged_in is True:
print "Loggin successful! _rnr_se value: {0}".format(_rnr_se)
else:
print "Loggin was unsuccessful"



If you are looking to open up a browser and automatically post form data, look at my other post here where I go into more detail.

Hope this helps!

If you want to download all of the examples here, here is a zip-file.

Wednesday, September 2, 2009

Python + Jquery: Open Browser and POST data

A few entries ago I talked about how I used Python to run some tests on a web page that I was creating. Python has a 'webbrowser' module that can open up your default web browser and point it to a specific URL. For the tests that I was running this worked well since the page expected a GET request - all parameters were passed in the URL, which I could change dynamically in my Python source. I didn't need to POST anything to the page with an HTML form.

I wanted a way to open up pages in my web browser with Python, but perform a POST request, sending data along with it. This functionality is not supported by the webbrowser module, nor do I see how it could. So, I came up with solution that meets my needs. It involves jQuery and creating then deleting a temporary file.

For those of you who do not know, jQuery is a JavaScript library that makes programming in JavaScript downright enjoyable, and provides easy solutions to common problems. Google has a Javascript API that lets you easily download stable versions of many different JavaScript libraries, including jQuery. You can do this wherever you would like, and makes it so you don't have to keep the source local (there are many benefits to this approach).

So, these are the steps we take to open a page with Python, and post our predefined (hard-coded or dynamic) form data to a page our our liking:
  1. Dynamically create a complete HTML file.
  2. Include jQuery on the page (this makes it much easier to know when the form is ready for submission).
  3. Create a form on the page with the appropriate action and method.
  4. Insert hidden form elements with their corresponding names and values.
  5. Submit the form when the DOM is finished (jQuery helps with that).
  6. Delete the file when we are done.
There are lots of ways that you could use this, but I made it as a way to test web pages I am working on. Just as an example, here is a script that you could use to open up a page after having entered your Gmail login credentials:


import os
import webbrowser

# Set up form elements - these will become the input elements in the form
input_value = {
'Passwd' : 'YOUR PASSWORD HERE',
'Email' : 'YOUR USER NAME',
'continue': 'https://mail.google.com'
}
action='https://www.google.com/accounts/ServiceLoginAuth?service=mail'
method='post'

#Set up javascript form submition function.
# I am using the 'format' function on a string, and it doesn't like the { and } of the js function
# so I brought it out to be inserted later
js_submit = '$(document).ready(function() {$("#form").submit(); });'

# Set up file content elements
input_field = '<input type="hidden" name="{0}" value="{1}" />'

base_file_contents = """
<script src='http://www.google.com/jsapi'></script>
<script>
google.load('jquery', '1.3.2');
</script>

<script>
{0}
</script>

<form id='form' action='{1}' method='{2}' />
{3}
</form>
"""

# Build input fields
input_fields = ""

for key, value in input_value.items():
input_fields += input_field.format(key, value)

# Open web page
with open('temp_file.html', "w") as file:
file.write(base_file_contents.format(js_submit,action, method, input_fields))
file.close()
webbrowser.open(os.path.abspath(file.name))
os.remove(os.path.abspath(file.name))


Let me know if you find this useful!

Tuesday, September 1, 2009

Python - Kronos Workforce Management Clock In/Out

I have been working on the Web Team of the BYU Marriott School for almost a year now. Overall it has been a very enjoyable experience, except for one thing: Kronos. Kronos is an online time management software solution that we must use to keep track of the hours that we work - you clock in when you come in, and you clock out when you leave. Pretty simple problem with a simple solution, but they have complicated this process by providing many unnecessary levels of navigation with unintuitive and inconvenient controls.

If you have two campus jobs (which I do) then it makes matters worse as the means of clocking in requires additional steps. Even though I only ever clock into one job through this website, there is no way for me to tell the system I want to use this job as my default job. It has hardwired itself to always use this other job as the default - there is no way to change that. I must explicitly tell it every time I clock in that I am clocking if for "Job 2". Also, there is also no way to quickly and easily check if you are clocked in (sometimes I forget if I clocked in). I have to go through 3 levels of navigation to find my time card and check if there is a clock-in time without a clock-out time next to it. The system won't even tell me - I have to determine my status by inspecting my time card personally.

There are many other shortcomings, but these are my biggest complaints. (I am not just a complainer, I already implemented my own time-tracker/management system that handles all of these problems in PHP that unfortunately I can't use at work for payroll purposes.)

So, I wrote a script in Python that will clock in and out of Kronos for me. It isn't too sophisticated, but it works well for my purposes. The script actually does not check to see if you are logged in or out, but based on the name of the file it will perform either the clock in or clock out actions. For example - if the name of the file is "Kronos Login.py" then it will clock you in, giving Kronos the correct job code (you hard-code that in, so I made my clock in job code "Job 2" since that is the only one I ever use). After it has done this, the script will change the name of the file it is in to "Kronos Logout.py". The next time it is run, is sees that it is named "Kronos Logout.py" and will clock out, then change it's name back to "Kronos Login.py". Like I said - not sophisticated, but it solves my problems: No clumsy navigation, and I can just check my Desktop for the script to see my clocked in/out status. I have been using it for a few days now without any problems.

Here is the source (or download it directly):


import sys, os
import urllib2, urllib

# Set up useful variables
file_name = os.path.basename(sys.argv[0])
full_path = os.path.abspath(file_name)
current_directory = os.getcwd() + '\\'

username = 'YOUR USERNAME'
password =

# URLS
kronos_home_page = 'https://kronprod.byu.edu/wfc/applications/suitenav/navigation.do?ESS=true'
kronos_login = 'https://kronprod.byu.edu/wfc/portal'
kronos_timestamp = 'https://kronprod.byu.edu/wfc/applications/wtk/html/ess/timestamp-record.jsp'

# If you have more than one job, set the number of the job you want to clock into here
# Example: '2' or '1' ('1' Kronos will assume you want job 1 by default, so you don't need to set that one)
# Leave blank if you only have one job.
job_number = '2'

# Create our opener
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())
urllib2.install_opener(opener)

login_credentials = urllib.urlencode({
'password': password,
'username': username
})

# Open home page to get cookies
opener.open(kronos_home_page)

# Login
opener.open(kronos_login, login_credentials)

#Clock in or clock out
if ('Logout' in file_name):
transfer = '' # Logging out - transfer parameter must be blank
else:
if job_number != '':
transfer = '////Job ' + job_number + '/'
else:
transfer = ''

time_stamp_parameters = urllib.urlencode({
'transfer' : transfer
})

opener.open(kronos_timestamp, time_stamp_parameters)

if ('Login' in file_name):
os.rename(full_path, current_directory + 'Kronos Logout.py')
else:
os.rename(full_path, current_directory + 'Kronos Login.py')


I hope to later turn this into a Windows sidebar gadget - if I am successful, I will post that code as well.

Thursday, August 20, 2009

Google App Engine - Cookie Handling with URL Fetch

I started working on creating a web based solution (on Google App Engine, using the Python API of course) to send mass SMS messages through your Google Voice account this week, but ran into a couple of problems right off the bat during some initial testing. The problem was that I could not log into my Google Account - the response always included a message that my "browser" didn't have cookies enabled. I was confused by this since I was using the exact method and code that I use in my Google Voice Command Line Script, which works perfectly:

self.opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())
urllib2.install_opener(self.opener)


Doing this will allow you to use the newly created "opener" object to open any URLs, and any Cookie data that is sent from the server is saved and resubmitted in the headers of each additional request. So, if you are visiting a site that requires authorization, you can send your credentials to the login page, and each subsequent request made with that opener will contain the Cookie info. This lets you access the protected areas/privileges of the site (such as sending SMS messages or making calls with your GV account) without much effort at all - it really is a nice feature of the language.

Well, I set this up in my app, but no luck. So I looked around for a bit and found out that the urllib2, urllib and httplib libraries perform requests using Google's URL fetch service (read more here). This isn't terrible, but the important thing is, is that the "urlfetch" service does NOT handle Cookies, even if you use a HTTPCookieProcessor, or a CookieJar (read more about that here).

This entry isn't a lesson on what Cookie's really are and how they work, but you should know that Cookie information is sent back from the server, and sent to the server in the headers portion of the request/response. Although the urlfetch service does not handle Cookie automatically, it does give you full access to process the header information received from a server, and also what information to send in the headers when making a request to the server. So, I built a separate class that I could use to open up URL's that would handle all Cookie information for me, as well as handle any redirects. The Google Account login system uses redirects when logging in - they forward you to a bunch of different sites to make sure you are actually logged in and that you have the right Cookie info.

Here is the class:

import urllib, urllib2, Cookie
from google.appengine.api import urlfetch

class URLOpener:
def __init__(self):
self.cookie = Cookie.SimpleCookie()

def open(self, url, data = None):
if data is None:
method = urlfetch.GET
else:
method = urlfetch.POST

while url is not None:
response = urlfetch.fetch(url=url,
payload=data,
method=method,
headers=self._getHeaders(self.cookie),
allow_truncated=False,
follow_redirects=False,
deadline=10
)
data = None # Next request will be a get, so no need to send the data again.
method = urlfetch.GET
self.cookie.load(response.headers.get('set-cookie', '')) # Load the cookies from the response
url = response.headers.get('location')

return response

def _getHeaders(self, cookie):
headers = {
'Host' : 'www.google.com',
'User-Agent' : 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729)',
'Cookie' : self._makeCookieHeader(cookie)
}
return headers

def _makeCookieHeader(self, cookie):
cookieHeader = ""
for value in cookie.values():
cookieHeader += "%s=%s; " % (value.key, value.value)
return cookieHeader



The class is simple, but works for my purposes. The class really has only one method that you should worry about: "open". Lets walk through that method: First, it checks to see if you are posting any data, and if not it makes it a GET request. Then, it starts into it's main loop. The first loop sends the data (if there is any) and some basic header information in the request, then saves the Cookie info it received in the response, then changes the request method to GET. It checks the headers for the "location" value to see if it needs to be redirected - if it does, it keeps going, saving and sending the received Cookie information along the way. Once it is done, it returns the response of the final location. Once you return the response, you can access the content by calling ".content" on the returned value.

Currently, it doesn't support GET requests with data, but mainly because I didn't need that for this project. It would be trivial to implement, however.

Here is an example on how to log in to your Google Voice account, then parse out the ever-so-important "_rnr_se" value using the URLOpener class:


opener = URLOpener()

loginParams = urllib.urlencode({
'Email' : email,
'Passwd' : password,
'continue' : 'https://www.google.com/voice/account/signin'
})

opener.open( 'https://www.google.com/accounts/ServiceLoginAuth', loginParams)

googleVoiceHomePage= opener.open('https://www.google.com/voice/#inbox').content

match = re.search('name="_rnr_se".*?value="(.*?)"', googleVoiceHomePage)

_rnr_se = match.group(1)

Hope this helps!