Every Day Scripting

Login problems solved!

2011-10-08T19:19:00.000-07:00

Thanks to another reader (known only to me by 'First Last') the login issues have been resolved!

It turns out that the problem was much more simple than I originally thought. 'First Last' pointed out that the query string parameter 'service' needed to be passed along with the value of 'grandcentral' (which is the name of the company that Google acquired in order to bring us Google Voice).

The change has been made, and the latest versions of the various Google Voices script can be downloaded from my github repo.

From now on, my scripts will be available at that location. It is much easier for me to maintain, and for users of github is it much easier for you to keep up with the latest version and contribute back if you wish (which would be nice!).

Login problems

2011-10-01T12:25:00.000-07:00

Some of you may have already noticed, but the gvoice scripts I wrote no longer appear to be working. The script will either crash or report that it cannot login.

I am aware of this issue and have looked into it. Google changed the login screen again, and despite my efforts over the last few days I am unable to find out how to authenticate programmatically as I was before.

If any of you have figured out how to do this, I would love to know!

To the rest of you: hang in there. I know that this script has been very important and useful for many people. Once I figure out how to get around this I will put an update up here and in the git repo.

gvoice.py - Updated!

2011-01-05T03:53:00.000-08:00

After too long of a wait...here is an update!

Big thanks to Mike who helped me by showing me where to get the newly required "tok" variable when downloading the contact list.

I haven't been able to setup py2exe on my current computer yet, so I do not have a standalone .exe version yet (any volunteers?) but the following files are now up to date:

- gvoice.py
- gvAllInOne.py
- gvMassContact.py

Get them all by downloading this Zip file

Happy texting!

Python - Easy jQuery lightBox Gallery

2010-01-23T12:29:00.000-08:00

If you ever have to put images up on the internet, I recommend using the excellent jQuery plugin lightBox.

Not long ago, a relative asked me to help her with her website. It was made by the previous webmaster using Microsoft Publisher, and was a pain to maintain. Updating the site was especially cumbersome when she needed several images put up in a gallery. It took a long time to resize all of the images and put them in a table - normally around an hour or more. I rewrote the site in PHP and created a neat tool to automatically create a lightBox gallery for me - all I needed to do was drag and drop the folder with the original images in it onto the script file and I had my gallery ready to go! I thought I should share the script for those who may find it useful.

Fist off, however, you will need to install the Python Image Library (PIL). This handles the resizing part of the script.
You will also need jQuery included on your page, as well as everything needed for lightBox.

Once you have that set up, you should be able to use my script: (I don't like the "all catching" except, but it was useful for debugging)


import os
from PIL import Image
import sys
import glob

try:
 base_dir =  '<insert the full path to the root directory of your site (eg. c:\\sites\\myawesomesite\\)>'
 img_dir = base_dir + '<insert the folder where you keep your images (based off the base_dir defined above (eg. \\images\\)>'
 thumbs_dir = img_dir + '<insert the folder where you want the thumbnails placed, relative to the images dir defined above (eg. thumbs\\)>'
 originals_dir = img_dir + '<insert the folder where you want the originals placed, relative to the images dir defined above (eg. originals\\)>'
 web_base_path = '<put the relative URL path to the images directory (eg. /images/)>'
 web_thumbs_path = web_base_path + '<put the relative URL path to the thumbs directory (relative to the images dir defined above) (eg. thumbs/)>'

 temp_gallery_file = base_dir + '<put the name of the file where you want the gallery code to be stored (eg. TEMP_GALLERY_FILE.html)>'

 column_amount = 3 # Put how many columns wide you want the gallery to be

 img_display = """\t\t<td>
         <div class="image">
             <a href="{0}" title="">
                 <img src="{1}" alt="" />
             </a>
             <span class="description"></span>
         </div>
         <br />
     </td>
 """
   
 if len(sys.argv) == 1:
  sys.exit(1)

 resource = sys.argv[1] 

 # Default width, if it is less, than the old width is used.
 thumb_width = 200
 max_orignal_width = 700

 # Keep the aspect ration (based on the width of the image)
 def aspectRatioSize(desired_width, original_size):
  original_width = original_size[0] + .0
  original_height = original_size[1] + .0
  return (desired_width, desired_width * (original_height / original_width))
  
 def getNextAvailableName(folder, img_name, ext):
  index = 1
  while os.path.exists(folder + img_name + '_' + str(index) + ext):
   index += 1
  return img_name + '_' + str(index)+ ext

 gallery_file = open(temp_gallery_file, "w") 

 gallery_file.write("<div class=\"gallery\">\n")
 gallery_file.write("<table>\n")
 gallery_file.write("\t<tr>\n")

 base_image_name = os.path.basename(resource)
 images = [image for image in glob.glob(resource + "/*.*")]
 # Loop through all images in directory, make thumbs and resize the original 
 for index, image in enumerate(images):
  index += 1
  print image
  img_name, ext = os.path.splitext(os.path.basename(image))
  im = Image.open(image)
  img_size = im.size
  thumb_name = getNextAvailableName(thumbs_dir,  base_image_name, ext)
  modified_large_name = getNextAvailableName(img_dir, base_image_name , ext)
  original_name = getNextAvailableName(originals_dir, base_image_name, ext)
  
  # Thumbnail
  thumb = im.resize(aspectRatioSize(thumb_width, img_size), Image.ANTIALIAS)
  thumb.save(thumbs_dir + thumb_name)
  
  # Modified Large image
  modified_large = im.resize(aspectRatioSize(max_orignal_width, img_size), Image.ANTIALIAS)
  modified_large.save(img_dir + modified_large_name)
  
  # Original
  im.save(originals_dir + original_name)
  
  if index != 1 and index % column_amount == 1:
   gallery_file.write("\t<tr>\n")
   
  gallery_file.write(img_display.format(web_base_path + modified_large_name, web_thumbs_path + thumb_name))
  
  if not index % column_amount:
   gallery_file.write("\t</tr>\n")
  
 if index % column_amount > 0:
  gallery_file.write("\t</tr>\n")
  
 gallery_file.write("</table>\n")
 gallery_file.write("</div>")


 gallery_file.close()
except Exception as e:
 print e
 raw_input()

I also use the following CSS to make the gallery, once created, look the way I want.


.gallery .image {
    margin: 0 auto;
    text-align: center;
    vertical-align: top;
}

.gallery .image .description {
    font-size: 11px;
    color: #000066;
    text-align: center;
    display: block;
    line-height: normal !important;
}

.gallery td {
    vertical-align: top;
}

.gallery table {
    margin: 0 auto;
    width: 90%
}

.gallery img {
    border: none;
}

Once you have all your paths defined correctly, you can use the script by simply dragging the folder with the pictures you want in your gallery onto the script. Keep in mind, that in the current script, it will rename the pictures based on the name of the folder they are in (folder_name_1, folder_name_2, folder_name_3...). I needed something like this to avoid name collisions (it checks to make sure the name is unique).

Once the script finishes, the HTML that the script creates will be in the TEMP_GALLERY_FILE.html file (or whatever you named it). Copy and paste that into the site you want the gallery to be, and you are good to go.

I will be posting a demonstration video shortly.

Python - Google App Engine + Boggle Solver + Web2Py

2009-12-26T08:05:00.000-08:00

Click here to go right to the solver

Lately I have been experimenting with a Python web development framework called Web2Py, and so far I have been very impressed. As with any language/platform/framework, there are those who love it and those who hate it. I happen to be in the group that loves Web2Py. I also happen to like Django, another framework from which Web2Py received a lot of inspiration.

One thing that I really like about Web2Py is that it is extremely easy to get running - no installation is needed whatsoever. Download, start up the built-in server and get to developing. No configuration is needed either (You can visit their Home Page to find out more).

More importantly though is that it runs on the Google App Engine without any modification whatsoever. That is one of the things that made me choose it over Django when deciding which framework to use to put up my latest side project. Django does run on the Google App Engine - lots of people have done it, but there are too many complicated steps to get things going. Not the case at all with Web2Py.

Several posts back I posted my Boggle-Board solving code. It runs from the command line, and is not too user friendly. I decided to wrap a nice GUI around it and put it online. The resulting Boggle Solver can be found here.

Things to note about the solver:

I wrote this app because I was unhappy with all the other online boggle solvers out there
For the random boards, I use actual Boggle dice configurations not just random letters

Written completey in Python using the Web2Py framework, running on the Google App Engine

Uses a dictionary of 170+ words (I am pretty sure it is the Enable2k dictionary)

Visit my blog to see how the words are found

Enjoy!

PS: If you are a CS student somewhere and you have this same assignment: Do your own work.

Python - Custom Google Voice API - Installable Module

2009-10-09T23:32:00.000-07:00

Google Voice is a great service, but it lacks a couple of important things:

An official API from Google
A way to send mass SMS messages

I have spend some time creating my own unofficial API that is, I hope, easy to use and extend. I am by no means a seasoned Python developer, but I know enough to get what I want done. Many of the posts on my blog that interact with Google Voice depend on a "gvoice.py" script that I wrote, which contains several helpful classes. These class allow you to perform some basic actions, such as:

Logging in to your Google Voice Account
Sending a text message
Placing a phone call

The script also allows you to:

Download your Google Contacts into memory (Used mainly to get contact information to call or send a message)
Programmatically select from you Contact Groups which people to contact
Retrieve the numbers you have set up to work with your Google Voice account

Currently the script has only been tested using Python 2.6 (Python 3 might work, but I am not sure).

You can download the module with installation instructions below. Or, just scroll down to the bottom of the post to see the source in it's entirety.

gvoice.py

Here are some of the posts where I show examples on how to use this module:

And the complete source:


"""
gvoice.py

Created by: Scott Hillman

http://www.everydayscripting.blogspot.com

This module comes as is an with no warranty. 
You are free to use, modify and distribute this 
code however you wish, but I ask that if you post 
it anywhere, you at least make reference to me and
my blog where you acquired it.
"""

import csv
import sys
import re
import urllib
import urllib2

class GoogleVoiceLogin:
   """ 
    Class that attempts to log in the Google Voice     using the provided 
    credentials. 
    
    If either no password or email is provided, the user will be 
    prompted for them.
    
    Once instantiated, you can check to see the status of the log in 
    request by accessing the "logged_in" attribute
    
    The primary usage of a GoogleVoiceLogin object is to be passed
    in to other constructors, such as the TextSender, or NumberDialer
    """

   def __init__(self, email = None, password = None):
       """
        Given the email and password values, this method will attempt to log
        in to Google Voice. The "response" attribute can be checked to 
        see if the login was a success or not.
         
        If the login was successful, the "opener" and "key" attributes will
        be available to use when creating other objects. 
        
        To use an this object with the other classes in this module, simply
        pass it in to the constructor. (ie text_sender = TextSender(gv_login))
        """

       if email is None:
           email = raw_input("Please enter your Google Account username: ")
       if password is None:
           import getpass
           password = getpass.getpass("Please enter your Google Account password: ")

       # Set up our opener
       self.opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())
       urllib2.install_opener(self.opener)

       # Define URLs
       self.login_page_url = 'https://www.google.com/accounts/ServiceLogin'
       self.authenticate_url = 'https://www.google.com/accounts/ServiceLoginAuth'
       self.gv_home_page_url = 'https://www.google.com/voice/#inbox'

       # Load sign in page
       login_page_contents = self.opener.open(self.login_page_url).read()

       # Find GALX value
       galx_match_obj = re.search(r'name="GALX"\s*value="([^"]+)"', login_page_contents, re.IGNORECASE)

       galx_value = galx_match_obj.group(1) if galx_match_obj.group(1) is not None else ''

       # Set up login credentials
       login_params = urllib.urlencode({
           'Email' : email,
           'Passwd' : password,
           'continue' : 'https://www.google.com/voice/account/signin',
           'GALX': galx_value
       })

       # Login
       self.opener.open(self.authenticate_url, login_params)

       # Open GV home page
       gv_home_page_contents = self.opener.open(self.gv_home_page_url).read()

       # Fine _rnr_se value
       key = re.search('name="_rnr_se".*?value="(.*?)"', gv_home_page_contents)

       if not key:
           self.logged_in = False
       else:
           self.logged_in = True
           self.key = key.group(1)

class ContactLoader():
   """ 
    This class is used to download and organize a csv file 
    of Google Contacts.It is often used in conjunction with 
    the ContactSelector class.
    
    Example:
    
    contact_loader = ContactLoader(gv_login)
    contact_selector = ContactSelector(contact_loader)
    """
   def __init__(self, gv_login):
       """ 
        Pass in a GoogleVoiceLogin object, and the persons Google Contacts
        Will be downloaded and organized into a structure called 
        "contacts_by_group_list"
        which is organized in form:
        
        [(1, ('group_name', [contact_list])), (2, ('group_name', [contact_list]))]
        
        Which allows for easy access to any group. 
        """
       self.opener = gv_login.opener
       self.contacts_csv_url = "http://mail.google.com/mail/contacts/data/export"
       self.contacts_csv_url += "?groupToExport=^Mine&exportType=ALL&out=OUTLOOK_CSV"

       # Load ALL Google Contacts into csv dictionary
       self.contacts = csv.DictReader(self.opener.open(self.contacts_csv_url))

       # Create dictionary to store contacts and groups in an easier format
       self.contact_group = {}
       # Assigned each person to a group that we can get at later
       for row in self.contacts:
           if row['First Name'] != '':
               for category in row['Categories'].split(';'):
                   if category == '':
                       category = 'Ungrouped'
                   if category not in self.contact_group:
                       self.contact_group[category] = [Contact(row)]
                   else:
                       self.contact_group[category].append(Contact(row))

       # Load contacts into a list of tuples... 
       # [(1, ('group_name', [contact_list])), (2, ('group_name', [contact_list]))]
       self.contacts_by_group_list = [(id + 1, group_contact_item)
                                      for id, group_contact_item in enumerate(self.contact_group.items())]

class Contact():
   """ 
    Simple class to contain information on each Google Contact person.
    
    Only stores information on:
    First Name
    Last Name
    Mobile Number
    Email address
    """
   def __init__(self, contact_detail):
       """ 
        Extract data from the given contact_detail
        
        The following attributes are available:
        
        first_name
        last_name
        mobile
        email
        """
       self.first_name = contact_detail['First Name'].strip()
       self.last_name = contact_detail['Last Name'].strip()
       self.mobile = contact_detail['Mobile Phone'].strip()
       self.email = contact_detail['E-mail Address'].strip()

   def __str__(self):
       return self.first_name + ' ' + self.last_name

# Class to assist in selected contacts by groups 
class ContactSelector():
   """
    Class with helps select contacts after using the ContactLoader
    object to download them.
    
    Provides methods to:
    1) Display the list of groups (get_group_list())
    2) Set the selected group to work with (set_selected_group(group_id))
    3) Get the contacts from the working list (get_contacts_list())
    4) Remove names from the working list(remove_from_contact_list(contacts_to_remove_list))
    """
   def __init__(self, contact_loader):
       """
        Initialize the object - a ContactLoader object is expected here
        """
       self.contacts_by_group_list = contact_loader.contacts_by_group_list
       self.contact_list = None

   def get_group_list(self):
       """
        Extract a list of all the groups. 
        List is in the form:
        [(1, 'Group Name'), (2, 'Groups Name'), ...]
        """
       return [(item[0], item[1][0]) for item in self.contacts_by_group_list]

   def set_selected_group(self, group_id):
       """
        Select the group to work with. 

        This method will make the working contact_list contain all the
        contacts from the selected group. 
        """
       self.contact_list = self.contacts_by_group_list[group_id - 1][1][1]

   # Return the contact list so far
   def get_contacts_list(self):
       """
        Return a list of all the contacts, and assign them each a number

        List is in the form:
        [(1, Contact), (2, Contact), ...]
        """
       return [(id + 1, contact) for id, contact in enumerate(self.contact_list)]

   def remove_from_contact_list(self, contacts_to_remove_list):
       """
         Accepts a one based list of ids, indicating which contacts
         to remove from the list.
         
         List needs to be a list in ints:
         [3, 6, 7]
        """
       if self.contact_list is None:
           return
       for id in contacts_to_remove_list:
           if id in range(0, len(self.contact_list) + 1):
               self.contact_list[id - 1] = None
       self.contact_list = [contact for contact in self.contact_list if contact is not None]

class NumberRetriever():
   """
    Class that will allow you to retrieve all stored phone numbers and their aliases
    """

   def __init__(self, gv_login):
       """
        Pass in the GoogleVoiceLogin object, this class will then
        download all the numbers and aliases of the persons GV Account
        """
       self.opener = gv_login.opener
       self.phone_numbers_url = 'https://www.google.com/voice/settings/tab/phones'
       phone_numbers_page_content = self.opener.open(self.phone_numbers_url).read()

       # Build list of all numbers and their aliases
       self.phone_number_items = [(match.group(1), match.group(2))
                                  for match
                                  in re.finditer('"name":"([^"]+)","phoneNumber":"([^"]+)"',
                                                           phone_numbers_page_content)]

   def get_phone_numbers(self):
       """
        Return the list of phone numbers in the form:
        [(1, number), (2, number)...]
        """
       return [(id + 1, (phone_number_item))
               for id, phone_number_item
               in enumerate(self.phone_number_items)]

class TextSender():
   """
    Class used to send text messages.
    
    Example usage:
    
    gv_login = GoogleVoiceLogin('username', 'password')
    text_sender = TextSender(gv_login)
    text_sender.text = "This is an example"
    text_sender.send_text('555-555-5555')
    
    if text_sender.response:
        print "Success!"
     else:
        print "Fail!"
    """
   def __init__(self, gv_login):
       """ 
        Pass in a GoogleVoiceLogin object, set the text message 
        and then call send_text
        """
       self.opener = gv_login.opener
       self.key = gv_login.key
       self.sms_url = 'https://www.google.com/voice/sms/send/'
       self.text = ''

   def send_text(self, phone_number):
       """
        Sends a text message containing self.text to phone_number
        """
       sms_params = urllib.urlencode({
           '_rnr_se': self.key,
           'phoneNumber': phone_number,
           'text': self.text
       })
       # Send the text, display status message  
       self.response = "true" in self.opener.open(self.sms_url, sms_params).read()

class NumberDialer():
   """ 
    Class used to make phone calls.

    Example usage:

    gv_login = GoogleVoiceLogin('username', 'password')
    number_dialer = NumberDialer(gv_login)
    number_dialer.forwarding_number = 'number-to-call-you-at'

    number_dialer.place_call('number-to-call')

    if number_dialer.response:
        print "Success!"
     else:
        print "Fail!"
    """
   def __init__(self, gv_login):
       self.opener = gv_login.opener
       self.key = gv_login.key
       self.call_url = 'https://www.google.com/voice/call/connect/'
       self.forwarding_number = None

   def place_call(self, number):
       """ 
        Pass in a GoogleVoiceLogin object, set the forwarding_number
        and then call place_call('number-to-call')
        """
       call_params = urllib.urlencode({
           'outgoingNumber' : number,
           'forwardingNumber' : self.forwarding_number,
           'subscriberNumber' : 'undefined',
           'remember' : '0',
           '_rnr_se': self.key
       })

       # Send the text, display status message  
       self.response = self.opener.open(self.call_url, call_params).read()

Python - Google Voice from the command line (CLI SMS and Calling)

2009-10-08T23:22:00.000-07:00

I have written about Google Voice a few times already, but the script that gets the most attention is the one that allows you to mass text/call people in your Google Contacts Groups, which you can find here.

Many people do not need such a large script, since they simply want to be able to use something to send one text message at a time through Google Voice at the command line. Even though my scripts were written with mass contacting in mind, you can still use my gvoice module to easily contact just one person.

To use this script, you will need to download my gvoice module. There are instructions about installing this module here, as well as some documentation on it.

Once you have the gvoice module installed, creating a command line script to text someone is trivial:


import gvoice
import sys

if len(sys.argv) < 3:
    print "You must provide both a number and a message!"
    exit()

gv_login = gvoice.GoogleVoiceLogin('username', 'password')
text_sender = gvoice.TextSender(gv_login)

text_sender.text = sys.argv[2]
text_sender.send_text(sys.argv[1])

if text_sender.response:
    print 'Success! Message sent!'
else:
    print 'Failure! Message not sent!'

Notice in this script I am explicitly setting the username and password. If you leave out those parameters, you will be prompted for them at run time.

You can now call the script (which I am naming"smser.py") like so:


python smser.py "555-555-5555" "This was sent from the command line!"

If you want to make a call to someone, it is just as easy:


import gvoice
import sys

if len(sys.argv) < 3:
    print "You must provide both a forwarding number and number to dial!"
    exit()

gv_login = gvoice.GoogleVoiceLogin('username', 'password')
number_dialer = gvoice.NumberDialer(gv_login)
number_dialer.forwarding_number = sys.argv[2]
number_dialer.place_call(sys.argv[1])

if number_dialer.response:
   print 'Success! You should hear the phone ringing shortly...'
else:
   print 'Call failed!'

You would use this script (which I am naming "caller.py") much the same way:


python caller.py "number-to-dial" "number-to-call-you-at"

Just remember the order of the numbers there, and you should be good to go.

Why would this CLI be helpful to you if you have the nice GUI of Google Voice? I can think of at least two good reasons:

1) It is faster to do at the command line (If you know the numbers!)
2) You can automate the sending of a text message. One reader shared with me the idea of executing something at the command line that would take a long time, and chaining on the the end of it "smser.py 'my-number' 'Finished!'", so he could walk away from his computer and know when the command had finished up. (Thanks Shane!) You could also set up cron jobs to send out a message on a given day and time (Another friend used this to wish himself Happy Birthday)

If you are on a Windows machine, you can add the folder where you download these scripts to your PATH environment variable so you can access them anywhere. If you want to go one step further and make it so you don't even have to type in "python" before the file name or ".py" after the file name when executing a Python script, follow the instructions here.

If you are in Linux, you should be able to create a symbolic link in /usr/bin if you want to achieve the same thing.

As usual, here is a zip file containing the source code of the examples shown above, plus the examples shown in the respective files. I also threw in there the installable gvoice module and the standalone gvoice module with some brief instructions on how to install either.

gv_cli_examples.zip

Let me know if you have any questions or problems using these scripts.

Python - Fixes to the Google Login script

2009-10-02T23:54:00.000-07:00

Just recently Google changed some things behind the scenes to their login page. More specifically, on the login page there is now a hidden form element called "GALX", the value of which must be passed along with the Username and Password when signing in - other wise, the sign in fails. This affected a few of my scripts, most notably my Google Voice Mass Contact script which a few visitors let me know about.

I modified the scripts that are contained in that post to work with the new system. So, if you have been having trouble with the mass contact script, go to that post and re-download the scripts.

Here is the complete login script, including how to get the _rnr_se value after logging in.


import urllib
import urllib2
import getpass
import re

email = raw_input("Enter your Google username: ")
password = getpass.getpass("Enter your password: ")

opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())
urllib2.install_opener(opener)

# Define URLs
loing_page_url = 'https://www.google.com/accounts/ServiceLogin'
authenticate_url = 'https://www.google.com/accounts/ServiceLoginAuth'
gv_home_page_url = 'https://www.google.com/voice/#inbox'

# Load sign in page
login_page_contents = opener.open(loing_page_url).read()

# Find GALX value
galx_match_obj = re.search(r'name="GALX"\s*value="([^"]+)"', login_page_contents, re.IGNORECASE)

galx_value = galx_match_obj.group(1) if galx_match_obj.group(1) is not None else ''

# Set up login credentials
login_params = urllib.urlencode( {
   'Email' : email,
   'Passwd' : password,
   'continue' : 'https://www.google.com/voice/account/signin',
   'GALX': galx_value
})

# Login
opener.open(authenticate_url, login_params)

# Open GV home page
gv_home_page_contents = opener.open(gv_home_page_url).read()

# Fine _rnr_se value
key = re.search('name="_rnr_se".*?value="(.*?)"', gv_home_page_contents)

if not key:
   logged_in = False
   print 'Failed!'
else:
   logged_in = True
   key = key.group(1)
   print 'Success!'

Python - Back to Basics: Accessing Online Content

2009-09-18T19:18:00.000-07:00

Many of the different scripts that I write in Python deal with accessing content online. I thought that it might be a good idea to go over some of the different methods Python provides, and give some examples of their every day uses.

Most of the functionality you may ever want to access online content can be found in one of two modules (and often you need to use both) : urllib and urllib2.

First situation: You want to download the content of some page, perhaps to scrape the page for information for later processing. You want the content to be stored in a single variable (a string) so that you can do some regular expression matching on it, or some other form of parsing. A few simple lines:


import urllib2

page_contents = urllib2.urlopen("http://www.google.com").read()

print page_contents

Notice how on the second line I have the ".read()" method call. If you leave this off, you will have just the socket file object, which you can later call ".read()" on to get the contents. I like to do it all in one line however. Once you do this, it is a regular string and you can do whatever you want to/with it.

Now, lets say that you don't want to just download the page contents - you want to download an actual file, like an image, video, Power Point Presentation, PDF or anything else. Also, just a few lines:



# Download the Python image from http://python.org
import urllib

# File will be called "Python-Logo.gif",and will be contained in the folder
# where the script was executed
urllib.urlretrieve("http://python.org/images/python-logo.gif", "Python-Logo.gif")

You can use the "urlretrive" method to download pretty much any file you want - you just need to know it's URL and what you want to call it when you download it. Keep in mind that if you don't specify anything but the name of the file, it will be downloaded to the folder where you run the script. You can put things like "C:\images\Python-Logo.gif" in there if you want it to go somewhere else.

Using what we have gone over in the first two example, we can write some useful scripts. W can get the page source, find all of the PDF files linked to on it, and download them all to a folder. I do this very thing for one of the classes that I am currently taking. Here is that script:


import urllib
import re
import os

# Specifiy where I want to download my files
download_folder = "C:\DropBox\My Dropbox\School\CS479\Slides\\"

# Download page contents 
# Note: I am using urllib, not urllib2, but only because urlretrieve does not exist in urllib2
# Both have this method, and do pretty much the same thing (for this kind of usage, that is)
page_contents = urllib.urlopen("https://cswiki.cs.byu.edu/cs479/index.php/Lecture_slides").read()

# Use regular expression to find all pdf files on site. 
# match.group(1) will contain the link to the file
# match.group(2) will contain the name of the file 
for match in re.finditer(r'<a href="(.*?\.pdf)"[^>]+>([^<]+)', page_contents):
   file_url = match.group(1)
   # Remove any characters that files cannot contain
   file_name = re.sub(r'(\\|/|:|\*|\?|"|<|>|\|)', "", match.group(2))
   # Check and see if I have already downloaded the file or not
   if not os.path.exists(download_folder + file_name + '.pdf'):
       print "Downloading new file {0}...".format(file_name)
       urllib.urlretrieve(file_url, download_folder + file_name + '.pdf')

Lets make things more interesting - lets say that you want to get the content of a page, but you must be logged in to get at it. You can handle this just fine as well. Lets say that you want to get into your Google Voice account and scrape the page for the ever so important "_rnr_se" value. (I know that I have shown this a few times before, but many people still wonder how to do this, and it is a good, practical example).

Here are the steps we need to do to make this happen:

Create an "opener" with the urllib2 module, through which all of our requests will be made. The opener will be created with a HTTPCookieProcessor that will handle all the Cookies from request to request (This allows us to stay "logged in").
Install the opener, so whenever we make a request, our opener that we created will be used (and any Cookie data that has been received by previous requests will be sent along and updated when necessary)
Prepare our login credentials, and URL encode them.
Post them to the login page
Do whatever we need once we are logged in.

This might seem like a lot, but it really isn't and it is very simple to do. Here is the script for that:


import urllib, urllib2
import re
from getpass import getpass

email = raw_input("Enter your Gmail username: ")
password = getpass("Enter your password: ")

opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())
urllib2.install_opener(opener)

# Set up login credentials for Google Accounts
# The 'continue' param redirects us to the Google Voice 
# homepage, and gives us necessary cookie info
login_params = urllib.urlencode( {
   'Email' : email,
   'Passwd' : password,
   'continue' : 'https://www.google.com/voice/account/signin'
})

# Perform the login. Cookie info sent back will be saved, so we remain logged in
# for future requests when using the opener.
# Once we log in, we will be redirected to a page that contains the _rnr_se value on it. 
gv_home_page_contents = opener.open( 'https://www.google.com/accounts/ServiceLoginAuth',  login_params).read()

# Go through the home page and grab the value for the hidden
# form field "_rnr_se", which must be included when sending texts and dealing with calls
match = re.search(r"'_rnr_se':\s*'([^']+)'", gv_home_page_contents)

if not match:
   logged_in = False
else:
   logged_in = True
   _rnr_se = match.group(1)

if logged_in is True:
   print "Loggin successful! _rnr_se value: {0}".format(_rnr_se)
else:
   print "Loggin was unsuccessful"

If you are looking to open up a browser and automatically post form data, look at my other post here where I go into more detail.

Hope this helps!

If you want to download all of the examples here, here is a zip-file.

Python + Jquery: Open Browser and POST data

2009-09-02T06:31:00.000-07:00

A few entries ago I talked about how I used Python to run some tests on a web page that I was creating. Python has a 'webbrowser' module that can open up your default web browser and point it to a specific URL. For the tests that I was running this worked well since the page expected a GET request - all parameters were passed in the URL, which I could change dynamically in my Python source. I didn't need to POST anything to the page with an HTML form.

I wanted a way to open up pages in my web browser with Python, but perform a POST request, sending data along with it. This functionality is not supported by the webbrowser module, nor do I see how it could. So, I came up with solution that meets my needs. It involves jQuery and creating then deleting a temporary file.

For those of you who do not know, jQuery is a JavaScript library that makes programming in JavaScript downright enjoyable, and provides easy solutions to common problems. Google has a Javascript API that lets you easily download stable versions of many different JavaScript libraries, including jQuery. You can do this wherever you would like, and makes it so you don't have to keep the source local (there are many benefits to this approach).

So, these are the steps we take to open a page with Python, and post our predefined (hard-coded or dynamic) form data to a page our our liking:

Dynamically create a complete HTML file.
Include jQuery on the page (this makes it much easier to know when the form is ready for submission).
Create a form on the page with the appropriate action and method.
Insert hidden form elements with their corresponding names and values.
Submit the form when the DOM is finished (jQuery helps with that).
Delete the file when we are done.

There are lots of ways that you could use this, but I made it as a way to test web pages I am working on. Just as an example, here is a script that you could use to open up a page after having entered your Gmail login credentials:


import os
import webbrowser

# Set up form elements - these will become the input elements in the form
input_value = {
    'Passwd' : 'YOUR PASSWORD HERE', 
    'Email' : 'YOUR USER NAME', 
    'continue': 'https://mail.google.com'
    }
action='https://www.google.com/accounts/ServiceLoginAuth?service=mail'
method='post'

#Set up javascript form submition function. 
# I am using the 'format' function on a string, and it doesn't like the { and } of the js function
# so I brought it out to be inserted later
js_submit = '$(document).ready(function() {$("#form").submit(); });'

# Set up file content elements
input_field = '<input type="hidden" name="{0}" value="{1}" />'

base_file_contents = """
<script src='http://www.google.com/jsapi'></script>
<script>
    google.load('jquery', '1.3.2');
</script>

<script>
    {0}
</script>

<form id='form' action='{1}' method='{2}' />
    {3}
</form>
"""

# Build input fields
input_fields = ""

for key, value in input_value.items():
    input_fields += input_field.format(key, value)
    
# Open web page    
with open('temp_file.html', "w") as file:
    file.write(base_file_contents.format(js_submit,action, method, input_fields))
    file.close()
    webbrowser.open(os.path.abspath(file.name))
    os.remove(os.path.abspath(file.name))

Let me know if you find this useful!

Python - Kronos Workforce Management Clock In/Out

2009-09-01T07:59:00.000-07:00

I have been working on the Web Team of the BYU Marriott School for almost a year now. Overall it has been a very enjoyable experience, except for one thing: Kronos. Kronos is an online time management software solution that we must use to keep track of the hours that we work - you clock in when you come in, and you clock out when you leave. Pretty simple problem with a simple solution, but they have complicated this process by providing many unnecessary levels of navigation with unintuitive and inconvenient controls.

If you have two campus jobs (which I do) then it makes matters worse as the means of clocking in requires additional steps. Even though I only ever clock into one job through this website, there is no way for me to tell the system I want to use this job as my default job. It has hardwired itself to always use this other job as the default - there is no way to change that. I must explicitly tell it every time I clock in that I am clocking if for "Job 2". Also, there is also no way to quickly and easily check if you are clocked in (sometimes I forget if I clocked in). I have to go through 3 levels of navigation to find my time card and check if there is a clock-in time without a clock-out time next to it. The system won't even tell me - I have to determine my status by inspecting my time card personally.

There are many other shortcomings, but these are my biggest complaints. (I am not just a complainer, I already implemented my own time-tracker/management system that handles all of these problems in PHP that unfortunately I can't use at work for payroll purposes.)

So, I wrote a script in Python that will clock in and out of Kronos for me. It isn't too sophisticated, but it works well for my purposes. The script actually does not check to see if you are logged in or out, but based on the name of the file it will perform either the clock in or clock out actions. For example - if the name of the file is "Kronos Login.py" then it will clock you in, giving Kronos the correct job code (you hard-code that in, so I made my clock in job code "Job 2" since that is the only one I ever use). After it has done this, the script will change the name of the file it is in to "Kronos Logout.py". The next time it is run, is sees that it is named "Kronos Logout.py" and will clock out, then change it's name back to "Kronos Login.py". Like I said - not sophisticated, but it solves my problems: No clumsy navigation, and I can just check my Desktop for the script to see my clocked in/out status. I have been using it for a few days now without any problems.

Here is the source (or download it directly):


import sys, os
import urllib2, urllib

# Set up useful variables
file_name = os.path.basename(sys.argv[0])
full_path = os.path.abspath(file_name)
current_directory = os.getcwd() + '\\'

username = 'YOUR USERNAME'
password = 

# URLS
kronos_home_page = 'https://kronprod.byu.edu/wfc/applications/suitenav/navigation.do?ESS=true'
kronos_login = 'https://kronprod.byu.edu/wfc/portal'
kronos_timestamp = 'https://kronprod.byu.edu/wfc/applications/wtk/html/ess/timestamp-record.jsp'

# If you have more than one job, set the number of the job you want to clock into here
# Example: '2' or '1' ('1' Kronos will assume you want job 1 by default, so you don't need to set that one)
# Leave blank if you only have one job. 
job_number = '2'

# Create our opener
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())
urllib2.install_opener(opener)

login_credentials = urllib.urlencode({
    'password': password,
    'username': username
    })

# Open home page to get cookies 
opener.open(kronos_home_page)

# Login
opener.open(kronos_login, login_credentials)

#Clock in or clock out
if ('Logout' in file_name):
    transfer = '' # Logging out - transfer parameter must be blank
else:
    if job_number != '':
        transfer = '////Job ' + job_number + '/'
    else:
        transfer = ''
       
time_stamp_parameters = urllib.urlencode({
    'transfer' : transfer
    })
    
opener.open(kronos_timestamp, time_stamp_parameters)

if ('Login' in file_name):
    os.rename(full_path, current_directory + 'Kronos Logout.py')    
else:
    os.rename(full_path, current_directory + 'Kronos Login.py')

I hope to later turn this into a Windows sidebar gadget - if I am successful, I will post that code as well.

Google App Engine - Cookie Handling with URL Fetch

2009-08-20T08:08:00.000-07:00

I started working on creating a web based solution (on Google App Engine, using the Python API of course) to send mass SMS messages through your Google Voice account this week, but ran into a couple of problems right off the bat during some initial testing. The problem was that I could not log into my Google Account - the response always included a message that my "browser" didn't have cookies enabled. I was confused by this since I was using the exact method and code that I use in my Google Voice Command Line Script, which works perfectly:


self.opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())
urllib2.install_opener(self.opener)

Doing this will allow you to use the newly created "opener" object to open any URLs, and any Cookie data that is sent from the server is saved and resubmitted in the headers of each additional request. So, if you are visiting a site that requires authorization, you can send your credentials to the login page, and each subsequent request made with that opener will contain the Cookie info. This lets you access the protected areas/privileges of the site (such as sending SMS messages or making calls with your GV account) without much effort at all - it really is a nice feature of the language.

Well, I set this up in my app, but no luck. So I looked around for a bit and found out that the urllib2, urllib and httplib libraries perform requests using Google's URL fetch service (read more here). This isn't terrible, but the important thing is, is that the "urlfetch" service does NOT handle Cookies, even if you use a HTTPCookieProcessor, or a CookieJar (read more about that here).

This entry isn't a lesson on what Cookie's really are and how they work, but you should know that Cookie information is sent back from the server, and sent to the server in the headers portion of the request/response. Although the urlfetch service does not handle Cookie automatically, it does give you full access to process the header information received from a server, and also what information to send in the headers when making a request to the server. So, I built a separate class that I could use to open up URL's that would handle all Cookie information for me, as well as handle any redirects. The Google Account login system uses redirects when logging in - they forward you to a bunch of different sites to make sure you are actually logged in and that you have the right Cookie info.

Here is the class:


import urllib, urllib2, Cookie
from google.appengine.api import urlfetch

class URLOpener:
  def __init__(self):
      self.cookie = Cookie.SimpleCookie()
    
  def open(self, url, data = None):
      if data is None:
          method = urlfetch.GET
      else:
          method = urlfetch.POST
    
      while url is not None:
          response = urlfetch.fetch(url=url,
                          payload=data,
                          method=method,
                          headers=self._getHeaders(self.cookie),
                          allow_truncated=False,
                          follow_redirects=False,
                          deadline=10
                          )
          data = None # Next request will be a get, so no need to send the data again. 
          method = urlfetch.GET
          self.cookie.load(response.headers.get('set-cookie', '')) # Load the cookies from the response
          url = response.headers.get('location')
    
      return response
        
  def _getHeaders(self, cookie):
      headers = {
                 'Host' : 'www.google.com',
                 'User-Agent' : 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729)',
                 'Cookie' : self._makeCookieHeader(cookie)
                  }
      return headers

  def _makeCookieHeader(self, cookie):
      cookieHeader = ""
      for value in cookie.values():
          cookieHeader += "%s=%s; " % (value.key, value.value)
      return cookieHeader

The class is simple, but works for my purposes. The class really has only one method that you should worry about: "open". Lets walk through that method: First, it checks to see if you are posting any data, and if not it makes it a GET request. Then, it starts into it's main loop. The first loop sends the data (if there is any) and some basic header information in the request, then saves the Cookie info it received in the response, then changes the request method to GET. It checks the headers for the "location" value to see if it needs to be redirected - if it does, it keeps going, saving and sending the received Cookie information along the way. Once it is done, it returns the response of the final location. Once you return the response, you can access the content by calling ".content" on the returned value.

Currently, it doesn't support GET requests with data, but mainly because I didn't need that for this project. It would be trivial to implement, however.

Here is an example on how to log in to your Google Voice account, then parse out the ever-so-important "_rnr_se" value using the URLOpener class:


opener = URLOpener()

loginParams = urllib.urlencode({
   'Email' : email,
   'Passwd' : password,
   'continue' : 'https://www.google.com/voice/account/signin'
})
      
opener.open( 'https://www.google.com/accounts/ServiceLoginAuth',  loginParams)

googleVoiceHomePage= opener.open('https://www.google.com/voice/#inbox').content

match = re.search('name="_rnr_se".*?value="(.*?)"', googleVoiceHomePage)

_rnr_se = match.group(1)

Hope this helps!

Python - Boggle Solver

2009-08-17T20:50:00.000-07:00

I have been a long time fan of the game Boggle. Last year in one of my CS classes we were required to create a program in Java that would find all the words on any given Boggle board - a very fun project, and one that I found useful for a number of reasons. The other day I was messing around with my old Java implementation, and thought it would be good practice to convert it to Python. During the process, I learned a few new tricks about Python and also some other solutions to problems that may come up in other places.

The first useful thing I learned how to do was to create a class that acted as if it had two __init__ functions, which of course Python cannot handle. There are a couple of ways to do it, but my favorite implementation uses the "@classmethod" decorator. The "@classmethod" decorator makes the method static (not like in Java or C++ - use @staticmethod for that) - so within the method you can initialize a new object, set the instance variables then return the newly created object once you are done. Here is a code snippet from my Word class where I used this:


class Word:
    def __init__(self):
        self.letter_sequence = ""
        self.used_board_coordinates = set()

    @classmethod
    def new(cls, row, column):
        word = cls()
        word.used_board_coordinates.add((row, column))
        return word

    @classmethod
    def new_from_word(cls, word):
        new_word = cls()
        new_word.letter_sequence += word.letter_sequence
        new_word.used_board_coordinates.update(word.used_board_coordinates)
        return new_word

One of the constructors (called "new") accepts two parameters - the originating coordinates. It is called like so:


word = Word.new(row, column)

The second takes one parameter - another word. It creates a new word, and then copies over the data it needs. You call it much like the first:


wordCopy = Word.new_from_word(word)

Another problem I had to solve was prefix look-up. I have a large dictionary, and while searching for words on the board I only wanted to pursue directions that could potentially lead to actual words - so I needed to be able to quickly see if a letter sequence is a valid word prefix. I considered implementing a Trie or something similar, or using a modified Binary Search like I did in my Java program. I didn't really like either solution, so I came up with a solution that can take up a lot of memory, but runs extremely quickly - a simple set(). For each word loaded from my dictionary, I run a for loop that adds incrementing amounts of the word into my prefix set.


for index in xrange(len(word.strip()) + 1):
    self.prefixes.add(word[:index]

So, when I receive the word "python" from my dictionary, I add "p", "py", "pyt", "pyth"...etc to my prefix set. I was first afraid at how long it would take to do this, since the dictionary I have has 170,000+ words in it. It takes less than 1 second, and the look-up times are top-notch.

I also found a use for a generator - it could have been done without it, but I like how it cleans up the code. I use one in the BoggleSolver class - it helps me loop through all of the valid moves (coordinates) for a given word. The rules are this for Boggle - you can only use each letter (coordinate) once when constructing a word, and you can only move to adjacent spaces that are on the board. Here an example from my code:


for row, column in self._get_valid_coodinates_for_word(word, row, column):
    if(self.dictionary.contains_prefix(word.letter_sequence + self.board[row][column])):
        self._find_words(Word.new_from_word(word), row, column)

def _get_valid_coodinates_for_word(self, word, row, column):
    for r in range(row - 1, row + 2):
        for c in range(column - 1, column + 2):
            if r >= 0 and r < self.board.side_length and c >= 0 and c < self.board.side_length:
                if ((r, c) not in word.used_board_coordinates):
                    yield r, c

There are a lot of "for"s and "if" statements there, I know - but I thought it made it more readable, and it doesn't affect the performance. So, to get the coordinates for the word, simply pass the word and the current coordinates to the _get_valid_coordinates_for_word generator, and you can iterate over the returned row, column values. Pretty slick, if you ask me.

The actual problem that this script solves may not be too interesting to others, but some of the problems it solves within itself might, so I am posting the source. If you want to use it, it is ready to be used at the command line. Type "python pyBoggle.py" then a space separated list of the letters on your board. The board must be a square - 3x3, 4x4, 5x5...etc for it to word (it checks).

Example:


python pyBoggle.py a d e g n u p t e m l p w e f t

You can download the source here.
Be sure to grab the dictionary here (Place it in the same folder as the script).

Here is the source if you wish to just view it:


import sys

class BoggleSolver:
    def __init__(self, letter_list):
        self.dictionary = Dictionary("C:\\Dropbox\\Programming Projects\\Python\\PyBoggle\\dictionary.txt")
        self.board = Board(letter_list)
        self.min_length = 4
        self.found_words = set()

        # Find all words starting from each coordinate position
        for row in xrange(self.board.side_length):
            for column in xrange(self.board.side_length):
                self._find_words(Word.new(row, column), row, column)

    def _find_words(self, word, row, column):
        word.add_letter(self.board[row][column], row, column)

        if (self._can_add_word(word)):
            self.found_words.add(word.letter_sequence)

        for row, column in self._get_valid_coodinates_for_word(word, row, column):
            if(self.dictionary.contains_prefix(word.letter_sequence + self.board[row][column])):
                self._find_words(Word.new_from_word(word), row, column)

    def _can_add_word(self, word):
        return len(word) >= self.min_length and self.dictionary.contains_word(word.letter_sequence)

    def _get_valid_coodinates_for_word(self, word, row, column):
        for r in range(row - 1, row + 2):
            for c in range(column - 1, column + 2):
                if r >= 0 and r < self.board.side_length and c >= 0 and c < self.board.side_length:
                    if ((r, c) not in word.used_board_coordinates):
                        yield r, c

class Board:
    def __init__(self, letter_list):
        self.side_length = len(letter_list) ** .5
        if (self.side_length != int(self.side_length)):
            raise Exception("Board must have equal sides! (4x4, 5x5...)")
        else:
            self.side_length = int(self.side_length)

        self.board = []

        index = 0
        for row in xrange(self.side_length):
            self.board.append([])
            for column in xrange(self.side_length):
                self.board[row].append(letter_list[index])
                index += 1

    def __getitem__(self, row):
        return self.board[row]

class Word:
    def __init__(self):
        self.letter_sequence = ""
        self.used_board_coordinates = set()

    @classmethod
    def new(cls, row, column):
        word = cls()
        word.used_board_coordinates.add((row, column))
        return word

    @classmethod
    def new_from_word(cls, word):
        new_word = cls()
        new_word.letter_sequence += word.letter_sequence
        new_word.used_board_coordinates.update(word.used_board_coordinates)
        return new_word

    def add_letter(self, letter, row, column):
        self.letter_sequence += letter
        self.used_board_coordinates.add((row, column))

    def __str__(self):
        return self.letter_sequence

    def __len__(self):
        return len(self.letter_sequence)

class Dictionary:
    def __init__(self, dictionary_file):
        self.words = set()
        self.prefixes = set()
        word_file = open(dictionary_file, "r")

        for word in word_file.readlines():
            self.words.add(word.strip())
            for index in xrange(len(word.strip()) + 1):
                self.prefixes.add(word[:index])

    def contains_word(self, word):
        return word in self.words

    def contains_prefix(self, prefix):
        return prefix in self.prefixes

if __name__ == "__main__":
    boggleSolver = BoggleSolver(sys.argv[1:])
    words = boggleSolver.found_words 
    print words

Enjoy.

Python + Google Voice. Mass SMS and Iterative Calling at the Command Line

2009-08-08T09:11:00.000-07:00

EDIT (Jan 2011): Google changed the way contacts are downloaded. Many of the scripts below will break. Use the updated version here.

EDIT (Oct 3 2009): Google changed a few things on their login page which broke my old scripts. I have updated the scripts in this post, including those that you can download. If you have been receiving "Could not log in with provided credentials" errors, these new scripts should fix that problem.

You can download a ready-to-use-no-Python-installation-required Windows executable version of the program here. Just download, unzip, find the GVMassContact.exe file and double click to use. This version's source is identical to those of scripts located below.

Click here to navigate to the downloadable Python scripts.

I have already written twice about Google Voice (here and here), but the scripts in this installment are an improvement over both of the scripts provided the other posts.

I wanted to create an interactive command line script that would allow me to either send an SMS message or call everyone in one of my Google Contacts Groups. Google Voice does not yet let you send mass text messages, or call people in an iterative fashion, so I came up with a script to let me do just that.

Instead of using Googles gdata Python API to access my Google Contacts, I decided to use Pythons "csv" module to parse an exported csv file downloaded from my GMail account. Doing this lets you run the script without needing to download any other libraries, assuming you are using a version of Python that has all the features used in this script (I use 2.6.2 currently).

The first thing I did was create a "gvoice.py" module with several helpful classes. These allow you to:

Log in
Gather all Google Contacts into separate groups
Selectively narrow down the contacts in a group
Gather the phone numbers that you have entered
Send SMS messages
Place Calls

I am by no means a seasoned Python developer, but what I have created works well for my purposes. I tried to make the classes as loosely coupled with the UI as possible in case I want to put a GUI around it sometime, but for now I am doing everything at the command line (which I often prefer).

The GoogleVoiceLogin class will allow you to log in, get the "opener" which lets you keep your log in cookie data for subsequent requests, and gives you access to the _rnr_se value (the "key") that is used when sending SMS messages or placing a call. Most of the other classes in the "gvoice" module will accept the opener and key as parameters to their constructors.

I created a sample script that uses the classes in my "gvoice" module to:

Prompt the user for his/her Google Account credentials
Allow the user to select a Group from his/her Google Contacts Groups.
Allow the user to narrow down the contact list of the Group (nothing permanent is done to the Contacts)
Choose whether to send an SMS message or call everyone in the list
If calling, it allows you to choose a number from your phone list, or enter a custom number (like the Google Voice site does when placing calls)

The script that I wrote assumes I always want to use the contact's mobile phone (the first one) to send SMS messages and place calls, but that is easy to change.

There are two files needed for this to work - here is the main driver program (gvMassContact.py):


from gvoice import *
import getpass
import sys
import re
import os

# Function used to create a separator
def separator():
return '-' * 25

def get_numeric_input(prompt):
try:
return int(raw_input(prompt))
except:
pass

# Function to clear the screen
def clear_screen():
if os.name == "posix":
# *nix systems
os.system('clear')
elif os.name in ("nt", "dos", "ce"):
# Windows
os.system('CLS')

# Main method to be run 
def main():
# Log in
print "Please enter your Google Account credentials"
email = raw_input("User name: ")
password = getpass.getpass("Password: ")

gv = GoogleVoiceLogin(email, password)
if not gv.logged_in:
print "Could not log in with provided credentials"
sys.exit(1)
else:
print "Login successful!"

# Use the ContactLoader to download Google Contacts 
contact_loader = ContactLoader(gv.opener)

# Use the ContactSelector to select the group and
# final list of contacts to contact
contact_selector = ContactSelector(contact_loader.contacts_by_group_list)

clear_screen()
group_list = contact_selector.get_group_list()
selected_group = None
while selected_group not in range(1, len(group_list)+1):
print "Your Google Groups"
print separator()
for group_item in group_list:
   print "{0}: {1}".format(group_item[0], group_item[1])
print separator()
selected_group = get_numeric_input("Enter the index of the group to select: ")

clear_screen()
# Now that a group is selected, narrow down the list of people in the group
contact_selector.set_selected_group(selected_group)
removing = True
while removing:
print "Contact List"
print separator()
for contact_item in contact_selector.get_contacts_list():
   print "{0}: {1}".format(contact_item[0], contact_item[1])
print separator()
try:
   input_list = raw_input("Enter a list of the indexes (coma, space or otherwise delimeted)\nof those contacts you do not wish to contact this session.\nPress enter when finished:  ")
   if input_list != '':
       contacts_to_remove_list  = [int(match.group(1)) for match in re.finditer(r"(\d+)", input_list)]
       contact_selector.remove_from_contact_list(contacts_to_remove_list)
   else:
       removing = False
except:
   pass

clear_screen()
# Print final list
clear_screen()
print "Final List:"
print separator()
for contact_item in contact_selector.get_contacts_list():
print "{0}".format(contact_item[1])
selected_option = None
while selected_option not in [1, 2]:
print separator()
print "Options: "
print "1: Send Text"
print "2: Call"
print separator()
selected_option = get_numeric_input("Select which action to take: ")

# Send texts to all people in contact list
if (selected_option == 1):
print separator()
text_sender = TextSender(gv.opener, gv.key)
text = raw_input("Enter text message. Press enter when finished: ")
text_sender.text = text
for contact in contact_selector.get_contacts_list():
   number = contact[1].mobile
   if number == '':
       print "{0} does not have a mobile number".format(contact[1])
   else:
       print "Sending message to {0} at {1}...".format(contact[1], contact[1].mobile),
       text_sender.send_text(contact[1].mobile)
       if text_sender.response:
           print "Success!"
       else:
           print "Failed!!"
  
# Call all people in contact list             
elif (selected_option == 2):
print separator()
number_dialer = NumberDialer(gv.opener, gv.key)

number_retriever = NumberRetriever(gv.opener)
phone_number_items = number_retriever.get_phone_numbers()

clear_screen()
# Get the forwarding number
forwarding_number_input = None
while forwarding_number_input not in range(1, len(phone_number_items) + 2):
   print "Select forwarding number"
   print separator()
   for phone_number_item in phone_number_items:
       print "{0}: {1}".format(phone_number_item[0], phone_number_item[1][0])
   print "{0}: {1}".format(len(phone_number_items) + 1, "Other")
   print separator()
   forwarding_number_input = get_numeric_input("Choose from your previously entered numbers, or select  \"Other\": ")

if forwarding_number_input in range(1, len(phone_number_items) + 1):
   forwarding_number = phone_number_items[forwarding_number_input - 1][1]
else:
   forwarding_number = ''
   while not re.match(r"\(?\b[0-9]{3}\)?[-. ]?[0-9]{3}[-. ]?[0-9]{4}\b\Z", forwarding_number):
       forwarding_number = raw_input("Enter the forwarding number to dial: ")
number_dialer.forwarding_number = forwarding_number

print separator()
# Loop through and make the calls
for contact in contact_selector.get_contacts_list():
   number = contact[1].mobile
   if number == '':
       print "{0} does not have a mobile number".format(contact[1])
   else:
       input = None
       while input not in ['', 'n','N', 'q', 'Q'] :
           input = raw_input("Press enter to call {0} at {1} ('n' to skip, 'q' to quit): ".format(contact[1], contact[1].mobile))
       if input == '':
           print "Calling {0}....".format(contact[1]),
           number_dialer.place_call(number)
           if number_dialer.response:
               print "Success!"
           else:
               print "Failed!!"
       elif input .upper() == 'N':
           pass
       elif input.upper() == 'Q':
           print "Call chain aborted."
           break
          
if __name__ == "__main__":
main()

And here is the other required script (must be named "gvoice.py"):


import csv
import sys
import re
import urllib
import urllib2

class GoogleVoiceLogin:
def __init__(self, email, password):
# Set up our opener
self.opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())
urllib2.install_opener(self.opener)

# Define URLs
self.loing_page_url = 'https://www.google.com/accounts/ServiceLogin'
self.authenticate_url = 'https://www.google.com/accounts/ServiceLoginAuth'
self.gv_home_page_url = 'https://www.google.com/voice/#inbox'

# Load sign in page
login_page_contents = self.opener.open(self.loing_page_url).read()

# Find GALX value
galx_match_obj = re.search(r'name="GALX"\s*value="([^"]+)"', login_page_contents, re.IGNORECASE)

galx_value = galx_match_obj.group(1) if galx_match_obj.group(1) is not None else ''

# Set up login credentials
login_params = urllib.urlencode( {
   'Email' : email,
   'Passwd' : password,
   'continue' : 'https://www.google.com/voice/account/signin',
   'GALX': galx_value
})

# Login
self.opener.open(self.authenticate_url, login_params)

# Open GV home page
gv_home_page_contents = self.opener.open(self.gv_home_page_url).read()

# Fine _rnr_se value
key = re.search('name="_rnr_se".*?value="(.*?)"', gv_home_page_contents)

if not key:
   self.logged_in = False
else:
   self.logged_in = True
   self.key = key.group(1)

class ContactLoader():
def __init__(self, opener):
self.opener = opener
self.contacts_csv_url = "http://mail.google.com/mail/contacts/data/export"
self.contacts_csv_url += "?groupToExport=^Mine&exportType=ALL&out=OUTLOOK_CSV"

# Load ALL Google Contacts into csv dictionary
self.contacts = csv.DictReader(self.opener.open(self.contacts_csv_url))

# Create dictionary to store contacts and groups in an easier format
self.contact_group = {}
# Assigned each person to a group that we can get at later
for row in self.contacts:
   if row['First Name'] != '':
       for category in row['Categories'].split(';'):
           if category == '':
               category  = 'Ungrouped'
           if category not in self.contact_group:
               self.contact_group[category] = [Contact(row)]
           else:
               self.contact_group[category].append(Contact(row))

# Load contacts into a list of tuples...
# [(1, ('group_name', [contact_list])), (2, ('group_name', [contact_list]))]
self.contacts_by_group_list = [(id  + 1, group_contact_item)
                              for id, group_contact_item in enumerate(self.contact_group.items())]

class Contact():
def __init__(self,contact_detail):
self.first_name = contact_detail['First Name'].strip()
self.last_name = contact_detail['Last Name'].strip()
self.mobile = contact_detail['Mobile Phone'].strip()
self.email = contact_detail['E-mail Address'].strip()

def __str__(self):
return self.first_name + ' ' + self.last_name

# Class to assist in selected contacts by groups
class ContactSelector():
def __init__(self, contacts_by_group_list):
self.contacts_by_group_list = contacts_by_group_list
self.contact_list = None

def get_group_list(self):
return [(item[0], item[1][0]) for item in self.contacts_by_group_list]

def set_selected_group(self, group_id):
self.contact_list  = self.contacts_by_group_list[group_id - 1][1][1]

# Return the contact list so far
def get_contacts_list(self):
return [(id + 1, contact) for id, contact in enumerate(self.contact_list)]

# Accept a list of indexes to remove from the current contact list
# Assumes 1 based list being passed in
def remove_from_contact_list(self, contacts_to_remove_list):
if self.contact_list == None:
   return
for id in contacts_to_remove_list:
   if id in range(0, len(self.contact_list)+1):
       self.contact_list[id - 1] = None
self.contact_list = [contact for contact in self.contact_list if contact is not None]

class NumberRetriever():
def __init__(self, opener):
self.opener = opener
self.phone_numbers_url = 'https://www.google.com/voice/settings/tab/phones'
phone_numbers_page_content = self.opener.open(self.phone_numbers_url).read()

# Build list of all numbers and their aliases
self.phone_number_items = [(match.group(1), match.group(2))
                          for match
                          in re.finditer('"name":"([^"]+)","phoneNumber":"([^"]+)"',
                                                   phone_numbers_page_content)]

def get_phone_numbers(self):
return [(id + 1, (phone_number_item))
       for id, phone_number_item
       in enumerate(self.phone_number_items)]

class TextSender():
def __init__(self, opener, key):
self.opener = opener
self.key = key
self.sms_url = 'https://www.google.com/voice/sms/send/'
self.text = ''

def send_text(self, phone_number):
sms_params = urllib.urlencode({
   '_rnr_se': self.key,
   'phoneNumber': phone_number,
   'text': self.text
})
# Send the text, display status message
self.response  = self.opener.open(self.sms_url, sms_params).read()

class NumberDialer():
def __init__(self, opener, key):
self.opener = opener
self.key = key
self.call_url = 'https://www.google.com/voice/call/connect/'
self.forwarding_number = None

def place_call(self, number):
call_params = urllib.urlencode({
   'outgoingNumber' : number,
   'forwarding_number' : self.forwarding_number,
   'subscriberNumber' : 'undefined',
   'remember' : '0',
   '_rnr_se': self.key
   })

# Send the text, display status message
self.response  = self.opener.open(self.call_url, call_params).read()

For the "gvMassContact.py" script to work, you need to have the "gvoice.py" file located in the same folder.

If you would rather not copy and paste, you can download the files here:

If you don't care about having the files separated, and would rather have to deal with just one script that you can keep on your desktop, you can download this file:

gvAllInOne.py

Here is a new, ready-to-use-no-Python-installation-required Windows executable version. Just download, unzip, find GVMassContact.exe in the GVMassContact folder, double click and go.

The only difference between the "allInOne.py" script and the others is that all of the classes contained in the "gvoice.py" file are located at the top of the file.

If you download the "gvAllInOne.py" script, you can keep it on your desktop and simply double click it to get the interactive command line session to come up.

Enjoy!

Easier Python script execution in Widows

2009-08-08T08:08:00.001-07:00

While this post does not contain a sample script as all previous posts do, it can make creating and executing scripts easier if you are working on a Windows machine. There are two things that I like to set up to accomplish this:

1) Make widows execute scripts at the command line without needing to type "python" before the name of the script, or the letters ".py" after the name of the script.
2) Put the folder that contains all of my commonly used scripts on the system Path.

To accomplish the first step, do this:
Control Panel -> System -> Advanced System Settings -> Environments Variables
In the new window that shows up, scroll down in the bottom list to find PATHEXT. Double click on it and add ";.PY" to the end.

Once you are done with that, double click on the "Path". Append ";[FULL PATH TO YOUR SCRIPT FOLDER]" to what is already there.

Done!

If you have a script in your folder that you just added to your Path called "helloWorld.py" you can now simply type "helloWorld" and it will be run. It isn't that much better than "python helloWorld.py", but I prefer this method.

Python - Drag and Drop Zip

2009-08-05T06:17:00.000-07:00

I often need to quickly zip a folder or file on my desktop to email off to someone. I have 7-zip installed, and it is a great program, but wanted to come up with something even easier to use as well as try something new.

I created a Python script that sits on my desktop called "Zip". To create a zip archive of a file or folder, I simply drag and drop the file onto the "Zip" script and a new .zip file will show up wherever the script is located. It works just fine, and does not require any non-standard Python modules. Tested on on Windows, Python 2.6.2.

Here is the script source:


import os, sys
import zipfile

# Make sure a folder or directory was specified
if len(sys.argv) == 1:
   sys.exit(1)

# Assign name of file or folder to resusable variable
resource = sys.argv[1]   

# Create name of new zip file, based on original folder or file name    
zipFileName = os.path.splitext(resource)[0] + '.zip'

# Create zip file
zipFile = zipfile.ZipFile(zipFileName, "w")
  
# Function to create the archive name 
# Otherwise the zip folder contains many, unnecessary sub folders    
def getArchiveName(resource, root, file):
   if root == resource:
       return (os.sep).join([resource, file])
   else:
       return (os.sep).join([resource, root, file])
      
# Write file(s) to the zipFile        
print "Creating zip archive..."
if os.path.isdir(resource):
   for root, dirs, files in os.walk(resource):
       for file in files:
           zipFile.write((os.sep).join([root, file]),
               getArchiveName(os.path.basename(resource), os.path.basename(root), file),
               zipfile.ZIP_DEFLATED)
else:
   zipFile.write(resource, os.path.basename(resource), zipfile.ZIP_DEFLATED)

# Close file when finished    
zipFile.close()

You can also download the script here: Zip.py

The unzip version is coming soon.

Python - Opening URLs in a webbrowser

2009-07-28T14:43:00.000-07:00

The other day at work I needed to test a web page that I had created. The page was a form that would allow certain logged in users to suggest changes to profile information that we have on a group of people. The form data was pulled from a database, and the page knew whose information to pull based on a URL parameter called "mem" - which contained the numeric primary key of the directory member's record in the database. There were several places where the page could crash, based on what was coming our of the database, as the original creator didn't conform to a standard way of representing a "NULL" value. Some fields held NULL, some 0, some an empty string - and some columns could be any of those 3, but all meant the same thing. To test the page, I would simply change the "mem" parameter to pull up someone else's data - this would help me find any other bugs that I may have overlooked. The page would crash, but in our testing environment the error message (detailing the error and the line number where it occurred) is embedded within the page. I wanted to be completely thorough with this testing, but there are several hundred members of this directory, and trying every single possibility by hand would have taken a VERY long time.

Setting up a proper testing environment with unit tests and everything would be ideal - but where I work that sadly isn't standard practice, and we don't have a good way of doing that. So, I improvised by writing a quick script in Python to test every single possibility for me. First, I grabbed a list of all the primary keys of the members in the database, and then set up a URL opener using a HTTPCookieProcessor, so I could get into the protected page. I looped through the list, assigning the "mem" parameter to the primary keys in the list, and checked for a HTTPError, which would happen only if the page crashed (HTTPError is thrown when there is a 501 Server Error).

Since there was a server error, the URL opener wouldn't let me download the page to process the error message - so I used the Python webbrowser module to simply open the page up in Firefox for me - that way I could look at the page myself and examine the error. Since the page I was testing is in a protected area of the site, I needed to first open up Firefox and log in - Python can't pass cookie data to Firefox. After that, all I needed to do was run the script. All the pages that had problems would open as new tabs in my currently running Firefox instance.

Here is the script (obviously changed since I can't be giving out login credentials):


import urllib2, urllib
import webbrowser
testMemberIds = []

opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())
urllib2.install_opener(opener)

loginParams = urllib.urlencode( {
'login' : '',
'password' : '',
} )
opener.open( 'http://',  loginParams)

for memNum in testMemberIds:
   print "Trying {0}...".format(memNum),
   try:
       opener.open("?mem={0}".format(memNum))
   except urllib2.HTTPError, e:
       print "Server error! - Member: {0}".format(memNum)
       webbrowser.open("?mem={0}".format(memNum))

This is a pretty hackish way of testing a site, but it got the job done for me. It would have taken hours to test every possible primary key in the list by hand, but this script only took a a few minutes to throw together, and I found all the errors that could possibly come up (with the data currently in the database that is) and fixed - all in less than 20 minutes.

This is the first time I used the webbrowser module to do anything truly helpful to me, but it got me thinking about some other uses for it - possibly opening up separate tabs for items that match certain criteria on eBay or CraigsList. Just a thought - might be something to play with later on.

Python Generators - examples and applications

2009-07-27T21:05:00.000-07:00

Python generators are very handy - albeit a little hard to understand if you have never worked with them before. It took me a while to find out what it is they actually do, and even longer to figure out a use for one.

If you don't know what a generator is, or what one can do, this is the definition I came up with while learning about them: A generator is like an extended function that remembers that state it was in the last time it was called, and will continue from there using that same state. Generators look a lot like regular functions, but they have that characteristic "yield" keyword that sets them apart from conventional functions.

Here is a simple example:


# Generator example
def printName(name):
    for section in name.split(' '):
        yield section

for section in printName("Guido van Rossum"):
    print section

All this generator does is it takes a name and splits it into the different sections that are separated by spaces. It's output will look like this:

In this example, we are treating the generator like an iterator - which is a very common usage of generators, but definitely not the only usage.

Another way we can use a generator is by assigning it to another variable - then, every time we want the next value "yielded" by the generator, we call "<variablename>.next()".


def getNextWordGenerator():
    yield "Hello"
    yield "this"
    yield "is"
    yield "an"
    yield "example"

generator = getNextWordGenerator()

print generator.next()
print generator.next()
print generator.next()
print generator.next()
print generator.next()

Running this will give us:

But, if we try to call "generator.next()" in this example, we will get an error. This is because there is nothing left in the generator to yield. This may be what you want to happen, but maybe not. Sometimes you would rather have it start all over again. In that case, you can just put all of your yield statements in a "while True:" loop, like so:


def getNextWordGenerator():
    while True:
        yield "Hello"
        yield "this"
        yield "is"
        yield "an"
        yield "example"

generator = getNextWordGenerator()

print generator.next()
print generator.next()
print generator.next()
print generator.next()
print generator.next()
print
print 'Second time around, but only a little'
print generator.next()
print generator.next()

The output this time around will look like this:

This is just the beginning, however - you can do more complicated and useful things (of course, that depends on your definition of useful).

Lets says, for example, you are a member of Project Euler and you need a function that will spit out Fibonacci numbers. There are a few ways of calculating Fibonacci numbers, but I think that Python generators is the easiest, and fastest way. Here is an example:


def fib():
    x,y = 1,1
    while True:
        yield x
        x,y = y, x+y

for num in fib():
    print num

Output:

Running this will print out Fibonacci numbers at an alarming rate. You will have to hit CTRL-C to kill the script, or it will keep on going forever - and fast. I felt...odd...putting a "while True:" loop in my code for this first time when doing this. That seemed like something I should avoid. Don't worry though - it is not going to peg your processor or anything - the loop will only be run when called. It will not be run in he background without you knowing about it.

This example also acts as a reminder that a generator can remember it's state. In this case, the generator keeps track of the fact that it is in a loop, and it remembers the values in "x" and "y". So, the next time fib().next() is called by the iterator (this all happens automatically in the for loop) it doesn't start over at the top of the generator with "x,y = 1,1", but in the loop where it last left off. Very handy.

So, how can this help in every day scripting situations? Well, I am not too sure. I have found them very useful in solutions to Project Euler problems, and I used a generator in a school project to return all valid adjacent points to a point that I passed into the generator - but other than that, I haven't found a great reason to use them frequently in every day situations. But, there are some libraries and other built in functions of Python that do use them heavily. For example - xrange uses a "generator" rather than creating a list first in memory like "range" does. "count" in the "itertools" uses a generator to give you the next number in sequence for as long as you want.

You can use generators to simulate continuation programming - which also takes some time getting used to, but is pretty neat when you see it. Here is an example of that:


def fib():
  x,y = 1,1
  while True:
      yield x
      x,y = y, x+y

def odd(seq):
  for number in seq:
      if number % 2:
          yield number
 
def underFourMillion(seq):
  for number in seq:
      if number > 4000000:
          break
      yield number 
         
print sum(odd(underFourMillion(fib())))

This program sums all of the odd Fibonacci numbers that are under 4 million, but at no point does it store anything in a data structure. The "sum" built-in method will add numbers coming from the "odd" generator, which yields any odd numbers coming from the "underFourMillion" generator, which yields any number that is under 4 million coming from the "fib" generator. Neat. It is easy to change any part of this program, or even add another "filter" generator to the mix.

Do you have any other uses for generators? Share them in the comments.

Redirect Python output

2009-07-21T15:03:00.000-07:00

EDIT: Updated scripts to use the new login system described here.

Today at work I was working on a script to create some reports for one of my superiors. During my testing I was outputting everything to the screen so I could view it quickly and easily. My boss needed me to send him a file containing the data I was generating, and I didn't look forward to changing all of my print statements (my script had many, many print statements) to something like "file.write(whateverwasbeingprintedbefore)". I could have done this, of course (and it could have been done in one easy step with the help of RegexBuddy) but that still seemed too inflexible of a solution. Python has a very simple solution for this problem, and it just takes an extra two lines of code.

My work example would have been boring here, so I thought up something else useful. Using my Google Voice Login class, I created another script to print out whatever is in my SMS inbox (up to the first 10) in a semi decent format. I used a couple of regular expressions, which I could not have created without RegexBuddy, to parse the page for the information I wanted. Here is that script (which requires the gvoice.py module I wrote):


# Print SMS inbox
from gvoice import GoogleVoiceLogin
import urllib
import re
import getpass

# Create an instance of a GoogleVoiceLogin object
# This will prompt you for your Google Account credentials
email = raw_input('Enter your Google Username: ')
password = getpass.getpass('Enter your password: ')

gv_login = GoogleVoiceLogin(email, password)

# Create our regular expressions (Created in RegexBuddy)
# Regular Expression to gather information on each SMS Conversation
sms_conversation_regex = re.compile(
   r"""<span\sclass="gc-message-name">\s*         # Get the message conversation info div
    <span\sclass="">([^<]*?)</span>\s*          # Get who originated the conversation
    <span.*?</span>.*?                          # Eat this div - we don't need it
    .*?>([^<]*).*?                              # Get who the conversation was started with
    <div\sclass="gc-message-message-display">\s*
    (<div\sclass="gc-message-sms-row">.*?       # Get all the messages of the conversation
    </div>\s*)*                                 # Close up our divs
    </div>""",
   re.DOTALL | re.VERBOSE)
  
# Regular expression to gather information on individual messages within an SMS conversaion
sms_details_regex = re.compile(
   r"""<div\sclass="gc-message-sms-row">.*?         # Get up the message div
    <span\sclass="gc-message-sms-from">\s*(.*?)\s*   # Get who the message was from
    </span>.*?
    <span\sclass="gc-message-sms-text">\s*([^<]*)\s*.*?</div> # Get the message data""",
   re.DOTALL | re.VERBOSE)

# Get the open (Cookie data still intact)
opener =gv_login.opener

sms_inbox_content = opener.open("https://www.google.com/voice/inbox/recent/sms/").read()

# Nested for-loops of regular expressions - not the best way, but the easiest for now. 
for sms_conversation_match in sms_conversation_regex.finditer(sms_inbox_content):
   print
   print "---{0} to {1} {2}".format(sms_conversation_match.group(1), sms_conversation_match.group(2), '-' * 50)
   for message_details_match in sms_details_regex.finditer(sms_conversation_match.group()):
        print "{0}: {1}".format(message_details_match.group(1), message_details_match.group(2))

Running this script will show you the first 10 SMS conversations in your inbox, if there are any. But, it is limited right now to displaying the information on the screen. Using these next two lines, you can put all of this information in a file of your choice:


import sys

sys.stdout = open("SMS Conversations.txt", "w")

To get this to work in the SMS inbox script, I insert the "import sys" command right below the other imports. The "sys.stdout" line goes right below the line where I instantiate the GoogleVoiceLogin object. I deliberately do that, otherwise I won't be able to see the prompt, because it will be written to the file!

This is a very easy way to change the output of your program - I hope you find it as useful as I did.

Here is the final script:


# Print SMS inbox
from gvoice import GoogleVoiceLogin
import urllib
import re
import sys
import getpass

# Create an instance of a GoogleVoiceLogin object
# This will prompt you for your Google Account credentials
email = raw_input('Enter your Google Username: ')
password = getpass.getpass('Enter your password: ')

gv_login = GoogleVoiceLogin(email, password)

sys.stdout = open("SMS Conversations.txt", "w")

# Create our regular expressions (Created in RegexBuddy)
# Regular Expression to gather information on each SMS Conversation
sms_conversation_regex = re.compile(
   r"""<span\sclass="gc-message-name">\s*         # Get the message conversation info div
    <span\sclass="">([^<]*?)</span>\s*          # Get who originated the conversation
    <span.*?</span>.*?                          # Eat this div - we don't need it
    .*?>([^<]*).*?                              # Get who the conversation was started with
    <div\sclass="gc-message-message-display">\s*
    (<div\sclass="gc-message-sms-row">.*?       # Get all the messages of the conversation
    </div>\s*)*                                 # Close up our divs
    </div>""",
   re.DOTALL | re.VERBOSE)
  
# Regular expression to gather information on individual messages within an SMS conversaion
sms_details_regex = re.compile(
   r"""<div\sclass="gc-message-sms-row">.*?         # Get up the message div
    <span\sclass="gc-message-sms-from">\s*(.*?)\s*   # Get who the message was from
    </span>.*?
    <span\sclass="gc-message-sms-text">\s*([^<]*)\s*.*?</div> # Get the message data""",
   re.DOTALL | re.VERBOSE)

# Get the open (Cookie data still intact)
opener =gv_login.opener

sms_inbox_content = opener.open("https://www.google.com/voice/inbox/recent/sms/").read()

# Nested for-loops of regular expressions - not the best way, but the easiest for now. 
for sms_conversation_match in sms_conversation_regex.finditer(sms_inbox_content):
   print
   print "---{0} to {1} {2}".format(sms_conversation_match.group(1), sms_conversation_match.group(2), '-' * 50)
   for message_details_match in sms_details_regex.finditer(sms_conversation_match.group()):
        print "{0}: {1}".format(message_details_match.group(1), message_details_match.group(2))
   print "---{0} to {1} {2}".format(sms_conversation_match.group(1), sms_conversation_match.group(2), '-' * 50)
   for message_details_match in sms_details_regex.finditer(sms_conversation_match.group()):
        print "{0}: {1}".format(message_details_match.group(1), message_details_match.group(2))

Python - Google Voice part 2

2009-07-21T14:38:00.000-07:00

EDIT: Google recently made some behind the scenes changes to the login page, which broke this script. Please see the new and improved script here.

My last post was also about accessing your Google Voice account through Python, but there is more that you can do with it than just send SMS messages. You can make calls, cancel calls, view your call/sms/voicemail history and a few more others, as pointed out by this blogger here while talking about his Firefox plugin.

Since all of these options require you to be logged in, I thought that it would be easier to create a separate class that would:
1) Log you in and let you know whether or not your attempt was successful
2) Provide a method to get the "opener" (which is what keeps cookie data in order during multiple requests),
3) Provide a method to get your "_rnr_se" value, which is required when sending SMS messages, making calls, and canceling calls.

Instead of hardcoding in Google Account credentials like in my last post, the new class will prompt you for your Google Account user name and password. Since I want to use this script in public, I used the getpass module to hide the input as I type it. Here is the script:


# Get URL handling support
import urllib2, urllib
# Get regular expression support
import re, getpass

class GoogleVoiceLogin:
def __init__(self):
print "Please enter your Google Account credentials"
self.email = raw_input("User name: ")
self.password = getpass.getpass("Password: ")

# Set up an opener with HTTPCookieProcessor
self.opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())
urllib2.install_opener(self.opener)

# Set up login credentials for Google Accounts
# The 'continue' param redirects us to the Google Voice 
# homepage, and gives us necessary cookie info
loginParams = urllib.urlencode( {
    'Email' : self.email,
    'Passwd' : self.password,
    'continue' : 'https://www.google.com/voice/account/signin'
} )

# Perform the login. Cookie info sent back will be saved, so we remain logged in
# for future requests when using the opener
self.opener.open( 'https://www.google.com/accounts/ServiceLoginAuth',  loginParams)

# Need to load the homepage to find user specific data
googleVoiceHomeData = self.opener.open('https://www.google.com/voice/#inbox').read()

# Go through the home page and grab the value for the hidden
# form field "_rnr_se", which must be included when sending texts and dealing with calls
match = re.search('name="_rnr_se".*?value="(.*?)"', googleVoiceHomeData)
if not match:
    print "Login Unsuccessful!"
    exit()
else:
    print "Login Successful!"
    self._rnr_se = match.group(1)

def getOpener(self):
return self.opener

def getRnrSe(self):
return self._rnr_se

The script, for the most part, should be self-explanatory. It pretty much follows the same pattern as the script in my previous post except it is more reusable. For example, here is the new script I use to send text messages from the command line:


# Send a text Message
from GoogleVoiceLogin import GoogleVoiceLogin
import urllib

# Create an instance of a GoogleVoiceLogin object
# This will prompt you for your Google Account credentials
gvLogin = GoogleVoiceLogin()

# Get the open (Cookie data still intact)
opener = gvLogin.getOpener()

# Get the _rnr_se value
_rnr_se = gvLogin.getRnrSe()

# Prompt for Text Message details
phoneNumber = raw_input("Destination number: ")
text = raw_input("Message to send: ")

# Insert blank line
print

# Set up parameters for sending text
sendTextParams = urllib.urlencode({
    '_rnr_se': _rnr_se,
    'phoneNumber': phoneNumber,
    'text': text
})

# Send the text, display status message  
response  = opener.open('https://www.google.com/voice/sms/send/', sendTextParams)
if "true" in response.read():
    print "Message successfully sent!"
else:
    print "Message failed!"
response.close()

Similar scripts could easily be created to call people, or perform other tasks on your account.

Google Voice - Python - SMS

2009-07-13T13:10:00.000-07:00

EDIT: Google recently made some behind the scenes changes to the login page, which broke this script. Please see the new and improved script here.

After a long wait, I finally received my Google Voice invitation last weekend. It offers a lot of features, but one that I was really excited about was the free SMS service. I don't have a cell phone, but nearly all my family members, friends and co-workers do. The majority of them are more prone to check their phone than their email, so sending a text is the best way to contact them. For a while, I used Gmail's SMS Lab to get in contact with the leaders and the Scouts of a Boy Scout troop that I participate in, but Gmail shut that down recently.

After I got my account, I began thinking about how neat it would be to be able to set up a script to send reminder text messages for me (preferably scheduled) using my account. I searched all over the internet to find out if there was an API provided by Google to do this -like their excellent gdata API - but no luck. I didn't give up there, there had to be a way.

A while ago, I learned that you can use HTTPCookieProcessor() from the urllib2 module to create an opener that will keep track of Cookie data sent back and forth from a server. So, with the help of Firebug (the best friend of many web-developers) I looked at the headers and POST data being sent when logging in to Google Voice, and when sending a text message. Turns out, it isn't very hard to log into your account, grab some necessary information from the Google Voice Home page (the inbox) and send a text.

This script will do just that one thing - log in your account, load the homepage to get a hidden form field's value, and then send one text message to one number. It will still show up in your outbox, so you aren't losing anything by using this method. This ability opens up a lot of possibilities, especially when coupled with the Gdata Contacts API. I have a few little scripts that I will be creating to send out weekly reminders to other people - and might use Google App Engine to make it even more useful and automated.


# Get URL handling support
import urllib2, urllib
# Get regular expression support
import re

# Google Account login credentials
email = 'YOUR_EMAIL'
password = 'YOUR_PASSWORD'

# Text message details
sendToNumber = 'NUMBER_TO_SEND_A_TEXT_TO'
messageToSend = 'I used Python to send this!'

# Set up an opener with HTTPCookieProcessor
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())
urllib2.install_opener(opener)

# Set up login credentials for Google Accounts
# The 'continue' param redirects us to the Google Voice 
# homepage, and gives us necessary cookie info
loginParams = urllib.urlencode( {
    'Email' : email,
    'Passwd' : password,
    'continue' : 'https://www.google.com/voice/account/signin',
} )

# Perform the login. Cookie info sent back will be saved, so we remain logged in
# for future requests when using the opener
opener.open( 'https://www.google.com/accounts/ServiceLoginAuth',  loginParams)

# Need to load the homepage to find user specific data
googleVoiceHomeData = opener.open('https://www.google.com/voice/#inbox').read()

# Go through the home page and grab the value for the hidden
# form field "_rnr_se", which must be included when sending texts
match = re.search('name="_rnr_se".*?value="(.*?)"', googleVoiceHomeData)
_rnr_se = match.group(1)

# Set up parameters for sending text
sendTextParams = urllib.urlencode({
    '_rnr_se': _rnr_se,
    'phoneNumber': sendToNumber,
    'text': messageToSend
})

# Send the text, store the return value  
f  = opener.open('https://www.google.com/voice/sms/send/', sendTextParams)
data = f.read()
print data
f.close()

Right now, if it succeeds nothing spectacular happens - it simply prints out the response: {"ok":true,"data":{"code":0}}. If it fails, {"ok":false,"data":{"code":20}}. Just look at the "ok" value to see if it worked.

You may notice that I don't have much (any) error checking in the script, nor do I usually when putting together scripts quickly just to test an idea out. When I do create something actually useful using this above script, I will put in the appropriate "checker" code. Just be aware that a text message can be 160 characters long, if you go over the remaining characters will be sent in a separate message.

Have fun, and don't be evil.

Syntax Highlighting - quick and easy

2009-07-10T20:08:00.000-07:00

Edit: I recently made my life easier by switching over to http://code.google.com/p/google-code-prettify/ for syntax highlighting.

Since this blog is mainly about Python examples, I needed a way to post syntax highlighted Python code. There were a few solutions, including a hosted javascript option. I didn't like that one too much for a few reasons. After short struggle, I was able to install Pygments on my computer which is a Python module that can apply syntax highlighting to many different programming languages' source. I wanted a quick and easy way of writing Python code in my favorite text editor, Notepad++, and getting it up on this blog. I thought it would be neat to create a syntax_highlighter.py script, and use the NppExec plug-in to run the syntax highlighter on the current document, and then paste the results on the clipboard. To do that last part, I had to download and install the pywin32 library (you can find that on sourceforge). Here is my final script:


from pygments import highlight
from pygments.lexers import PythonLexer
from pygments.formatters import HtmlFormatter
import win32clipboard
import sys,re

# Replace tab characters with 4 spaces for better look on webpage
def replaceTabs(matchobj):
return "    "

# Open up and store the souce file contents
rawCode = open(sys.argv[1], "r").read()

# Apply syntax highlighting
highlightedCode = highlight(rawCode, PythonLexer(), HtmlFormatter())

# Replace tabs with spaces
finalCode = re.sub("\t", replaceTabs, highlightedCode)

win32clipboard.OpenClipboard()
win32clipboard.EmptyClipboard()
win32clipboard.SetClipboardText(finalCode)
win32clipboard.CloseClipboard()

This will put on my clipboard code that has been surrounded with the appropriate html tags to make it appear nicely formatted and highlighted. I created another function to replace all tabs with 4 spaces to make it look better (I found it easier to do that with the regular expression module).

If you use Notepad++, I used this to run the script on the current open file:

python "FULL_PATH_TO_YOUR_HIGHLIGHTING_SCRIPT" "$(FULL_CURRENT_PATH)"

The first argument should be the full path to the syntax_highlight.py script - the next ,"$(FULL_CURRENT_PATH)", is a standard variable that the NppExec plug-in uses to define the current open file. Once run, the result will be on your clipboard ready to be pasted. Once set up, having it ready to paste on this blog is a CTRL-F6 keystroke away. This could work for any language that Pygments supports.

The clipboard portion of the script was a lot easier than I thought that it would be. The pywin32 library has many useful features in it - I know that similar solutions exist for Linux.

NOTE: I ran


from pygments import highlight
from pygments.lexers import PythonLexer
from pygments.formatters import HtmlFormatter

print HtmlFormatter().get_style_defs('.highlight')

to get my CSS rules, which I was able to paste in my template.

What is this?

2009-07-10T09:25:00.000-07:00

Many of my co-workers and class-mates have been asking me what Python can actually be used for, so I decided to start this blog demonstrating some of the ways that I have been employing its power and simplicity.

I am by no means a Python expert, but I like it quite a bit. I have been using it to perform little tasks at work and school for a while now and hope that by seeing some of my examples, you can see what it might be able to do for you. Each post here will contain a code snippet that I have created to help perform some task. I won't be really talking much about events happening in the Python community - there are plenty of other blogs for that. Here is a blog of simple, practical examples that might come in handy.