First and foremost is my own mother. She brought me into this world, helped raise me (along with my other brothers) and did her best with what she had (sometimes with more than what she had) in order to give me what I required to succeed at a young age. Sadly, recalling concretely how my mother has impacted my life is rather difficult as I am not aware of all the decisions she made on my behalf while I was unable to comprehend them. I do recall, however, after my parents divorce when money was no longer in abundance, a handful of situations where my mother made financial compromises in order to provide me with the essentials for the sports I played, the classes I took, or the other activities I participated in. Most notably she gave in to my request (possibly demand) for broadband Internet thus allowing me learn and explore new facets of technology at a pace that kept me glued to the computer. My mother, whether intentionally or not, gave me many freedoms to do what I desired. This freedom significantly contributed to my independence, which I strongly feel has helped me throughout my adult life. My mother has helped shape me as a person and as such I am incredibly grateful for her impact on my life.

By high school I was a fairly independent person. Due to having access to a car that I paid for, in combination with having two homes not more than ten miles apart, I effectively removed any method of discipline my parents wanted to enforce. While I was occasionally disrespectful to my parents at this time, that disrespect was never directed toward anyone else. When I started dating my high school girlfriend, I was quickly welcomed into her family. As such, Cheri, my high school girlfriend’s mother, became like a second mother to me at a time when I most needed it. There are numerous times that Cheri (and Tom, but this is Mother’s Day) encouraged, supported and pushed me to accomplish something. Most notably, Cheri ensured I completed what was necessary to apply for college, when the thought never significantly occurred to me. I distinctly remember sitting in their office on the day of the (extended) UC application deadline being pushed to finish my personal statement, and going through a number of revisions with her. The support I received from Cheri that day may very well have had the single largest impact on my future, and for that I am eternally grateful to Cheri, my second mother.

Both these women, have played a crucial rule in my life. My mother playing a significant part in shaping who I am, and Cheri in pushing me to a brighter future. I would not be where I am today without the numerous things that these women have done for me. Thank you both for everything you have done. I love you both. Happy Mother’s day.

2012-11-04 01:59:58
2012-11-04 01:59:59
2012-11-04 01:00:00
2012-11-04 01:00:01

The written timestamps make it appear as if time went back one hour, when of course time continued to proceed linearly. This perceived repetition of an hour can be a problem for computer systems that output dates in localized formats. For instance, python’s logging package produces log entries using a human-readable time. However, by default, these human readable times are in localized formats. That means an event could be logged as happening before a preceding event when the localized time zone experiences a change in daylight saving time.

An obvious solution to this problem is to not take into consideration daylight saving time when working with time. For people in the pacific time zone, that could mean always using pacific standard time (PST). Likewise for people in the eastern time zone, they could always use eastern standard time (EST). Continuing with the log file example, this means all events in a log files will appear linearly. The trade-off is anyone looking at log entries during daylight saving time needs to keep in mind the one hour difference when determining how long ago a logged event actually occurred.

A separate issue has to do with keeping track of dates and times between machines in different time zones. For simplicity, we’ll ignore the problem of clock synchronization and assume all the machines have the correct time set. The issue is that even when using standard times (non daylight saving times) on respective machines, dates and times on different machines cannot be compared in a human readable format without including time zone specific information.

Fortunately computers need not use a human readable format to store dates and times. In fact, most computers already internally keep track of time using unix timestamps that store dates and times as the number of seconds since an agreed upon date and time in coordinated universal time (UTC). In spite of this solution to synchronize dates and times from different time zones, issues such as the logging problem still occur. In the remainder of this article we will look at a few causes of those problems and how to solve them in the context of python.

Within python there are two primary ways to store date and time information. The first, is the unix timestamp, which was previously discussed. Although unix timestamps are time zone agnostic, the major problem is that it is not simple for a human to determine properties about a unix timestamp without first converting to another format. For instance, what day of the month does the unix timestamp 1325448000 represent? And, how many years does the previous unix timestamp differ from 473414400? To help answer such questions, programming languages usually have a date and time representation that can be created from unix timestamps. In python, this representation is accomplished through the datetime object:

>>> from datetime import datetime
>>> print(datetime.fromtimestamp(1325448000))
2012-01-01 12:00:00
>>> print(datetime.fromtimestamp(473414400))
1985-01-01 00:00:00

Using the datetime representation you should be able to more easily answer the previous questions. If you execute the above code, some of you might wonder why print(datetime.fromtimestamp(1325448000)) does not produce the same result on your computer. The difference is due to the fact that datetime objects are created in the context of the local time zone by default. On my system, the local time zone is “America/Los_Angeles”, which corresponds to either PST or PDT depending on the time of year. With that in mind let’s look at instances of datetime around the end of daylight savings time 2012.

>>> a = datetime.fromtimestamp(1352019599)
>>> b = datetime.fromtimestamp(1352019600)
>>> print((b - a).total_seconds())
-3599.0

Although the timestamp for a is one second less than the timestamp for b, it appears as if b represents a time 3599 seconds (one second less than one hour) prior to a. The reason is that, in addition to datetime objects using the localized time zone by default, they do not maintain the time zone information by default. Thus, naïve datetime objects, that is, those without time zone information, are only compared by their values. Below are the values for a and b from the previous example:

>>> print(a)
2012-11-04 01:59:59
>>> print(b)
2012-11-04 01:00:00

While these values are correct, we should also be able to compare them correctly. Fortunately python’s datetime objects allow us to also associate time zone information. Unfortunately, the easiest way is through the non-standard package pytz. With pytz we can easily create time zone specific datetime objects. Doing so will solve the comparison problem:

>>> import pytz
>>> a = datetime.fromtimestamp(1352019599, pytz.timezone('America/Los_Angeles'))
>>> b = datetime.fromtimestamp(1352019600, pytz.timezone('America/Los_Angeles'))
>>> print((b - a).total_seconds())
1.0
>>> print(a)
2012-11-04 01:59:59-07:00
>>> print(b)
2012-11-04 01:00:00-08:00

In the previous example we get a tzinfo object by calling pytz.timezone('America/Los_Angeles') and use that in creating the datetime object. Note that the time zone selected only affects how the datetime object is represented. Observe that the following two datetime objects are the same despite their different representation:

>>> a = datetime.fromtimestamp(1352019599, pytz.timezone('America/Los_Angeles'))
>>> b = datetime.fromtimestamp(1352019599, pytz.timezone('America/New_York'))
>>> a == b
True
>>> print(a)
2012-11-04 01:59:59-07:00
>>> print(b)
2012-11-04 03:59:59-05:00

It should now be obvious that it is always a good idea to create datetime objects with time zone information included. Two other common ways of creating datetime objects with time zone information are datetime.now(pytz.timezone('America/Los_Angeles')) to create a datetime object of the current date and time and datetime(2012, 11, 4, 1, 0, tzinfo=pytz.timezone('America/Los_Angeles')) to create a datetime object for a specific date and time.

With time zone specific datetime objects, it’s trivial to get a localized representation for any time zone through the astimezone function:

>>> print(a)
2012-11-04 01:59:59-07:00
>>> print(a.astimezone(pytz.timezone('America/New_York')))
2012-11-04 03:59:59-05:00
>>> print(a.astimezone(pytz.UTC))
2012-11-04 08:59:59+00:00

Thus far we’ve shown that using unix timestamps is great for date and time consistency between machines at the cost of not being human readable. We then gave a brief overview of datetime objects, including how to create them from unix timestamps. Most importantly we showed that datetime objects are time zone agnostic by default. We then showed how to include time zone information in datetime objects and how to convert between various local datetime representations. We will now show how to correctly convert from datetime objects back to the correct unix timestamp.

In the python standard library there is nothing that directly converts from datetime objects to unix timestamps. However, there are two functions, time.mktime, and calendar.timegm that will create timestamps from struct_time objects. These functions, in combination with the datetime timetuple functions, allow for the conversion. The primary difference between time.mktime and calendar.timegm is that time.mktime expects the struct_time object to store the date and time information in the system localized time zone, whereas calendar.timegm expects the date and time information to be stored in UTC.

Making no assumptions about the time zone the datetime object is currently represented in, the following shows how to convert a datetime object to a unix timestamp using time.mktime:

>>> import time
>>> import pytz.reference
>>> a = datetime.fromtimestamp(1352019599, pytz.timezone('America/New_York'))
>>> time.mktime(a.astimezone(pytz.reference.LocalTimezone()).timetuple())
1352019599.0

Notice that we must first convert the datetime object, a, into its identical representation in our localized time zone before calling timetuple. The pytz.reference.LocalTimezone() code prevents us from needing to hard-code our localized time zone.

The second approach to convert from a datetime object to a unix timestamp is to first convert to UTC and then use calendar.timegm:

>>> import calendar
>>> calendar.timegm(a.utctimetuple())
1352019599

When using calendar.timegm we can conveniently use the datetime utctimetuple function rather than using the astimezone function.

At this point you should have a decent grasp on converting between unix timestamps and time zone sensitive datetime objects in the context of python applications. If you are anything like me, you will now always create datetime objects with time zone information and ponder why that is not the python library default.

In a future blog post (if I ever get around to it) I will look at storing and retrieving datetime objects in databases using sqlalchemy. A few questions I have are:

• When the python application and the database are in different time zones, how are the datetime objects represented?
• What happens when either the database or the python application changes time zones?

That’s all for now. Happy coding!

Update: I just discovered that python 3.3 now has a datetime.timestamp function which handles the conversion from datetime objects to unix timestamps (source).

Immediately from links on their landing page it was apparent that Gogo permitted access to its own website, as well as the airline’s website and a few other third-party sites such as Living Social, and Eventful. Attempting to access any other website resulted in a redirect to Gogo’s landing page. Gogo appears to accomplish this by responding with an HTTP redirect to any standard HTTP request, i.e., port 80, that is sent to a non-permitted IP address.

Two common techniques for bypassing pay-for-wireless providers are TCP-over-ICMP and TCP-over-DNS. In a nutshell, TCP is the protocol required to browse the web, ICMP is the protocol used by the command `ping` to determine if a host is available, and DNS is required to resolve domain names like www.google.com to an IP address. When TCP is not completely available yet either ICMP or DNS is, then it is possible to encapsulate TCP connections over the other available protocol. Gogo blocked my ping attempts, hence TCP-over-ICMP was not possible. However, while Gogo doesn’t permit direct access to external DNS servers, Gogo’s DNS server recursively resolved the DNS queries I made. Therefore, Gogo appears to be susceptible to TCP-over-DNS. Despite not being able to verify this finding, I will simply state that Gogo can remedy this, if necessary, by only responding to white-listed DNS queries for non-authenticated users.

As a quick aside, I would like to mention I made an attempt to responsibly disclose this information to Gogo prior to posting this article [1, 2]. I contacted one of their twitter representatives via email who informed me the right person would contact me. After a few days with no response, I sent a follow up email to the same twitter representative. While that representative has continued to tweet since my followup, I have received no replies. Thus, I have come to the conclusion that Gogo is uninterested. I proceed with the knowledge that this authentication bypass in no way compromises Gogo’s security; therefore, it is of negligible importance to Gogo’s existing customers. Finally, I proceed knowing very well that Gogo’s sysadmins can correct the underlying problem in a very brief period of time and I even present them with the solution.

♫Now back to the good part!♫

I use Google Chrome as my primary browser and along with it I have installed a few extensions that depend on connections to various Google services. When Chrome is open and does not have a connection to the Internet, some of these extensions, such as the Google Reader Notifier, adjust their icon to indicate their disconnected status. As expected, the Google Reader Notifier extension indicated that I was not connected to the Internet. However, as I was browsing around Gogo’s landing page and permitted sites, I was shocked when the Google Voice extension notified me of a new text message and shocked again when I successfully replied. I had mistakenly stumbled across a hole in Gogo’s access policies and decided to dig deeper.

As I previously mentioned, Gogo redirected to their landing page upon any standard HTTP request. However, the same was not true for HTTPS connections, which by default occur on TCP port 443. Thus, for some unknown reason, my computer was able to connect to www.google.com via HTTPS. Immediately I tried other Google services, like mail.google.com for Gmail, and docs.google.com for Google docs; neither worked. Nevertheless, any Google service accessible via HTTPS and addressed under the www.google.com domain worked flawlessly.

After poking around to find a list of Google services meeting these requirements, I had an epiphany. While I probably learned this information at some point in my past, I had a hunch that Google’s front-end web servers would likely provide the correct response to any Google web service. Hence, I opened up my /etc/hosts file on my OS X machine, and added the following line:

74.125.225.40 mail.google.com

Voila! Immediately, I was able to access Gmail. The IP address, 74.125.225.40, corresponds to the IP address my computer was successfully able to connect to in order to access services under the www.google.com domain. By adding this manual entry to my hosts file, I informed my operating system to use the provided IP address when attempting to connect to mail.google.com. In order to additionally receive successful access to Google+, YouTube, Google Docs, Google Code, and Google’s chat interface, I updated the line to the following:

74.125.225.40 mail.google.com plus.google.com youtube.com docs.google.com code.google.com chatenabled.mail.google.com

What I previously described is how I was able to bypass Gogo’s Inflight Internet Authentication in order to access a number of Google services for free. What remains is how to utilize this information to access the entire Internet. The answer to this question lies with Google AppEngine.

Google AppEngine (GAE) is a cloud-based web application hosting service provided by Google. Anyone can run a GAE application on Google’s servers. As I previously described, Google’s front-end web servers respond to requests for any Google web service, including third-party GAE applications. The final piece of the puzzle is that GAE allows its applications to themselves make web requests. Therefore, a well written GAE application can operate as a proxy server. In fact, there are a number of available packages for running simple GAE proxy servers. Digital Inspiration and Windows Guides each provide tutorials for setting up such proxies. The Windows Guides tutorial even links to a working example at https://jttm-server-prox.appspot.com/. A user of Gogo’s Inflight Internet need only add jttm-server-prox.appspot.com to their hosts file in order to utilize this proxy to access much of the Internet.

I have previously described how Gogo’s Inflight Internet authentication can be bypassed through the combination of a custom hosts file and a GAE proxy server. The remainder of this article will detail how Gogo can fix this problem.

The root question, is: why does Gogo allow non-authenticated access to some Google IPs? While my answer to this question is purely speculation, the solution I offer will absolutely solve the problem. I speculate that Gogo allows access to some Google IPs because of Google Analytics and Google Adsense, indicated in client scripts by the domains www.google-analytics.com and ad.doubleclick.net respectively. If correct, Gogo’s sysadmins simply white-listed a few too many IP addresses. Regardless, the point is moot because any Google front-end web server IP address, including those for Analytics and Adsense, will serve the reply for all Google web services, most notably GAE applications. Hence, by white-listing any single Google front-end web server IP address, Gogo is essentially providing access to the entire Internet. It follows, that the simplest solution to this bypass is for Gogo to completely disable direct access to Google IPs from non-authenticated users.

While the above solution absolutely works, it prevents Gogo from tracking users and serving ads using Google services. Of course, serving ads is pointless when non-authenticated users cannot visit the target site of the advertisement, so let’s forget about Google Adsense. Google Analytics on the other hand, is a useful tool for monitoring users’ access to a website. Let’s now proceed under the assumption that it is essential for Gogo’s non-authenticated users to have access to whatever Google service Gogo intended to allow, and for simplicity assume it is the Google Analytics tracking service. The question now is: how can Gogo allow access to the Google Analytics tracking service without allowing access to all other Google services? This answer partially lies with Server Name Indication.

Server Name Indication (SNI) is implemented on most modern web browsers and allows a web server with a single IP address to serve multiple TLS certificates. The proper certificate is returned in response to the hostname provided in the TLS client handshake. The use of SNI is why, despite using the same IP address for all the aforementioned services, Google is able to return the appropriate domain-specific TLS certificate. Therefore, Gogo can additionally white-list on the SNI-hostname in order to prevent the authentication bypass and still allow access to Google Analytics. However, things are not so simple, as Google’s servers do not depend on the presence of the SNI-hostname in the TLS client handshake.

In the absence of SNI, Google’s front-end web servers return the default TLS certificate for that particular server, yet, the server still responds with the requested content as indicated in the HTTP host header. Because the HTTP host header is encrypted, as well as the entire response, it is not possible to discern desired requests from undesired requests. One solution is to only allow such access to Google Analytics if the client supports SNI and the SNI-hostname is in the white-list. While this approach would work, it appears it would restrict access to Google Analytics for some browsers that are still commonly used.

Regardless, I suspect that Google’s front-end web servers do not verify consistency between the SNI-hostname and the HTTP host header. Under that assumption, it would still be possible, in the presence of an SNI-hostname white-list, to bypass Gogo’s authentication with a custom browser (or local proxy server) that negotiates TLS handshakes using only the expected SNI-hostname yet still makes the desired Google web service request. While I don’t believe anything is readily accessible to make this process easy for 99.9% of Gogo’s potential customers, it only requires a single person’s determination to make it so. Thus, the only be-all-end-all solution to prevent this bypass is for Gogo to completely disable direct access to Google IPs from non-authenticated users.

There is, of course, one other possible solution: Gogo could man-in-the-middle the desired Google web services in order to perform filtering on the HTTP host header. However, this approach could have unforeseen consequences. Therefore, I do not recommend it.

In conclusion, I have described how one can currently bypass Gogo’s Inflight Internet authentication to access much of the Internet using a custom host file and a Google AppEngine proxy. I have also described how Gogo can somewhat block this bypass by supplementing their IP white-list with both an SNI requirement and an SNI-hostname white-list. Finally, I speculated that even with the aforementioned solution, it would still be possible for a determined user to bypass their authentication. Thus, I stated the only real solution is for Gogo to not allow any direct access between non-authenticated users and Google web servers.

Happy “free” surfing, for now anyway

March 12, 2012
2:00 PM
HFH 1132

Committee:
Diana Franklin (chair), Tim Sherwood, Kevin Almeroth

Title: Automated Feedback and Assessment of Student Submitted Code

Abstract:
Increases in class sizes pose a problem for computer science instructors. With a larger number of students to educate, instructors are forced to choose between assigning fewer assignments and devoting more of their time to satisfying the additional demands of larger class sizes. The first option, reducing the number of assignments, decreases the overall quality of a course, whereas the latter option, working more, scales only so far. Fortunately, a variety of tools exists which provides instructors with a third option. Automated feedback and assessment systems can provide significantly more detailed feedback to large numbers of students than instructors are capable of providing, and do so nearly instantaneously. The use of such systems allows instructors to continue to assign a similar number of comprehensive assignments despite growing class sizes, and also makes it possible for instructors to focus their time on students who require additional assistance with coursework.

In this talk I will provide a brief history of automated feedback and assessment systems, followed by an in-depth analysis of state-of-the-art automated feedback and assessment systems. Finally, I will provide an overview of additional research opportunities provided in other areas of computer science as a byproduct of using automated feedback and assessment systems.

Everyone is welcome to attend!

In spite of my continuing recommendation to use python for such a task, I was curious what a C solution might look like. Thus, I set out to write, and did write, a simple process-based job management system in C that offers similar functionality as python’s multiprocessing.Pool class. Take note that writing this system required more time than the equivalent python code would have, however, the point is moot because python has the multiprocessing.Pool class. Furthermore, whoever wants to use this system must manually include it in their project, thus introducing a third-party dependency in their code. Finally, this code will not compile in the typical Windows development environment as it relies on POSIX-based system calls. Hence, for all the reasons I previously mentioned, this C implementation is quite inferior to python’s multiprocessing.Pool class.

The source that I reference throughout the remainder of this post can be found and downloaded in entirety. On a POSIX-based machine the code can be compiled via gcc main.c job_management.c and the test program can be run via ./a.out.

The system, or library, is contained in two files: job_management.h and job_management.c. Anyone familiar with C or C++ development knows that job_management.h is a header file which contains the structures and function prototypes whereas job_management.c contains the implementation. The entire library functionality is exposed through three functions: manager_init, manager_run_jobs, and manager_destruct.

void manager_init(struct manager *manager, int num_workers,
		  void (*input_callback)(char *),
		  void (*output_callback)(char *),
		  void (*work_function)(char *));

Aside from the pre-allocated manager structure that all the functions require as their first parameter, the manager_init function requires four other parameters: num_workers, input_callback, output_callback, and work_function. The num_workers parameter specifies an upper bound on how many worker processes to start. Each of the remaining parameters are function pointers to functions that must each have a single char pointer parameter and return no value. I will describe the purpose of these functions in the order that they execute when considering only a single job.

The input_callback function is called by the master process, once per job, in order to fetch the input for the job. The callback function stores the job input in the provided buffer before returning. The work_function function is the reason for the entire system as this function performs the job processing. In this function, the provided buffer serves both as job input and job output. On invocation, the buffer contains the result of a prior call to input_callback which has been sent over a pipe between the master process and worker process. On return, the buffer should contain the job output which will then be sent back to the master process. The output_callback function is called by the master process as each job is completed. In this case, the buffer contains the output of a completed job.

int manager_run_jobs(struct manager *manager, int num_jobs);

Once the manager has been properly set up, a number of jobs can be started via manager_run_jobs which takes one additional argument, num_jobs. Intuitively this parameter represents the number of jobs to process. This function contains the most complicated piece of the system as it is responsible for forking the processes, setting up the pipes between the master and workers, cleaning up the address space and file descriptor table for the workers, and properly using I/O multiplexing in order to both send job input and receive job output.

One additional complication that the manger_run_jobs function has to handle is the issue that started this entire series of blog posts; that is, the want to gracefully exit the job management system when a keyboard interrupt (SIGINT), occurs. Gracefully exiting means any jobs that are running at the time of the keyboard interrupt will successfully complete prior to exiting, thus giving the master process the chance to save partial results. This function handles the desired behavior through the combination of a global volatile value that’s periodically checked within this function’s primary loop, and a SIGINT handler function that updates the value. Worker processes are unaffected by SIGINT as they are set to ignore it.

void manager_destruct(struct manager *manager);

The last function, manager_destruct, conveniently frees the contents of the manager structure. It is important to note that this function is also called by each worker as part of its address space and file descriptor table clean up. For this reason, the manager_destruct function handles the closing of some FILE structures. Otherwise, this function is incredibly simple.

As previously mentioned, my job_management library is simple and thus is no where near as flexible as python’s multiprocessing.Pool class. Furthermore, my library has no support for data serialization as it relies on messages being no larger than MAX_MESSAGE bytes in size, and has a limitation that the message contents span only a single line. These restrictions were chosen for simplicity and are, by no means, a limitation of C. However, again, I must point out that object serialization is provided by default with python, and is built into the multiprocess.Pool class.

Below is an example program that demonstrates using my process-based job management system. Note that this code really isn’t that complicated, especially when compared to the python solution that gracefully handles the keyboard interrupt. Of course, the python solution is without dependencies and much more portable.

Happy coding!

We started with a discussion of performance. It’s no secret that C will outperform python in any task; after all python is an interpreted language and the standard python implementation is written in C. However, Ali and my discussion was not focused on the performance of the jobs themselves, but rather on the performance of a job management system that could be gracefully interrupted. In such a system, where the majority of work is done in job processing, the question of the job management system’s performance is irrelevant; it makes little difference if C can launch all the workers thirty times faster when python still does it in well under a second. As an aside, it’s trivial to call a C program from python, or even to write a python C module for fast job processing performance.

Once the irrelevance of the job management performance was agreed upon, Ali and I discussed development time, dependencies and portability. Having written both a significant amount of C and python code pertaining to operating system concepts, I know that anyone with equivalent experience can much more efficiently code a process-based job management system in python than in C. This holds true, even without the multiprocessing module and when using only the low level features provided by the os package. The two primary reasons for faster development time are dynamic types, and garbage collection. Additionally, python provides a lot more out-of-the-box functionality, e.g., string manipulation and basic data structures such as lists and dictionaries; all features that contribute to faster development time.

Of course, if one had all the needed libraries in C, then my only argument in favor of python is the lack of types and implicit memory management. The choice between C and python in this case is simply up to personal preference. However, having the required C libraries is a problem on its own. This is where my lack of real-world C experience appears as I’m not familiar with well-written C libraries that provide python-like string manipulation and basic data structures, nor am I familiar with a well written process-based job management library. Nevertheless, even if these libraries were easily available, they are additional dependencies that must be present in order to run any code utilizing them.

Managing dependencies can be a pain, especially in the absence of an automated process for obtaining them. For example, I wrote some code for a computer graphics class a few years ago that I recently wanted to compile. As I recall, it took me a few hours to find and install all the required OpenGL dependencies before I could compile my own code. Compare that to the 90%+ of python programs I’ve written which have no external dependencies. This vast amount of code still works out-of-the-box across multiple platforms.

Portability is my final argument in favor of python over C. While ANSI-C based libraries should work across multiple platforms, dealing with process management is a whole other can of worms as this drastically differs between operating systems. Again, my inexperience shows as I am not familiar with any process management libraries that are compatible with both Windows and POSIX-based systems. Moreover, I am not even sufficiently familiar with the Windows API to know its equivalent functions to fork, exec, pipe and dup that are needed to write a process-based job management system for Windows. Nevertheless, with python, one needn’t worry about such things because the majority of the functionality in the os package is supported on both Windows and POSIX-based systems. Most importantly the multiprocessing module is supported on both, and thus anyone who installs python, independent of their operating system, can run the multiprocessing examples I previously provided [1, 2].

While Ali and I had a great discussion going, we still hadn’t gotten to the root of his question. Ali didn’t care about development time, nor portability. The problem was that he didn’t see any significant advantages to using the multiprocessing.Pool class when he could instead quickly write a process-based job management system in C utilizing fork to spawn processes and pipe to handle the inter-process communication (IPC). As it turns out, Ali wasn’t aware that the python multiprocessing.Pool class internally handles all the IPC. Without the built-in IPC support, Ali’s concerns would be valid as some other communication mechanism would need to be explicitly written to communicate between the master process and its workers. Fortunately, in python we get all that for free with both the multiprocessing.Pool and multiprocessing.Queue classes.

Personally, the choice of using python for such a task is a no-brainer because I can develop the system faster without the need for external dependencies and have confidence that it will work on any operating system that python runs on. Nevertheless, my curiosity peaked and I set out to write a simple process-based job management system in C. The system is complete, though I’ve yet to write about it and I’ve now written about it. Read the followup: Implementation of a Process-Based Job Management System in C

John suggests that by having the worker processes ignore SIGINT, the signal that results in python’s KeyboardInterrupt, the entire problem can be solved. Astute readers will note that I actually used the same approach in my second update to my aforementioned post, which suffered from the problem that intermediate results could not be processed, i.e., jobs that completed prior to the keyboard interrupt. While, John’s solution did educate me as to the existence of the initializer and initargs parameters to the multiprocessing.Pool function, his solution in-fact does not work. The only reason it appears to work is due to the time.sleep(10) in his try block. In most code this sleep call would not exist, rather the code would immediately call join() on the pool object.

In the absence of the delay introduced by the sleep call, John’s code still suffers from the original problem which is the KeyboardInterrupt exception does not reach the main process until all of the jobs have completed. The proper solution to the problem would be to fix the multiprocessing library to allow the join function to be interrupted. Until then, my suggestion of rolling your own pool functionality is the best solution I am aware of.

Below is a verbatim copy of my original solution for your convenience:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

#!/usr/bin/env python
import multiprocessing, os, signal, time, Queue
 
def do_work():
    print 'Work Started: %d' % os.getpid()
    time.sleep(2)
    return 'Success'
 
def manual_function(job_queue, result_queue):
    signal.signal(signal.SIGINT, signal.SIG_IGN)
    while not job_queue.empty():
        try:
            job = job_queue.get(block=False)
            result_queue.put(do_work())
        except Queue.Empty:
            pass
        #except KeyboardInterrupt: pass
 
def main():
    job_queue = multiprocessing.Queue()
    result_queue = multiprocessing.Queue()
 
    for i in range(6):
        job_queue.put(None)
 
    workers = []
    for i in range(3):
        tmp = multiprocessing.Process(target=manual_function,
                                      args=(job_queue, result_queue))
        tmp.start()
        workers.append(tmp)
 
    try:
        for worker in workers:
            worker.join()
    except KeyboardInterrupt:
        print 'parent received ctrl-c'
        for worker in workers:
            worker.terminate()
            worker.join()
 
    while not result_queue.empty():
        print result_queue.get(block=False)
 
if __name__ == "__main__":
    main()

Naturally, the obvious solution is to tether your laptop to your phone and if you have an Android phone the process is incredibly simple. In fact, there are a number of pages on the Internet that tell you exactly how to do it. Primarily, there is an Android Forums post from November, 2009 that recommends using an Android application, Azilink, in combination with an OS X application Tunnelblick, and a shell script to configure the tether correctly. There is also a great blog post from March of this year that is a bit more user friendly than the forum post in describing how to tether your OS X machine to your Android phone.

Despite the last guide being fairly recent, it unfortunately is slightly out of date due in part to OS X Lion. The older versions of Tunnelblick seemingly don’t work with OS X Lion, thus an update to a newer version of Tunnelblick is required. The newer version (currently 3.2beta32) of Tunnelblick breaks the shell script that the previous guide uses, thus I am writing this post simply to provide the Internet with an update to that shell script. Please note that everything else mentioned on the blog post currently works as described.

The fix to the shell script is actually really simple, just replace the line shown in the first block of code below, with the line shown in the second:
sudo /Applications/Tunnelblick.app/Contents/Resources/openvpn --dev tun \ sudo /Applications/Tunnelblick.app/Contents/Resources/openvpn/openvpn-2.2.1/openvpn --dev tun \

Finally for your convenience, I created a github gist with the updated shell script. Observant readers will notice that I updated some of the comments in the script. Happy tethering!

In exchange for event sponsorship, we include each company’s logo on our website, in the conference proceedings, and in a visible display at the workshop itself. For many of the previous GSWCs, we created and printed up an expensive poster with the theme of the event as well as logos of all the corporate sponsors. While that worked well, it required a decent effort to design the poster, and, as I said before, was considerably expensive to print. Thus, when I was the chair of the GSWC last year, I decided to take a different approach to how we display the corporate logos during the workshop. Rather than using a static poster, I created an animated display that was projected on one of the room’s walls during the conference.

I occasionally provide consulting for a local Santa Barbara company, Worldviz who makes a product, Vizard, that allows one to quickly construct 3D environments in python. Using Vizard I was able to quickly create a 3D logo display in which the logos follow a circular path along the z-axis, as shown in the following photos. While this worked well, Vizard unfortunately only works on Windows and thus could not be run easily from my Mac laptop.

We wanted to use the same display for this year’s GSWC so I sought to rewrite the display in a cross platform and free manor. While I have some previous opengl experience in C, I really wanted to write the display in python so I looked at the various python opengl options. From the StackOverflow thread, OpenGL with Python, I quickly settled on pyglet. Since I already knew the equation to move points around in a circular path, my only challenge was to figure out how to map a texture to a quad so that I could position the quad in 3D space. I quickly asked a StackOverflow question, Moving an image around in 3D space in hopes that while I was in the process of figuring it out, someone would provide me with the solution. Unfortunately, crowdsourcing didn’t pay off, nevertheless, I eventually found the solution and wrote a simple cross platform logo animation program.

In order to run the following program, you will first need to install the required libraries (an exercise left to the reader), create a folder “imgs” and place whatever images you want in that folder and then run the following script (download animator.py). Note: If you want to run this script on 64-bit OS X you’ll need to run it via: VERSIONER_PYTHON_PREFER_32_BIT=yes ./animator.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92

#!/usr/bin/env python
import math, os, pyglet, sys
from pyglet.gl import *
 
class World(pyglet.window.Window):
    def __init__(self, scale=10, center_pos=(0, 0, -15), speed=1.0,
                 *args, **kwargs):
        super(World, self).__init__(*args, **kwargs)
        self.scale = scale
        self.center_pos = center_pos
        self.speed = speed
        glClearColor(1.0, 1.0, 1.0, 0.0)
        glEnable(GL_DEPTH_TEST)
        self.textures = self.load_textures()
        self.clock = 0
        pyglet.clock.schedule_interval(self.update, 1 / 60.0)
 
    @staticmethod
    def load_textures():
        img_dir = 'imgs'
        textures = []
        if not os.path.isdir(img_dir):
            print 'Could not find directory "%s" under "%s"' % (img_dir,
                                                                os.getcwd())
            sys.exit(1)
        for image in os.listdir(img_dir):
            try:
                image = pyglet.image.load(os.path.join(img_dir, image))
            except pyglet.image.codecs.dds.DDSException:
                print '"%s" is not a valid image file' % image
                continue
            textures.append(image.get_texture())
 
            glEnable(textures[-1].target)
            glBindTexture(textures[-1].target, textures[-1].id)
            glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, image.width, image.height,
                         0, GL_RGBA, GL_UNSIGNED_BYTE,
                         image.get_image_data().get_data('RGBA',
                                                         image.width * 4))
        if len(textures) == 0:
            print 'Found no textures to load. Exiting'
            sys.exit(0)
        return textures
 
    def update(self, _):
        self.on_draw()
        self.clock += .01
 
    def on_draw(self):
        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)
        glLoadIdentity()
        self.draw_images()
 
    def draw_images(self):
        angle_base = (self.clock * self.speed * 50) % 360
        angle_delta = 360. / len(self.textures)
 
        for i, texture in enumerate(self.textures):
            angle = math.radians((angle_base + i * angle_delta) % 360)
            dx = math.sin(angle) * self.scale
            dz = math.cos(angle) * self.scale
 
            if texture.width > texture.height:
                rect_w = texture.width / float(texture.height)
                rect_h = 1
            else:
                rect_w = 1
                rect_h = texture.height / float(texture.width)
 
            glPushMatrix()
            glTranslatef(dx + self.center_pos[0], self.center_pos[1],
                         dz + self.center_pos[2])
            glBindTexture(texture.target, texture.id)
            glBegin(GL_QUADS)
            glTexCoord2f(0.0, 0.0); glVertex3f(-rect_w, -rect_h, 0.0)
            glTexCoord2f(1.0, 0.0); glVertex3f( rect_w, -rect_h, 0.0)
            glTexCoord2f(1.0, 1.0); glVertex3f( rect_w,  rect_h, 0.0)
            glTexCoord2f(0.0, 1.0); glVertex3f(-rect_w,  rect_h, 0.0)
            glEnd()
            glPopMatrix()
 
    def on_resize(self, width, height):
        glViewport(0, 0, width, height)
        glMatrixMode(GL_PROJECTION)
        glLoadIdentity()
        gluPerspective(65.0, width / float(height), 0.1, 1000.0)
        glMatrixMode(GL_MODELVIEW)
 
 
if __name__ == "__main__":
    window = World(width=800, height=600)
    pyglet.app.run()

Forensics 100
The forensics 100 challenge indicated to find the key, and provided a png file that was 19025×1 in resolution. Immediately our team thought we could simply change the resolution to 25×761 and would be on to something. After working with the resulting image for sometime I finally thought about converting it to 761×25. That was our first break through when we read some text along the lines of “ILoveMeSomesheepysheepies” followed by binary that includes capital ‘O’s in place of some of the ’0′s. After no success with different permutations of that message we incorporated an idea the other team members had about the blue offset pixels that occur at regular intervals. Our first attempt at wrapping the image at the blue pixel boundaries (every 450 pixels) resulted in success! The key “thankYouSirPleasemayIhaveAnother” appeared and worked. The following is my simple python solution for Forensics 100:

1
2
3
4
5
6
7
8
9
10
11

#!/usr/bin/env python
import sys, Image
 
def main():
    orig = Image.open('f100.png')
    img = Image.new('RGBA', (450, 43))
    img.putdata(orig.getdata())
    img.show()
 
if __name__ == '__main__':
    sys.exit(main())

Forensics 300
Forensics 300 was quite an interesting challenge. I don’t have the original file, nevertheless, one had to extract the initial file with a password to get a dmg containing a dump from an iphone. I came into the challenge a little late, after one of my teammates had gone through all the images, videos, and audio files looking for Waldo and ‘grep’ing for various relevant keywords. Further more, my teammates had previously used the iPhoneTracker on the consolidated.db file to see where the phone had been, however San Antonio didn’t prove to be very useful.

While the iPhoneTracker app seemed pretty cool, I wanted to programmatically see where the phone had been the most. Thus, after figuring out what was what with respect to the consolidated.db file I wrote a little python script to find the most visited places rounded to less precision to account for some variance. The top three results were the following where the first number represents the number of occurrences in that location, and the two numbers between the parenthesis represent the latitude and longitude respectfully.

• 30 (‘-77.846′, ’166.677′)
• 18 (’0.000′, ’0.000′)
• 10 (’36.106′, ‘-115.173′)

When I did a google search for the coordinates -77.846 166.667 I knew immediately that it was no coincidence that I was centered in a small town in Antarctica. Unfortunately, Google maps doesn’t have a name for this location so I had to revert to Bing (for the first time ever) to figure out that this location is called Ross Island. From that point we simply attempted different “places” listed Ross Island’s wikpiedia page until “McMurdo Station” submitted successfully. Below is the script I used to find the coordinates from the consolodated.db input file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

#!/usr/bin/env python
import os, sys
 
def main():
    os.system('sqlite3 consolidated.db "select Latitude, Longitude '
              'from CellLocation;" > tmp')
 
    uniq = {}
    for line in open('tmp'):
        pos = tuple('%.3f' % float(x) for x in line.split('|')[:2])
        if pos in uniq:
            uniq[pos] += 1
        else:
            uniq[pos] = 1
 
    for pos, count in sorted(uniq.items(), key=lambda x:x[1]):
        print count, pos
 
if __name__ == '__main__':
    sys.exit(main())

You can find links to solutions to other Defcon 19 Quals challenges at the following locations: Rogunix, negative foo, VNSecurity site.