The paper is situated in the field of ubiquitous computing. The goal of Mark Weiser’s vision of ubiquitous computing was calm computing, where calm reflects the desired state of mind of the user. Invisibility in ubicomp is more about enabling seamless accomplishment of task than staying beneath notice. Just as a good, well-balanced hammer “disappears” in the hands of a carpenter and allows him or her to concentrate on the big picture, computers should participate in a similar magic disappearing act. Calm computing moves between the center and the periphery of attention. The periphery informs without overwhelming the user, but the user can still move to the center to get control. The implicit interaction framework presented in this paper contributes to how calm computing can be effectively realized. It allows to reason about the way users can mitigate actions the system takes, and is a good complement to Eric Horvitz’s work on mixed-initiative interaction.

Implicit interactions enable communication and action without explicit input or output.One way that an action can be implicit is if the exchange occurs outside the attentional foreground of the user (e.g. auto-saving files, filtering spam, and ubicomp interaction). The other way is if the exchange is initiated by the computer system rather than by the user (e.g. an email alert, screen saver, etc.). Although it may seem strange that something that grabs attention is implicit, the key factor is that the interaction is based on an implied demand for information or action, not an explicit one.

The implicit interaction framework divides the space of possible interactions along the axes of attentional demand and initiative:

Attentional demand is the degree of cognitive and perceptual load imposed on the user by the interactive system. Foreground interactions require a greater degree of focus, concentration and consciousness and are exclusive of other focal targets. Background interactions are peripheral, have less demand and can occur in parallel with other interactions.

Initiative is an indicator of how much presumption the interactive system uses in the interaction. Interactions that are initiated and driven by the user explicitly are called reactive interactions, while interactions initiated by the system based on inferred desire or demand are proactive interactions.

The implicit interaction framework builds on Bill Buxton’s background/foreground model. Buxton’s model assumes attention and initiative are inherently linked. On the contrary, this framework decouples attention and initiative in two separate axes. Buxton’s foreground corresponds to the reactive/foregound quadrant, while his background corresponds to the proactive/background quadrant.

An example: a word processing program that …

• auto-saves because you command it to is situated in the reactive/foreground quadrant
• auto-saves because you have set it to do so every 10 minutes is situated in the reactive/background quadrant
• auto-saves because it feels that a lot of changes have been made is situated in the proactive/background quadrant

Being proactive means the word processing program is acting with greater presumption with respect to the needs and desires of the user.

Designers can manipulate the proactivity and reactivity by (1) dictating the order of actions (does the system act first or wait for the user to act?); or by (2) choosing the degree of initiative (does the sytem act, offer to act, ask if it should act, or merely indicate that it can act?); or by (3) gathering more data to ensure the certainty of the need for an action or when they design features to mitigate the potential cost of error for the action. Even in the reactive realm, the degree of initiative can vary based on the amount that the user needs to maintain ongoing control and oversight of an action in progress.

Ju discusses 3 implicit interaction techniques:

• user reflection
• system demonstration
• override

User reflection is how the system indicates what it feels the users are doing or would like to have done.

A good example are modern spell-checking programs. Early versions had to be invoked explicitly, and engaged the user in an explicit dialog about potentially misspelled words to repair. Current spell-checking programs run continuously in the background allowing users to more easily notice potential errors. The implicit alert of this interaction is far more seamless than that of earlier spell-check programs. A similar example is the continous compilation used by modern IDEs such as Eclipse and Visual Studio. Earlier programs only showed compile errors when the user explicitly activated the compile command.

System demonstration is how the system shows the user what it is doing or what it is going to do.

In Range, the whiteboard animates its transition from ambient display mode (where it displays a set of images related to the workspace) to drawing mode (where users can make sketches and diagrams) as a demonstration-of-action that calls more attention to the mode change than a sudden switch would, and provides a handle for override (see later).

Override techniques allow users to repair misinterpretation of the user’s state or to interrupt or stop the system from engaging in proactive action.

This usually occurs after one of the previous techniques (user reflection or system demonstration) alert the user to some inference or action that is undesirable. Override is distinct from “undo” because it is targeted at countering the action of the system rather than reverting a command by the user.

An example of override in Range is that in the transition between modes users are able to “grab” digital content to use it as part of the whiteboard contents, or to stop the motion of objects that are being moved to make space for drawings.

The main contribution of this framework compared to prior models for implicit interaction lies in the key variable of initiative. Without this variable, it would not be possible to distinguish user reflection techniques from system demonstration techniques or to map the role of override.

In conclusion, a very interesting paper that offers a framework to reason about proactive user interfaces and make sure that users are always in control.

One of the samples is the script inquiry.py which allowed me to detect nearby devices in just a few lines of code. Here it detected mine and Petr’s cell phones:

I had to position the camera a bit lower, because it would sometimes not recognize my face when I was looking at the bottom of my screen. Having a dual-monitor setup doesn’t help either, since you should ideally position the webcam in the middle of the two displays to have good coverage. I positioned the webcam in the middle of the two displays at the bottom and tilted it upwards, which gives good results:

I guesstimated that if the system couldn’t detect a face for a period of 10 seconds, I would be away from my desk (or at least not paying attention to the screen). I played around a bit with pygtk and notify-python to create a status icon and show notification bubbles whenever my status changed. Here’s what happens when it detects that I’m away from my desk (notice the bubble in the upper right corner):

When I return, the system will change its status icon accordingly and notify me again:

OpenCV is fun!

I didn’t know where to start, so I searched for existing OpenCV face detection examples. I found a blog post by Nirav Patel explaining how to use OpenCV’s official Python bindings to perform face detection. Nirav will be working on a webcam module for Pygame for the Google Summer of Code.

I managed to rewrite Nirav’s example to get it working with CVtypes:

Here’s the code. Although it’s just a quick and dirty hack, it might be useful to others. It requires CVtypes and OpenCV, and was tested on Ubuntu Hardy with a Logitech QuickCam Communicate Deluxe webcam. You will need Nirav’s Haar cascade file as well.

import sys
from CVtypes import cv
 
def detect(image):
    image_size = cv.GetSize(image)
 
    # create grayscale version
    grayscale = cv.CreateImage(image_size, 8, 1)
    cv.CvtColor(image, grayscale, cv.BGR2GRAY)
 
    # create storage
    storage = cv.CreateMemStorage(0)
    cv.ClearMemStorage(storage)
 
    # equalize histogram
    cv.EqualizeHist(grayscale, grayscale)
 
    # detect objects
    cascade = cv.LoadHaarClassifierCascade('haarcascade_frontalface_alt.xml', cv.Size(1,1))
    faces = cv.HaarDetectObjects(grayscale, cascade, storage, 1.2, 2, cv.HAAR_DO_CANNY_PRUNING, cv.Size(50, 50))
 
    if faces:
        print 'face detected!'
        for i in faces:
            cv.Rectangle(image, cv.Point( int(i.x), int(i.y)),
                         cv.Point(int(i.x + i.width), int(i.y + i.height)),
                         cv.RGB(0, 255, 0), 3, 8, 0)
 
if __name__ == "__main__":
    print "OpenCV version: %s (%d, %d, %d)" % (cv.VERSION,
                                               cv.MAJOR_VERSION,
                                               cv.MINOR_VERSION,
                                               cv.SUBMINOR_VERSION)
 
    print "Press ESC to exit ..."
 
    # create windows
    cv.NamedWindow('Camera', cv.WINDOW_AUTOSIZE)
 
    # create capture device
    device = 0 # assume we want first device
    capture = cv.CreateCameraCapture(0)
    cv.SetCaptureProperty(capture, cv.CAP_PROP_FRAME_WIDTH, 640)
    cv.SetCaptureProperty(capture, cv.CAP_PROP_FRAME_HEIGHT, 480)    
 
    # check if capture device is OK
    if not capture:
        print "Error opening capture device"
        sys.exit(1)
 
    while 1:
        # do forever
 
        # capture the current frame
        frame = cv.QueryFrame(capture)
        if frame is None:
            break
 
        # mirror
        cv.Flip(frame, None, 1)
 
        # face detection
        detect(frame)
 
        # display webcam image
        cv.ShowImage('Camera', frame)
 
        # handle events
        k = cv.WaitKey(10)
 
        if k == 0x1b: # ESC
            print 'ESC pressed. Exiting ...'
            break

A known problem is that pressing the escape key doesn’t quit the program. Might be something wrong in my use of the cv.WaitKey function. Meanwhile you can just use Ctrl+C. All in all, the face detection works pretty well. It doesn’t recognize multiple faces yet, but that might be due to the training data. It would be interesting to experiment with OpenCV’s support for eye tracking in the future.

Update: the script does recognize multiple faces in a frame. Yesterday when Alex stood at my desk, it recognized his face as well. I think it didn’t work before because I used cv.Size(100, 100) for the last parameter of cv.HaarDetectObjects instead of cv.Size(50, 50). This parameter indicates the minimum face size (in pixels). When people were standing around my desk, they were usually farther away from the camera. Their face was then probably smaller than 100×100 pixels.

Just a quick note on ctypes. I remember when I created PydgetRFID that I tried to use libphidgets’ SWIG-generated Python bindings, but couldn’t get them to work properly. I had read about ctypes, and decided to use it for creating my own wrapper around libphidgets. Within a few hours I had a working prototype. When you’re struggling with SWIG-generated Python bindings, or have some C library without bindings that you would like to use, give ctypes a try. Gary Bishop wrote about a couple of interesting ctypes tricks to make the process easier.

I was curious to see if things changed. It seemed that both systems have improved in speed, but there were no shocking results.

There are some differences between our experiments though. Jordan wonders how long it took for me to commit my changes after adding. In fact, I didn’t do that directly but did a diff before committing. I don’t know why, I think I just forgot to commit. This clearly showed the big difference between bzr and git before and after committing. Bzr was a lot faster than git before committing, while git outperforms bzr after committing. I guess this is because git makes heavy use of data structures created during a commit.

Bzr took almost 4 minutes to diff after committing, while git only took about a second. This clearly showed a problem with bzr, it started walking the entire tree again, while git could immediately see that nothing changed. Bzr exacerbated the same problem when performing a status, it was unnecessarily slow. Jordan’s tests showed that the status problem was solved, but doing a diff when nothing has changed is still a major performance issue on big trees.

Committing a small change was a lot faster for both git and bzr. And it should! This is after all a really common operation.

Jordan did another comparison with bzr, git and hg, which showed that hg is actually fairly close to git in terms of performance.

I liked it very much: the conference had good quality papers but was still reasonably small (around 150 attendants), and of course the weather and the Italian food were great We arrived on Tuesday which gave us some time to explore the city and take the ferry to Capri (a great suggestion by Robbie).

I am not going to discuss the conference program into detail this time, but will just highlight a couple of interesting papers. Possibly one of the coolest papers was “Exploring Video Streams using Slit-Tear Visualizations” by Anthony Tang (video). Another presentation I enjoyed was “TapTap and MagStick: Improving One-Handed Target Acquisition on Small Touch-screens” by Anne Roudaut (video). It seems there is lots of related work in this area (e.g. Shift, ThumbSpace, etc.). Peter Brandl presented two interesting papers: Bridging the Gap between Real Printouts and Digital Whiteboards and Combining and Measuring the Benefits of Bimanual Pen and Direct-Touch Interaction on Horizontal Interfaces. He was brave enough to do an impressive live demo for the first paper Oh, and he also covered the conference in a blog post.

The first paper in our session, titled “A Mixed-Fidelity Prototyping Tool for Mobile Devices” by Marco de Sá, introduced a tool to easily design prototypes and evaluate them in real-life situations. The system was well thought out and serves a real need. I can imagine that we could use this kind of tool in a user-centered UI design course. The second paper in our session was “Model-based Layout Generation” by Sebastian Feuerstack. I already met Sebastian at CADUI 2006. They presented a generic layout model based on constraints. It reminded me a bit of the layout model Yves worked on for our EIS 2007 paper. They used the Cassowary constraint solver, which I also used for my MSc thesis on constraint-based layouts for UIML. Sebastian told me he got the idea from my demo at CADUI 2006. I forgot to add a certain constraint (the layout of the UI was thus underconstrained), which by coincidence had no effect on the user interface everytime I tested it. Of course, when I showed the demo it did have an effect This clearly illustrated that constraint solvers are sometimes unpredictable (see Past, present, and future of user interface software tools by Myers et al.). Sebastian’s solution to this problem was to hide the constraints from the designer and generate them automatically from a graphical layout model.

Juan Manuel Gonzalez Calleros — who I met at CADUI 2006, TAMODIA 2006 and a few other occasions — presented a poster and a paper at the workshop on haptics. He took a few pictures while Jan was presenting (thanks again Juan!). Here are Juan and Jan discussing UsiXML vs UIML

Overall, the comments on our work were positive, although of course one of the biggest problems is still the lack of support for multi-screen interfaces. As Jan is actively hacking on Gummy these days, I don’t think it will take very long for this to be included in the tool

So here’s a short demo video of a Solitaire game running in a Smalltalk environment (via David Buck). It clearly illustrates features such as full introspection (e.g. by using the object browser) and live “fix-and-continue” debugging:

I found another video showing live code updates in Smalltalk while invoking native libraries in the background (more specifically, OpenGL):

I think these videos are useful for demonstrating the power of Smalltalk’s environment. On a side note, I discovered Objective-C supports some form of “fix-and-continue” debugging as well.

In my personal opinion, this video (or at least the message that Dan tries to communicate) should be better integrated in OOP courses at universities. Another invaluable resource for grasping these ideas is Design Principles Behind Smalltalk, again by Dan Ingalls. Of course, it’s difficult to understand what OOP is about if you have to learn it through a weak implementation. We learned the basics of OOP in C++ for example, which would be blasphemy to Alan Kay He once said Actually I made up the term object-oriented, and I can tell you I did not have C++ in mind. Here’s his definition of OOP:

OOP to me means only messaging, local retention and protection and hiding of state-process, and extreme late-binding of all things. It can be done in Smalltalk and in LISP. There are possibly other systems in which this is possible, but I’m not aware of them.

Before I looked into Smalltalk, to my understanding objects just contained a bunch of methods or functions that had access to the object’s context. I did not really grasp the idea that objects just respond to messages (or method calls in my definition). The real difference about this is that in Smalltalk messages are dynamically dispatched at runtime. A method is the function or subroutine that is invoked in response to the sending of a message, which will be matched to the message name (or selector) at runtime. In contrast, method calls in C++, Java and C# are statically bound at compile-time. There is thus a distinction between the semantics (or message) and implementation strategy (or method) in Smalltalk. Decoupling these allows for more flexibility, such as objects that cache all incoming messages until their database connection is fully set up, after which they replay these messages, or objects that forward messages to other objects (which might have even been passed in at runtime). This is of one of the aspects of extreme late binding in Alan Kay’s definition of OOP.

It’s exactly this run-time lookup of methods that enables effortless polymorphism. As explained in the video, at some point the intermediate factorial result will become an instance of LargeInteger, while in previous iterations it was an instance of SmallInteger. The multiplication message (*) is sent to this object, after which the correct method in the class LargeInteger is looked up for handling the message, allowing the existing code to continue to work. Java, C# and C++ have all inherited this feature (although C++ requires explicitly declaring methods as virtual for this to work, due to efficiency reasons). Smalltalk can even realize polymorphism without inheritance (also known as duck typing), although this is not shown in this video. Smalltalk has implicit interfaces: an object’s interface is the messages it responds to. If two objects both respond to a certain message, they are interchangeable (even at runtime). Traditional languages such as Java or C++ only support inheritance-based polymorphism (although something similar to duck typing can be achieved with C++ templates). Here’s the explanation by Dan Ingalls:

Polymorphism: A program should specify only the behavior of objects, not their representation.

A conventional statement of this principle is that a program should never declare that a given object is a SmallInteger or a LargeInteger, but only that it responds to integer protocol. Such generic description is crucial to models of the real world. Consider an automobile traffic simulation. Many procedures in such a system will refer to the various vehicles involved. Suppose one wished to add, say, a street sweeper. Substantial amounts of computation (in the form of recompiling) and possible errors would be involved in making this simple extension if the code depended on the objects it manipulates. The message interface establishes an ideal framework for such an extension. Provided that street sweepers support the same protocol as all other vehicles, no changes are needed to include them in the simulation:

More details on the differences between Smalltalk and current OOP languages are explained in Smalltalk: Getting The Message. I believe that understanding the original philosophy behind OOP helps you be a better object-oriented programmer in any language. Ramon Leon discusses the common mistake of magic objects which is an interesting read.

But let’s get to the point of why I started looking into Smalltalk again. At the moment, I mostly program in C# (and sometimes in Java), but I often feel frustrated with both languages. After being exposed to Ruby and Python, I feel like static typing requires me to write too much code and helps the compiler more than it helps me. Furthermore, Java seems to be overly engineered with all the factories, manager, readers and writers, while C# is often inconsistent or lacking in its implementation (e.g. anonymous methods are not really closures). Both languages are becoming increasingly complex with the addition of more and more features. Generics for example is just not necessary in a dynamically typed language. The problem with scripting languages such as Ruby and Python however, is that they are often interpreted and slow. I experimented a bit with JRuby (a Ruby implementation in Java with full access to Java’s class library) but that didn’t satisfy my needs either. After trying to code a simple Hello World Swing application in JRuby, I was stunned that it still required me to wrap code inside an ActionListener like Java does, while I really just wanted to pass in a Ruby block.

Update:: Nick Sieger pointed out that a newer version of JRuby does allow blocks to be passed in.

Other people have also been struggling with languages such as Java or C# (e.g. Jamie Zawinski, Mark Miller and Steve Yegge) or are looking for alternatives (e.g. Martin Fowler and Tim Bray). I think the popularity of Ruby might motivate more people to have a look at Smalltalk. Furthermore, if you know Ruby, it’s easier to get acquainted with Smalltalk. Besides lots of similarities in the class library (the Kernel class, the times message on numbers, etc.), Ruby already introduces the notion that everything is an object, objects in Ruby communicate through messages and Ruby has blocks. However, Ruby is not really equivalent to Smalltalk yet. Ruby introduced extra syntax to be more familiar to people that were used to C-style programming languages, thereby losing part of Smalltalk’s flexibility. In fact, the beauty of Smalltalk is that its entire syntax easily fits on a postcard. If you look closely at this example, even a conditional test in Smalltalk is implemented using messaging on objects. You just send the message ifFalse to an instance of the class Boolean, and pass in a code block you want to have executed when the value is false. It’s turtles all the way down.

Another problem I came across when developing in Java or C# (or in any other OOP language I used) was the difficulty of changing class hierarchies. Very often, due to time constraints, a design is just left in its original state, and the new requirements are supported by performing a quick hack. I suspect this problem is especially prevalent in so-called “research code” It gets even worse when programming in teams. Although this problem is generally known in software engineering and several strategies have been proposed to deal with it, I wondered why the promise of OOP failed here. Wasn’t OOP supposed to improve the situation and make spaghetti code obsolete?

Jeffrey Massung asked himself a similar question: What if the philosophy (OOP) wasn’t the problem, but the implementation (language) was?, and decided to write a 2D DirectX game in Smalltalk. It seems Smalltalk did indeed allow for easier design changes. Self (a language derived from Smalltalk) tries to alleviate the aforementioned problem by specializing through cloning of existing objects instead of through class hierarchies. It’s funny to note that the problem wasn’t that bad in Smalltalk, since you could still easily change the hierarchy, unlike in languages such as Java or C++.

The real power of Smalltalk is not its syntax, but the entire environment. I believe this is also key to understanding OOP. The current languages and tools (e.g. IDEs) we use for doing object-oriented programming are just weak implementations of the original Smalltalk environment. When working in Smalltalk, you are working in a world of running objects, there are no files or applications, everything is an object. For example, version control systems in Smalltalk are actually aware of the semantics of your code, they are not just text-based. When merging code they can show you what methods have been changed, added or removed, what classes were changed, allow you to decide which changes you want to keep, etc.. Although I think Bazaar is a great, it doesn’t come close to this way of working. Smalltalk allows live debugging and code changes, which is tremendeously useful. Ever wished that you could fix a problem while you’re debugging and immediately check if your solution works without having to recompile your application and start the entire process again? In Smalltalk (and Lisp) that’s possible. If you want to find out more about why Smalltalk is way ahead of current mainstream OOP languages, have a look at Ramon Leon’s Why Smalltalk.

Update:: Scott Lewis commented that I should have emphasized that Smalltalk is mostly written in Smalltalk: So when you subclass any object, you can go back up the chain of inherited objects and see how everything works. Likewise when you hit an error/bug, the debugger lets you delve about as deeply as you could possibly want into what is going wrong, and why it is an error. This is indeed a powerful aspect of Smalltalk, and an example of how it was influenced by Lisp.

Besides reading about Smalltalk, I have also been experimenting a bit with Squeak. Squeak is an open source implementation of the Smalltalk programming language and environment, created by its original designers. Squeak runs bit-identical on many platforms (including Windows CE/PocketPC). I will leave my Squeak experiments for another blog post though

To conclude, it seems that we are very good at ignoring the past. We just take our current systems for granted, and use them as a reference frame for future innovations. Marshall McLuhan once phrased it like this: We drive into the future using only our rearview mirror. I believe this is true in HCI research as well, as people like Dan Olsen have pointed out. He argued that our existing system models are barriers to the inclusion of many of the interactive techniques that have been developed. He gave the example of the recent surge in vision-based systems and multi-touch input devices, which get forced in a standard mouse point model because that is all that our systems support:

Multiple input points and multiple users are all discarded when compressing everything into the mouse/keyboard input model. Lots of good research into input techniques will never be deployed until better systems models are created to unify these techniques for application developers.

Research on toolkits is a lot less popular these days. We try to map everything into existing models, and always feel like we have to support legacy applications, which hampers significant progress. Bill Buxton has also studied innovation in HCI, and questioned the progress we made in the last 20 years.

I think the reason why so many great work was done by the early researchers in our field (e.g. Ivan Sutherland, Douglas Engelbart and Alan Kay) is — besides that they were very creative and intelligent people — that there was not that much previous work, they just had to start from scratch. Alan Kay once asked Ivan Sutherland how it was possible that he had invented computer graphics, done the first object oriented software system and the first real time constraint solver all by himself in one year, after which Sutherland responded I didn’t know it was hard.

Here’s part of the product description:

Office OneNote 2007 is a digital notebook that provides people one place to gather their notes and information, powerful search to find what they are looking for quickly, and easy-to-use shared notebooks so that they can manage information overload and work together more effectively.

I have been using a combination of Gmail, Google Calendar, Google Docs, Google Notebook and del.icio.us to organize (and capture) information (partially inspired by this setup). The big problem here was synchronization. I ended up copying URLs to Google Notebook since I would never be confronted with them again if I stored them in del.icio.us. There were no explicit links between meetings, documents, and other resources (e.g. websites or short notes in notebooks). I would add gadgets for each of these apps to my iGoogle page to keep an overview. Although I could cope with this setup, it was not ideal. Gmail and Google Calendar are great services which I still love to use, but for quick notes and jotting down ideas I often resorted to paper notes.

Although OneNote is not perfect either, combined with a laptop (or tablet PC) it has the potential to eliminate most paper note taking. To get a good overview of what’s possible with OneNote, have a look at these resources:

• Max Zuckerman’s OneNote: the Mega-Tool for Learning
• Dian Chapman’s OneNote Introduction
• the OneNote 2007 Guide if you have OneNote installed on your PC.

A lot of people have been impressed with OneNote and have blogged about it.

Here are a few of OneNote’s features that I like:

• text search in images and audio
• audio and video recording with synchronized notes
• shared notebooks
• embedding any file as a printout
• screen clippings
• the ability to write and draw anywhere on a page
• tags (e.g. todo, important, question, etc.)
• calculator support
• inking support (if only I had a tablet PC)

Here’s an example of how I used OneNote to summarize an intuitive explanation of Bayesian Reasoning by Eliezer Yudkowsky:

Here’s the first version (June 2007):

A version with roughly the same UI but lots of architectural improvements (November 2007):

The current version with an improved UI (March 2008):

I think the most significant improvement is the new toolbox. Designers can now more easily distinguish between different widgets. In previous versions, some widgets were hard to distinguish. We added labels to each widget in the toolbox, and improved the inline rendering of the widgets. Notice that these images are still not predefined icons, but real widgets rendered to a bitmap. However, in the current version we chose to render them at their optimal size and then scale the images down.

Rendering widgets to bitmaps also forced us to migrate to Windows. We started our development on GNU/Linux using Mono but had to switch due to an annoying bug in Mono’s implementation of Control.DrawToBitmap. Future versions should be able to run in Mono again when this bug is fixed though.

Update: Jan sent me a screenshot of an even older version of Gummy, dated back to December 19, 2006: