Dave Briccetti’s Blog

AI Class Hidden Markov Chain Scala Solution

2011-11-22T00:34:00.000-08:00

I just took my Stanford Artificial Intelligence class midterm and got 98%. I missed one part of one problem. The problems we do are somewhat complicated and there are many opportunities to make mistakes. I was astounded that I got over 80%, doing all the problems by hand, with a calculator and a spreadsheet.

It took me many hours to review the material and take the test. I realized while taking the test that I should have encoded the procedures for solving the various types of problems (probability, Bayes Networks, Markov Decision Processes and the like) into computer programs. Then I could have simply run those programs given the data values in the test and produced very likely correct results. I will do this from now on. Let me start with a solution for solving a Hidden Markov Chain problem.

Given probabilities that a rainy day will follow a sunny day and vice versa, and the starting state, this program produces the correct probabilities that it will rain on the next three days.

Thanks for the Help Today, Indrajit (Silicon Valley Code Camp, Working on Lift)

2011-10-09T21:56:00.000-07:00

I spent the day today at Silicon Valley Code Camp with Lift developer Indrajit Raychaudhuri. First he helped me teach Kojo to a small group of kids. Then we did a talk together on Scala, Lift and TalkingPuffin Web. Then came the best part: We spent the next several hours as you see us here in the picture, with him teaching me how to be a Lift committer. As I took careful notes, he walked me through all the pulling, building, branching, pushing, ticket creating, diffing, and Review Board creating needed to make a change to Lift.

Interesting Scala/Kojo/Lift People Join Me at Silicon Valley Code Camp

2011-10-08T13:18:00.000-07:00

What fun it is to be in the Scala community, and to know many nice and interesting people!

At the last minute, while preparing for my Kojo talk for kids at Silicon Valley Code Camp, I emailed Kojo creator Lalit Pant to see if he wanted to say hello from India to my classroom of 4th to 12th graders by videoconference, and he wrote back with an enthusiastic “Yes!”

Once again, Ramnivas Laddad and his talented programming son Shadaj will help me teach the kids.

While planning my “Talking Puffin Web: A Scala/Lift Webapp” talk, I thought, why not ask super Lift committer Indrajit Raychaudhuri, who is in the Bay Area for JavaOne, to share the stage with me and introduce Scala and Lift. He was very pleased to accept!

Both classes are Sunday, October 9th, 2011.

Practicing Math (Regression Line) with Scala

2011-09-26T17:47:00.000-07:00

As you may know, I am taking Stanford’s online Artificial Intelligence Course taught by Sebastian Thrun and Peter Norvig. To prepare myself I am learning more math. The Khan Academy videos have been a huge help. I learned how to find the straight line that best fits a set of given points, and then played around with the calculation in Scala. I made a video showing what I did.

Visualizing List.scala’s Reverse Method

2011-09-16T17:19:00.000-07:00

I encountered a list.reverse in one of my Scala projects, and was curious as to how that method in the Scala library worked. So I opened the source code (seen at the top left of the picture) and worked through it in the REPL (Scala interactive environment), diagramming it as I went along. I thought it might be interesting to others.

Kojo Creator Lalit Pant Visited My Classes

2011-07-17T12:14:00.000-07:00

I started teaching with Kojo this summer, and it has gone very well. Kojo’s creator Lalit Pant was visiting the Bay Area, and he spent an afternoon with my grade 4–10 programming classes at DVC College for Kids. We published a two-part video of one of his presentations, which you can find in my Kojo playlist.

Using Implicit Conversions to Simplify Using Java APIs

2011-07-12T08:43:00.000-07:00

At last night’s Bay Area Scala Enthusiasts meeting, Dick Wall shared many excellent tips for effective Scala programming. One that especially interested me was using implicit conversions to simplify the use of Java API calls that require an object that implements an interface such as Runnable, with its single run method. In the Java way, one might write code like this:

new Timer().schedule(new TimerTask {
    def run() {
      println("Hello")
    }
}, 100L)

Wouldn’t it be nicer to supply the code for the TimerTask to run without wrapping it in a run method?

I wrote an example based on what I learned from Dick, and then Kris Nuttycombe suggested a refinement in response to a post of mine on Twitter.

Kojo (Scala), Python and Scratch for summer teaching

2011-05-17T00:50:00.000-07:00

This summer at Diablo Valley College College for Kids I will be teaching:

I am dropping Alice, at least for the time being. Its progress is glacially slow, and I wonder if version 3 will ever release (it’s been several years).

Being in the Scala community I have been aware of Kojo, but hadn’t made plans to teach it until my teaching assistant Ethan Kuefner said he really wanted to use it.

For the Python course I think we’ll drop Pygame, xturtle and the like, and concentrate on Python 3 basics, and steer those students interested in graphics to Kojo.

Scratch will continue as usual, along with GIMP, Google Sketch Up and Inkscape for graphics, and Audacity for sound generation.

TalkingPuffin Demo and Code Walkthrough of Link Indirection

2011-03-18T19:02:00.000-07:00

TalkingPuffin can automatically move past intermediate pages like DZone’s, taking you to the ultimate target of a link you find in a tweet. Here is a video demonstration of the feature, and a walkthrough of the Scala code that implements it.

TalkingPuffin Web Hashtags Analysis

2011-03-18T00:47:00.000-07:00

TalkingPuffin Web now shows hashtags and the number of times each occurs in the set of tweets analyzed.

Screencast from TalkingPuffin Talk at BASE

2011-03-15T02:58:00.000-07:00

We had a great session at Bay Area Scala Enthusiasts Monday. I presented TalkingPuffin Web, and got some great feedback and code improvements from the audience. I published an edited recording.

Showing TalkingPuffin Web at Bay Area Scala Enthusiasts

2011-03-13T20:58:00.000-07:00

On Monday, March 14th, I will show the Scala and Lift code behind TalkingPuffin Web (pictured), at Bay Area Scala Enthusiasts.

TalkingPuffin Version 1.0 Released

2011-03-06T19:22:00.000-08:00

Congratulations and thanks to everyone who has helped with the project. We have a 1.0 release of TalkingPuffin!

TalkingPuffin is an open source Twitter client written in the Scala programming language, and running on Java. The desktop application is built with Swing, and the Web application runs on Lift. Two of its strengths are filtering and user analysis.

New TalkingPuffin Web Feature: Tweets Timeline

2011-03-02T01:51:00.000-08:00

The new TalkingPuffin Web app is coming along nicely. The latest feature shows a user’s tweets on a timeline, and lets you move the mouse over them and read them. My motivation for creating the feature was to make it easier to decide whether I want to follow someone I discover. If their tweet density is not too high, and the content is good, I may follow.

The timeline is JavaScript based, using the Flot jQuery plugin. The Web app is created with Scala and Lift, and uses Lift’s Flot widget.

TalkingPuffin is now 6,827 lines of Scala code, in 134 source files.

TalkingPuffin Version 1.0 Release Candidate 1

2011-02-21T18:39:00.000-08:00

Two years ago I created a “Simple Twitter Client” (that was actually the name of it for awhile). I wanted a project for learning Scala, and I wanted to have a Twitter client that I could enhance any way I wanted. Along the way many people helped, and some even contributed code. Some of their Twitter names appear on Github with the source code.

At long last, I think this project is ready for release, so I now announce TalkingPuffin Version 1.0 Release Candidate 1.

TalkingPuffin Code Changes Overview for Twitter4J Conversion

2011-02-20T10:48:00.000-08:00

TalkingPuffin now uses Twitter4J to access the Twitter API. This video shows some of the coding changes I made in order to use it.

TalkingPuffin Coming Back After OAuth Conversion

2011-02-16T14:39:00.000-08:00

TalkingPuffin, the open source Twitter client written in Scala, is making its way back into service following a delayed conversion to OAuth (the new way that Twitter does client authentication).

Today we made Version 0.80d Development Version (not quite a release candidate) available online.

I am very excited to be using TalkingPuffin again, as those of you who follow @dcbriccetti may have already read. Despite its unrefined—some would say ugly—appearance, it is an extremely powerful tool for filtering and reading tweets in a variety of ways.

As always, I invite other interested Scala developers to contribute to the project, and again thank those now helping and those who have helped in the past.

Using the VI Java API from Scala

2011-01-24T21:46:00.000-08:00

My primary client these days is VMware. Recently my team has been using the VI Java API to fetch information about running virtual machines in vSphere. Steve Jin, the author of VI Java, asked me to write a “Hello VM” (sort of a “Hello World” for a virtualized environment) program in Scala.

I sent him some code and he wrote a blog post about it. You can find the code in his post. Here it is, formatted nicely. Some smart folks on the #scala channel on IRC reviewed it for me and made some great suggestions. One challenge was wrapping the searchManagedEntities call (line 31)—which can return either null, or an array of zero or more ManagedEntity elements—into something more Scala-like, a List of VirtualMachines.

Dice Throw Simulation in Java and Scala

2011-01-23T22:32:00.000-08:00

It’s fun from time to time to see what Java programs look like when converted to Scala. This program simulates throwing two six-sided dice 200 times, and creates a crude histogram of the results, like this:

2: XXXX
3: XXXXXXXXXX
4: XXXXXXXXXXXXXXXXXXXXXX
5: XXXXXXXXXXXXXXXXXXXXX
6: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
7: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
8: XXXXXXXXXXXXXXXXXXXXXXXXXX
9: XXXXXXXXXXXXXXXXXXXXX
10: XXXXXXXXXXXXXX
11: XXXXXXXXXXXX
12: XXXX

The Java version is 31 lines, and the Scala version, 18. Some things I notice that make the Scala version shorter:

Methods with only one line can skip the braces, and in fact, can take up just one line (lines 6, and 16–17).
The histogram bars can be created with the multiplication method (*) (line 17).

[Update: see the comments for much shorter versions of both the Java and the Scala versions.]

Monitoring Spam Control using Hyperic HQ

2011-01-03T12:09:00.000-08:00

I’ve stepped up my efforts recently against spam to my davebsoft.com domain. SpamAssassin does a good job of moving suspected spam to folders to review, but I have been finding hundreds of emails a day in these folders. So I am doing more blocking of spammers’ domains using Sendmail’s access.db.

Using Hyperic HQ

Hyperic HQ is a system monitoring tool. (The company that develops it is owned by VMware, a client.) I wrote a very simple plugin for it to monitor spam activity on davebsoft.com. Here is one of the displays.

RBL per minute is the number of emails blocked per minute using realtime blackhole lists. From my sendmail.mc:

FEATURE('dnsbl', 'zen.spamhaus.org')dnl
FEATURE('dnsbl', 'bl.spamcop.net')dnl

Denied per minute shows what I block in access.db. I have many lines like this:

hoghsluteo.com REJECT
guysenandi.com REJECT
elampupbid.com REJECT

Not shown in the display are probably-spam and maybe-spam, which are counts of messages SpamAssassin has found to be very likely and somewhat likely to be spam.

The Hyperic HQ Plugin

The plugin that enables this report consists of two files. This XML descriptor:

and this script:

spam-stats.sh

Finding Domains to Block

Once I collect some spam, I find the most offending domains and block them. I run this:

grep '^From: ' ~/imap/*-spam|cut -d'@' -f2|cut -d'>' -f1|sort|uniq -c|sort -r|head -15

which produces this:

6 shvigerebm.com
5 tramake.com
5 rainsha.com
5 klerebm.com
⋮

Six spams came from the nonsense domain shvigerebm.com, five came from the next several, and so on. I add these names to my Sendmail access file to block them.

Spam Situation Improved

Hyperic HQ, with my simple plugin, helps me monitor the spam activity on davebsoft.com, and actively blocking the most offending domains has greatly reduced the number of possible-spam messages to review every day.

Presenting at Silicon Valley Code Camp in October

2010-06-06T10:34:00.001-07:00

I have proposed three sessions for Silicon Valley Code Camp 2010. Two are about programming for kids, and one is about Scala and Lift.

Talk Slides for BirdShow: A Lift App for Showing Flickr Photos

2009-12-14T22:48:00.000-08:00

Good lightning talks at Bay Area Scala Enthusiasts, and a good presentation on Groovy from our friend Andres Almiray.

I have published my talk slides. [Corrected link]

BirdShow Scala Lift Application Lightning Talk at BASE

2009-12-13T16:32:00.000-08:00

For a lightning talk at Bay Area Scala Enthusiasts at Twitter HQ
Monday, I will show BirdShow, a Lift application that shows photos
from Flickr. Perhaps you’ll try it out and offer suggestions here:

http://groups.google.com/group/liftweb/browse_frm/thread/3c1493bd25f37b96

http://briccettiphoto.com
http://github.com/dcbriccetti/bird-show

Scala Higher-Order Functions Example with Swing

2009-12-01T20:54:00.000-08:00

Here is a trait I created that I was mixing in with java.awt.Components to see if they were “active.” These Components reside inside JInternalFrames or JFrames, and I really want to know if the frame ancestor is active. It’s a bit ugly, but I use SwingUtilities.getAncestorOfClass to find the ancestor of type JInternalFrame or JFrame, and then call the appropriate method, isSelected or isActive, to determine if the frame is active.

Now I not only want to see if a Component is active, but I want to set it active. So I want to add a new method, activate, to the trait. I will use the same approach with SwingUtilities.getAncestorOfClass, but I don’t want to duplicate the code. Here is the result.

I created a private method, doWithAncestor, which expects two functions as arguments. If the ancestor is a JInternalFrame, then doWithAncestor calls the doForInternalFrame function. Otherwise it calls the doForJFrame function.

doWithAncestor is parameterized in T. When called from isActive, which returns Boolean, T is Boolean. When called from activate, which returns Unit, T is Unit.

Filtering Twitter Noise Slides from SHDH 35

2009-11-07T23:52:00.000-08:00

Here are the slides from my Filtering Twitter Noise lightning talk at SuperHappyDevHouse.

Thanks, Palantir, for hosting us. I’ll repay them slightly by passing on that they are looking for good software engineers. I might be interested myself if they were doing Scala (they have looked into it a bit, and I think over time they might be persuaded).