Is that supposed to go there? A blog by olizilla

Scala's polite compiler fictions & the surprising difficulties of asking an AnyVal it's MaxValue

2011-02-19T00:00:00+00:00

Scala’s polite compiler fictions & the surprising difficulties of asking an AnyVal it’s MaxValue

In day to day programming I have found Scala allows me to make my code clearer and significantly more fun to write.

Yet somehow a simple code demo to get the numeric types to print their max and min values turned into a multi-man-hour-brain-teaser. That’s not something I want from my tools, so what follows is the highlights of the ensuing research, presented here so that Scaloids of the future may one day unearth these findings and laugh at the trifling matter that once troubled us. Some class names may have been changed, to protect the innocent, and make me laugh.

“Hey, how do I write a method to print out the max and min value of Byte, Short, Int, Long…”

“hmm, that should be pretty simple, let’s take a look at what you got.”

object NumberWang {
  def main(args: Array[String]) {
    println("Max Byte:  " + Byte.MaxValue )
    println("Max Short: " + Short.MaxValue)
    println("Max Int:   " + Int.MaxValue  ) 
    println("Max Long:  " + Long.MaxValue )
  }
}

Output

scala> NumberWang.main(null)                               
Max Byte:  127
Max Short: 32767
Max Int:   2147483647
Max Long:  9223372036854775807

“All good so far, but how would we write a method to get rid of all the repeated code?”

“hmm. Let’s have a go”

scala> def maxor(numericType: AnyVal) {
     |   println(numericType.MaxValue)
     | }
 <console>:6: error: value MaxValue is not a member of AnyVal
          println(numericType.MaxValue)
                              ^

Rats. I’d assumed something like MaxValue would be common to all the AnyVals. Compiler says no. Let’s actually find out where MaxValue is defined. And here lies the first surprising difficulty:

Int.MaxValue does not currently appear anywhere in the ScalaDoc API.

That’s weird, since it’s now the recommended way of getting that information

Hmm, so where is MaxValue defined?

Well, from first principles, when we say Int.MaxValue in Scala we are talking about a member of the Int companion object rather than a static member of the Int class as would be the case in Java. So let’s take a look at the docs for that:

Int
object Int extends AnyValCompanion
A object representing 'object scala.Int'. It should never be used directly

Value Members
	def toString () : String
		Returns a string representation of the object.

http://www.scala-lang.org/api/current/scala/runtime/Int$.html

Great. toString and a bizzarre note that this object “Should never be used directly”. I’m gonna put that into the pile marked “misleading documentations I have known”, and carry on. Not that this alley gets us very far, as apparently Int.MaxValue is not a notable enough public member to warrant documenting. Putting the frustration back in the box, this sort of documentation mishap often boils down to refactorings and communication issues, let’s see if we can’t get some background on this troublesome value…

object Math
object Math extends MathCommon
deprecated: use scala.math package instead

val MAX_INT : Int
deprecated: Use scala.Int.MaxValue instead

http://www.scala-lang.org/api/current/scala/Math$.html

Ok, so we can see that there has been some noodling around where best to put the max value information, this would explain some of the weirdness. More interestingly, Scala is your flexible friend, so what can we do about it? Let’s have another go at that maxor function, this time with feeling, and maybe some thinking too.

What do we want?

A function to takes one of Byte, Short, Int, Long and maybe even Float and Double too, as a parameter.
Returns unit and has the side effect of printing the largest number that numerical type can represent.

Obviously if this were part of a larger system, we would make a proper function that returned a String rather than causing side-effects, but right now that’s the least of our troubles.

So why is this even remotely difficult?

Well, we’d like to pass in a variety of AnyValCompanion objects (Int, Byte, etc) and then call a MaxValue function on that object. We know each one implements MaxValue but it isn’t defined as part of the AnyValCompanion object contract.

hmm…

Stand back, I am going to try duck-typing

scala> def maxDuck( numericType: { def MaxValue: AnyVal } ) {                       
      |  println( numericType.MaxValue )
      |}
maxDuck: (numericType: AnyRef{def MaxValue: AnyVal})Unit

scala> maxDuck( Int )
	java.lang.NoSuchMethodException: scala.runtime.Int$.MaxValue()
	at java.lang.Class.getMethod(Class.java:1605)
	at .reflMethod$Method1(<console>:6)
	at .maxDuck(<console>:6)

Kabooom. Hmm… that should have worked! Is this a sign from the capacitors that duck typing is evil and should be avoided? No, and before you all shout at me, yes, I know this isn’t really duck typing but if you find a simple way to explain what Scala is up to there, do let me know. I read it as “the parameter to maxDuck can be any object that has a member called MaxValue that returns an AnyVal”, and seeing as we only want to call toString on the result of MaxValue, that seems to capture our requirements.

The MaxValue member that we are interested in has a different signature on each numeric type; Int’s returns an Int, and Byte’s returns a Byte. More importantly MaxValue is common to all the numeric types but isn’t defined on any of their common parents or traits, so we went duck style, and so maxDuck says: if it quacks like something with a MaxValue, then that’s good enough for us. I found some other dreadnought weight solutions, involving creating a type for each numeric type, but that kind of boilerplate is exactly what I came to Scala to avoid.

So let’s press on. Our error appears to be telling us that there isn’t a MaxValue() method defined on scala.runtime.Int$. The Int$ tells us it is trying to look it up on the companion object, but I thought we already proved in the first few lines that it does exist, even if undocumented. And so to surprising difficulty #2:

"The AnyVal companion classes are are added synthetically by the compiler"

http://www.scala-lang.org/node/6650

I read this as, N.B. here be beasties, take care, and a sword. But then I do have a tendency to see patterns in noise. Let’s see if we can prove that it’s compiler voodoo at the root of all this:

scala> def max(maxable: {def MaxValue: Any}) { println( maxable.MaxValue ) }
max: (maxable: AnyRef{def MaxValue: Any})Unit

scala> object Lung { val MaxValue: Long = 1046 }
defined module Lung

scala> object Boot { val MaxValue: Byte = 8 }   
defined module Boot

scala> max( Boot )
8

scala> max( Lung )
1046

It works! Ha! In your face compiler. You have been caught out in a lie, or, performing “polite compiler fictions” as mentioned in the wonderfully evocative Scala commit msg that ends this reign of voodoo.

root/scala/trunk/src/library/scala/Byte.scala

Revision 24068, 5.7 KB (checked in by extempore, 4 weeks ago) The AnyVal? types become source files instead of polite compiler fictions.

!! You’ll need a serious “ant all.clean” now. !!

As of this commit the system is fully bootstrapped and the synthetic code eliminated: only the source files remain. The sort-of-AnyVal?-companions in scala.runtime.* have all been eliminated because the actual companions can do everything; deprecated vals in the scala.runtime package object point to the companions. This left AnyValCompanion? as the only AnyVal? related thing in the runtime package: that made little sense, so I deprecated and moved it as well.

Starr is based on r24066 plus this commit. Closes #4121. Review by rytz, odersky.

https://lampsvn.epfl.ch/trac/scala/browser/scala/trunk/src//library/scala/Byte.scala

Hat’s off to extempore for providing the fix, and introducing me to the phrase “polite compiler fictions”. I for one welcome our new French speaking overlords.

Conclusions

Scala is ace, written by poetic wizards.
Compiler magic leads to nuances, nuances lead to frustration, and blog posts.
A function that prints the MaxValue of anything with a MaxValue can be simply expressed, but on Scala 2.8.1, it doen’t work for Int, Byte and co. due to a social faux-pas by an otherwise polite compiler.
It has been noted and fixed by the aforementioned wizards.

Credits and links

Thanks to Martin & Cemal for finding the issue and discussing the solutions.
Thanks to extempore for services to both computer and human language.
Thanks to @_alanshaw & @davidparry for pulling typos and patterns out of this noise.
http://www.scala-lang.org/node/6650
http://www.scala-lang.org/node/739
http://www.scala-lang.org/node/136
https://lampsvn.epfl.ch/trac/scala/browser/scala/trunk/src//library/scala/Byte.scala
http://www.scala-lang.org/api/current/scala/runtime/Int$.html

Git on Lenny, a love story

2010-12-07T00:00:00+00:00

Git on Lenny, a love story

“You are running Debian stable, because you prefer the Debian stable tree. It runs great, there is just one problem: the software is a little bit outdated compared to other distributions.”

http://backports.debian.org/

Exactly…

So I have Debian Lenny running nicely as a production server. Along comes git. A long period of flirtation follows and eventually I find myself calling her everyday. Time to make a real git of her and get her into the workflow. So I save up 3 months wages, get down on one knee and roll out a:

# DONT DO THIS! Skip to the end if you prefer facts to love and stories.
$ sudo aptitude update
$ sudo aptitude install git
... pregnant pause while internets are downloaded
... Done

So I’m all happy as to my eyes that’s a yes, but then things get weird. She pulls off her face and reveals that she is infact a transitional dummy package to pull in the renamed gnuit package. It can be safely removed.

What!? Please expand on that you freak…

$ aptiude show gnuit
Package: gnuit
...
Provides: git
Description: GNU Interactive Tools, a file browser/viewer and process viewer/killer

Erm… hold it right there. You are not the git I love. Git would never claim to be a file viewer killer like that. It would say something more sexually aggressive like fast, scalable, distributed revision control system or something. I hate you fakey git imposter. Be gone from this box:

$ sudo aptitude purge git

OK now think fast dammit, where did you see her last.

$ aptitude search git
git-core - fast, scalable, distributed revision control system

A-HA!, git-core!, well of course I love your new hair-do, I just didn’t recognise you, what with getting your name changed and all, come here:

$ sudo aptitude install git-core
... pregnant pause while internets are downloaded
... Done

thank god it’s you. Still I hope you don’t mind but I’d just like to take a quick look at your birth certificate over here…

$ git --version
git version 1.5.6.5

NO NO NO. If there is one thing worse than installing completely unrelated packges, it’s finding the package you want is older than your nan.

$ sudo aptiude purge git-core

Oh wrinkly torment I’ve just purged your grandmother. You were a youthful git 1.7 when we last met, what have I done to deserve this… Googling commences, hmm debian sources… yes yes, git-core, uh huh, (obsolete), oh what have i done, stackoverflow… git-core… not you again, …wait… what… backports you say? install lenny-backports-keyring you say? that all sounds a bit extreme, I just want git… hyperlinking now…

backports.debian.org is where it’s at! To complete the quote at the top:

”…the software is a little bit outdated compared to other distributions. This is where backports come in.

Backports are recompiled packages from testing (mostly) and unstable (in a few cases only, e.g. security updates) in a stable environment so that they will run without new libraries (whenever it is possible) on a Debian stable distribution.”

Go there and be sure to check the instructions page which is a couple of steps simpler than the stackoverflow suggestion, or just read on…

Installing git 1.7 rather than gnuit or git 1.5 on Debian Lenny

In /etc/apt/sources.list add

deb http://backports.debian.org/debian-backports lenny-backports main

Update your apt indexes

$sudo apt-get update

Check you can now see the latest git version. The lesson from all that rambling above is to always ask apt what it is you are about to install before doing it.

$ aptitude -t lenny-backports show git
Package: git
State: not installed
Version: 1:1.7.1-1.1~bpo50+1
Description: fast, scalable, distributed revision control system

Go install!

$ sudo aptitude -t lenny-backports install git
... Done
$ git --version
git version 1.7.1

Love, git init your brains out and have git babies. See below:

Now I can get on with that web focused git workflow. If you are wondering how to make git do all your work for you then I highly recommend Mr joemaller’s article on the subject.

Credits

stackoverflow: http://serverfault.com/questions/157383/why-does-the-debian-lenny-git-package-not-install-git
backports: http://backports.debian.org/
workflow: http://joemaller.com/990/a-web-focused-git-workflow/

Web by Proxy; Secure surfing on someone else's SOCKS

2010-11-25T00:00:00+00:00

Web by Proxy; Secure surfing on someone else’s SOCKS

Today I found myself looking down the barrel of an unknown and unsecured wireless access point. Some kindly soul has gone to the trouble of geting an adsl line into Brixton Village Market and then gone the extra mile of leaving the wireless unsecured (no WAP or WEP password in sight) so anyone may connect, and I assume deliberately as the access point in question comes secured by default. (if not, sorry, I’ll buy you a federation coffee.)

Such a display of generosity warms the cockles, but alas, as a techno-noodler my thoughts immediately turned to the worrying, if over-simplified idea that connecting to such a device would force my laptop to start publicly smearing the contents of my browser, every request, url and search term, over everything in a several meter radius.

When you connect to a secured wireless access point, your communications with that device are encrypted, hindering the casual eavesdropper. When it’s not secured, then you are broadcasting your communications in clear text, a bit like shouting your banking details down the telephone at the call center operator whilst sitting on a bus. Then there is the question of whether to trust the access point itself. What if you keep your voice down but the call center operator goes on to use your details to maliciously check your balance‽

Still the technoodler can only see this as an opportunity to put those funix lessons to the test, so here is what I did to keep my business secure on an unknown and unsecured access point:

Encrypting browser traffic using only an SSH tunnel, a trusted remote server, Firefox, OSX, Terminal.app, a Goat & some Googling.

As a control test, check your current public ip, probably that of the wireless point, point your browser to: ipgoat.com the finest goat based ip service in the world. Go on, click the head.

open -a firefox http://ipgoat.com

Set up the tunnel from port 8000 on your machine to the trusted remote server:

ssh -CN -D 8000 user@remoteserver

…where:

C means compress things please, which may help speed up the network traffic where possible.
N means don’t do anything yet, just sit and wait.
D 8000 user@remoteserver is what tells ssh to listen on port 8000 and act as a SOCKS server forwarding requests via user@remoteserver.

…and coincidentally spelling out CND, providing you with a nuclear disarming mnemonic

Set up the Firefox to use the tunnel, as a SOCKS proxy connection:

Firefox 3.6 > Preferences… > Advanced > Network > Settings
Manual Proxy configuration:
SOCKS Host: localhost
Port:8000
Ok, done.

kick the tires:

open -a firefox http://ipgoat.com

If all is working then you should now find that ipgoat still loads and that you’ve fooled it’s little goaty face into thinking your public ip is that of your trusted remote server, mainly because to all intents an purposes, it is. You are now sending http requests to the remote, and it is kindly and blindly executing those requests on your behalf, and returning the results to port 8000 on your machine, all encrypted by your trusty SSH client.

I guess access to a trusted remote machine is the stumbling block for most people. In that case, you’ll have to take your chances, or keep googling. At the very least be sure to look out for the little https / SSL padlock in your browser when you have to enter any private information, but I’m pretty sure that facebook update can wait till you get home.

Credits

funix - darq.org.uk Free Unix Command Line Workshops every week in Shoreditch. “Using the ancient art of words we show you how to disregard the graphical interface, and do what you need from the command line.”
calomel.org The most relevant google search result told me exactly how to do it, and is https, which was nice.
commandlinefu.com A similar but more robust solution for bonus credits, but requires autossh, not on osx by default but only a: sudo port install autossh …away

Selenium And Castro Testing The Canvas

2010-11-05T00:00:00+00:00

Selenium and Castro testing the Canvas

Notes from Painless product demos & how to test ‘untestable’ applications. Original talk by Jason Huggins (@hugs) of Sauce Labs presenting at the London Software Craftsmanship Community, 4th Nov 2010.

The Selenium toolbox

Selenium IDE for Firefox - record and replay browser sessions. Use the output as a starting point for creating selenium tests.
Selenium Remote Control - an http proxy and library to drive a browser from test code
Selenium Grid - drive multiple Selenium RCs from a hub, to test multiple browser/os combinations. Run your tests in parallel rather than serially.
Sauce on demand - selenium testing as cloud service - http://saucelabs.com/ondemand

Painless product demos

AKA: Pilfering ideas for software development from filmmaking.

Dailies - The raw unedited footage, that shows general direction and progress from previous days work. Helps stakeholders seeking assurance that the work meets expectations.

In software dev, we have stakeholders too, and sometimes the html test reports aren’t quite enough to keep them happy. Wouldn’t it be nice if they could see our dailies, some artefact of our days work, so they can see for themselves that all is well, and those new features are on the way.

Selenium + Castro = Automated screencasts

Write your Selenium tests as normal, then bookend them with calls to Castro. Selenium will drive the browser through some interesting aspect of your web app, and Castro will record the screen as it goes. Run the test suite every day, and publish the generated screencasts along with the rest of your test reports. Anyone who wants to see how things are going now has a shiny video walkthrough of where the project is at.

c = Castro()
c.start()
// Your tests here
c.stop()
c.publishToStakeholders()  // ! tweet(), youtube(),

How to test ‘untestable’ applications

“Wouldn’t it be cool if you could motorise all the pins in a Pin Art box”

pinthing.com provides our example of the untestable web app. The bulk of the UI is created in a new fangled html5 canvas element using three.js to draw 3d pins on to it. The pins are arranged in a grid and can be given commands to raise and lower them independently to form patterns and glyphs.

All this UI magic is exposed as a single canvas node in the dom, hiding its internal structure. This poses a problem for automated testing; you cannot inspect and make assertions about the state of a canvas like you can with the dom.

Well, the theory is, the Javascript object model is good place to start (the JOM!? @hugs coins a new acronym). As you programatically build up your canvas you create a js structure that you can inspect, make assertions and even trigger interactions on.

pinthing has an in page js powered terminal that allows you to interact with the pins.

show(3)         // lifts pins 1 and 2, as its a binary display
show("3")       // lifts the pins to diplay a 3 glyph
show(3 << 5)    // bit shifing pin lifting, ye haa.

So in this case a Selenuim test can write expressions to the terminal, simulating user interactions.

HTML5 video players present a similar problem for testing. One solution already in use relies on exposing a js api to your player controls and instrumentation to fire events to the console during playback. This allows for Selenuim tests that trigger playback and then assert that the video started playing by listening for the events.

“It’s kinda like particle accelerators. You have to press the fire button and watch for the backsplatter to know you just smashed atoms”

It’s one step removed from actually being able to read and inspect the dom, but what else are you going to do? To continue automated UI testing at the frontiers of html5 you are going to have to think about exposing some sort of testable api to your new widgets. With a canvas tag built up with js you get something sensible for free. With a video player you are going to have to give it some more thought.

The union

Screencasting the automated testing of the untestable app.

In the pinthing.com example, having a visible in page terminal window is the key. It shows the watcher of the screencast what the Selenium test is up to; they can see the action and the result. The screencast becomes part of the test process, a QA person can scan through the video to check for UI anomalies.

Personally I love the idea. It’s not meant as a replacement for all the other reporting metrics, but a user friendly complementary document, that provides a background radiation style way to demonstrate progress to them that care to look. It depends on the project and the stakeholders involved how much value it adds, but automate it and dump it on your continuous integration server (thanks hudson) and it might just be the thing that keeps your investors investing, and the testers testing.

You also end up with an archive of screencasts for your project which you can turn into your blooper reel, complete with directors commentary after a release. Add dramatic music and playback the hi-speed dubbing version for a photo a day, watch how I change but stay the same, youtube favourite, style video…

Notes, Quotes, Links, & Thanks

A big thank you to @hugs for the talk and going above and beyond the call of duty in proof-reading this post. All quotes, ideas, and new acronyms mentioned here are his.
“Selnium 2, look out for a good beta by christmas.”
“There are 2 problems in the world… the off by 1 error…”
“Think of your user interface test recording like your apple commercial”
Flexpilot - integrates with selenium for flex testing.
http://saucelabs.com “Cloud-scale Selenium Testing.”
http://www.meetup.com/london-software-craftsmanship
http://github.com/hugs/castro “screen/cast ro/bot, a tiny fork of pyvnc2swf, with a smidge of awesome on the side”
http://pinthing.com “A @hugs creation, the culmination of a 10 year itch. A browser controlled pin art simulator with the power to move the real thing.”

Upgrade With Sbaz The Scala Bazaar System

2010-10-08T00:00:00+01:00

Upgrade with sbaz, the Scala Bazaar System

So you installed a Scala a while ago. Some shiny new features make you want more, but your install is just too old. Before reaching for the download button on the latest version, hear this!

A Scala installation can update itself

What’s that you say? I don’t have to scour some backwater of a recently re-branded site to find the sdk, with a side order of netbeans and open office bundle?

Oh me oh my. That’s nice. So a:

sbaz update

will update just the lists of what sbaz knows about, where as:

sbaz available

will show you what’s new since you last upgraded, and a simple:

sbaz upgrade

will upgrade you to the latest and greatest version, with all the shiniest features you can have.

Better still, you can ask sbaz to add all sorts of goodies to your vanilla Scala install. Take a closer look at the output of sbaz available:

sbaz available
...
scala-android (2.7.0-final, 2.6.1-final)
scala-cldc (2.7.0-final, 2.6.1-final)
scala-devel (2.8.0.final, 2.7.7.final, 2.7.6.final, ...)
scala-devel-docs (2.8.0.final, 2.7.7.final, 2.7.6.final, ...)
scala-documentation (2.8.0.final, 2.7.7.final, 2.7.6.final, ...)
scala-library (2.8.0.final, 2.7.7.final, 2.7.6.final, ...)
scala-msil (2.8.0.final, 2.7.7.final, 2.7.6.final, ...)
scala-swing (0.5, 0.4, 0.3, ...)
scala-test (2.8.0.final, 2.7.7.final, 2.7.6.final, ...)
scala-tool-support (2.8.0.final, 2.7.7.final, 2.7.6.final, ...)
scalacheck (1.7, 1.6, 1.5, ...)
...
56 package names
371 total packages

Of note:

scala-documentation providing some light reading and tutorials for the interested reader.
scala-android for all your mobile development needs.
scala-tool-support for syntax highlighting and other mods to trick out your text editor of choice.

Tell sbaz to go get them for you like so:

sbaz install scala-tool-support

For the other packages you can get some more info with a: sbaz show package-name

Of course no codes would be complete without some small piece of arcanea for you to forget, fortunately sbaz’s are pretty simple.

If you downloaded a Release Candidate you will always be offered the bleeding edge

So you got all excited about Scala 2.8 and downloaded 2.8.0.RC1, discovered the eclipse plugin still had serrated edges, and forgot about it. Now when you try a sbaz available it will list just the RC’s for future versions of Scala, very much the bleeding edge. You probably want the stable release for now. So, in the bombastic parlance of software we must “switch universe”, which contrary to it’s name actually boils down to swapping one path for another. No wormholes for you here.

Get Scala to update itself from release candidate to stable release

To point sbaz at the stable release list, we need to pass it the path to the poorly named “scala-dev” descriptor file which resides in:

misc/sbaz/descriptors/scala-dev

relative to your Scala install’s root directory. So navigate to the your Scala install dir, something like:

cd /usr/local/scala-2.8.0.RC1

Point sbaz at the universe for stable releases:

sbaz setuniverse misc/sbaz/descriptors/scala-dev

The path is relative to your install directory, so the above will fail if you are not in the root of your Scala install.

As a control test, show me what I have now:

sbaz installed

Give me the latest stable release:

sbaz upgrade

and for empirical evidence of success, show me what I have now:

sbaz installed

Kick back and wait for more shiney features.

watch sbaz available

or in my case, remember to go and rename that scala-2.8.0.RC1 directory to something more appropriate.

Get Scala to update itself to the bleeding edge

Should you yern for the wild west of release candidates you can flip to the lamp-rc universe for the bleeding edge releases. Navigate to the your Scala install dir, e.g /usr/local/scala-2.8.0.RC1

sbaz setuniverse misc/sbaz/descriptors/lamp-rc
sbaz upgrade

Sources

For a more authorative and intellectual discussion on sbaz please retune your dial to: http://www.Scala-lang.org/node/93

Arcade Fire Knows Where You Live

2010-10-06T00:00:00+01:00

Is that where you grew up?

http://www.thewildernessdowntown.com

Who knew pop-up windows could be so emotionally engaging.