Daniel O'Connor

Consuming GeoRSS and querying OpenStreetMap

2013-05-13T23:22:00.001+09:30

In Adelaide, there was an unexpected fire. It was a bushfire, and I'd just been mapping the area earlier that day - ironically, tracing some of the forests which were being burnt out at that very point in time.

I decided to go and see what was available for fire alerts around the nation.

It turns out the majority of fire authorities publish GeoRSS. I took the feeds and aggregated them, plus put in a transformation of the data to JSON.

Using Leaflet, I added a rendering of particular fires.

Using the OpenStreetMap Overpass API, I queried for "all areas in a 500m bounding box", and again using Leaflet, rendered those polygons onto a map.

Here's the Project:
https://github.com/CloCkWeRX/burning-down

Here's a few screenshots:

Coverage, for most of Australia. WA doesn't provide openly licensed data, and South Australia's does not contain lat/long info.
The few you see in those states are from Sentinel, a satellite based system for detecting 'hot spots' that may indicate a fire.

Here, a number of resident buildings are marked as 'at risk' from a fire. In reality, these are probably safe, due to fires in these areas tending to be contained to residential areas.

On the other hand, you can certainly consider adding an Open Street Bug report if someone's house may have burned down!

This is more interesting - there's a fire or other emergency right next to a national park. National parks are made from trees, which in my experience tend to catch on a whole lot of fire!

Obviously this sort of incident is well in hand if the fire authorities are aware of it, but it is interesting to see that fuel sources are being picked up.

My next efforts here are going to be around encouraging edits near the area - if there isn't much detail on water sources (dams, swimming pools, rivers, lakes); it certainly makes sense to identify those if there's already been a fire once.

I also made a bit of effort to aggregate the feed output into sqlite - I might go run up a basic server and run this as a service at some later point, as aggregating this data seems useful.

The thing I like most about this - it's trivial to take any GeoRSS feed and query OpenStreetMap for nearby things - I've done fairly wide ranging queries with a narrow bounding box so far; but those are easy enough things to tweak.

Neat ad-hoc networking trick

2013-05-13T19:05:00.002+09:30

I didn't realise it for a long time, but instead of relying on DNS or directly entering IP addresses, most machines in my home network actually self organise. Many of you will already know this, but for those that dont...

Most devices support a thing called Multicast DNS - apple devices also do this.

Here's the article:
http://en.wikipedia.org/wiki/.local

This is why things like Transmission, SickBeard, XBMC and SSH keys are useful: you can simply point to xbmc.local:8080 and get to a basic remote control.

This lets you simply refer to whatever the machine is and start talking to it. On a local wifi network, it's great.

Wish:
Ubuntu made this much clearer in some fashion - the first time you join a network, it should flash a welcome message. If it hears about a particular host, it should notify "xbmc.local is available" or similar, indicating a few machines it has detected.

Making code review on BitBucket suck less

2013-05-08T23:20:00.001+09:30

I'm a big fan of code review, I feel that almost everything should be given a once over. I'm used to it in various work places, I'm especially used to it when working in an open source project.

BitBucket is a follower, not a leader, and has managed to get a lot of what makes Github work well added - inline comments are great.

BitBucket has lead in one area though - much like a 'merge pull request', on each change there's an Approve button.

If you are happy with a commit, it's a click away - you don't even have to say 'LGTM', if not, have a discussion.

This adds a handy tick in the commits list, letting you see what is accepted or what needs to be talked about.

There are two problems with this though:

Every time you hit the button, notifications are sent
The review is per commit, not per feature or branch. If you use a simple 'all in' model, it's hard to group a stream of work - later fixes are applied that invalidate your review, all in the same push.

The best answer I have for issue 1 is to create email rules. If it has to works "commit approved" from bitbucket.org, mark as read and archive it.

This dramatically changes the signal to noise ratio, but lets you still leverage the visual indications of review having taken place.

For the second, feature branches and pull requests are the next best bet - BitBucket allows you to do a pull request from a feature branch on your repository to master.

This lets review take place on sets of code and features; and works extremely well for integrating stable changes only.

Called it!

2013-05-05T17:54:00.000+09:30

http://clockwerx.blogspot.com.au/2012/10/mapping-rosewater-arg-and-much-more.html and http://clockwerx.blogspot.com.au/2012/05/things-i-want-to-build.html meets
Augmented Reality Games using open data.

A quick guide to code review

2013-04-25T01:38:00.000+09:30

In my opinion, it's good practice to have a strong culture of peer review - it ensures your team knows what is going on, catches simple issues, and makes you a better group of developers.

Bitbucket and Github both provide a quite intuitive interface for reviewing code. You can simply look at a list of recent commits, and find a particular line to add a comment.

If you are new to git, I'd recommend you find an existing project and try this out.

Find the commits tab - https://github.com/jaz303/tipsy/commits/master for example
You can see a number of commits, with references like 3545aa6ae2 - these are hashes that uniquely identify the change.
It shows the commit message, author, as well as date/time.
Viewing one brings up the review user interface.

Code review is a fairly simple process. Different teams do different things, but I generally do:

Does the code change meet the description?
Is the code self describing / self explanatory?
Are there tests?

If you have questions, or don't understand something - ask. It may not be that you don't understand, it may be an oversight by the original developer.

Give it a go on an open source project today!

A first time using Git, with Bitbucket - add existing code.

2013-04-24T01:26:00.000+09:30

If you are new to Git, it can be a bit much to get your head around. The content below assumes you are making use of http://bitbucket.org/ - a good alternative for small commercial teams, which is roughly as good as Github in a number of areas.

Git (Windows)
First, let's get a git client. If you are on windows, I recommend one called msysgit. It gives you access to a start menu item, known as Git Bash - this will look and feel a lot like a Linux terminal.

http://code.google.com/p/msysgit/downloads/list
Grab Git-1.8.1.2-preview20130201.exe, the "Official installer for windows"
Install, next, next, next, leave the line endings settings to whatever it defaulted to, next some more, SUCCESS!

Bitbucket

Much of the bitbucket UI for signup, create repository, etc is fairly self explanatory. Let's go ahead and create a repository.

When you've done that successfully, you'll be confronted with a screen:

I am starting from scratch
I have code to import

Selecting "I have code to import" will give you a number of git commands as examples - to initialize a repository, to add your content, and to add a remote.

There are two main protocols that git can make use of - https and ssh. The https URLs look like

https://you@bitbucket.org/you/repository_name.git

... and for the moment, are the simplest to understand. If you already know about SSH and SSH keys, use the other protocol.

When you add a remote, you give it a name - in this case, origin. This is just a shortcut for the bitbucket server - you can choose any name you like; however there is a widespread convention to use origin with the main repository.

For windows users,

Start
Git Bash - this is much like the windows command line (cmd), with a few different commands.
Find your existing code - cd /c/path/on/your/hard/disk - note the direction of the slashes!

In Git Bash, instead of seeing a prompt like c:\>, the prompt ends in a $.

To get to your c:\ drive

$ cd /c/

To get to another path

$ cd /c/path/to/location

If you aren't sure, you can hit the TAB key repeatedly to autocomplete your filenames or show you a list of suggestions.

Once you have located the right directory with your existing code, we want to initialize a repository

$ git init

Next, add a remote called origin pointing at bitbucket.
$ git remote add origin https://you@bitbucket.org/you/repository_name.git

Tell Git to monitor the files - this is like an svn add.
$ git add .

To see the affect of git add, do
$ git status

# On branch master
#
# Initial commit
#
# Changes to be committed:
#   (use "git rm --cached ..." to unstage)
#
#       new file:   A.xml
#       new file:   B.java
#       new file:   C.wsdl
#       new file:   D.wsdl
#
you@yourcomputer ~/path/to/local/code (master)

You can see the code is now being tracked. It wants you to commit.

So, perform your first commit:
$ git commit -m "Importing my code" -a

[master (root-commit) 0bba05d] Importing my code

 4 files changed, 127 insertions(+)

All done, right? Not quite. You've made a commit, but this is only local to your repository. Think of it as the save command in a word document - it doesn't publish anything, but it lets you undo/redo changes.
From here you can do as many changes and as many commits as you want.

$ git commit -m "fix typo"
$ git commit -m "remove swearwords"

Also, you probably saw some warnings like:

*** Please tell me who you are.

Run

  git config --global user.email "you@example.com"
  git config --global user.name "Your Name"

It's worth doing this, as each commit is attributed to a user - you'll only have to do this once. Don't worry too much: your commit will still have succeeded.

Anyway, you can check the status again with

$ git status
# On branch master
nothing to commit, working directory clean

It should tell you if there are any untracked files (git add them), or uncommmited files (git commit them) - but ideally, it should look like the above.

To publish your changes, you need to push them. For simplicity, we'll also set a default destination (via -u) - the origin remote repository; master branch.

$ git push -u origin master

Password for 'https://you@bitbucket.org':
Counting objects: 6, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 3.81 KiB, done.
Total 6 (delta 1), reused 0 (delta 0)
remote: bb/acl: you is allowed. accepted payload.
To https://you@bitbucket.org/you/repository_name.git
 * [new branch]      master -> master
Branch master set up to track remote branch master from origin.
you@yourcomputer ~/path/to/code (master)

... and check bitbucket.org - your code should be published!

Great - but what just happened? If you look at the diagram, we've done everything on the left hand side - have a local, have a remote, and pushed changes. It's now possible for someone else to clone your repo, or fork it, and pull changes.

You can see that there can be many repositories - sometimes teams will maintain a 'core' repo, and people on the outside take a 'fork'.

New OpenStreetMap Editor available.

2013-04-06T22:09:00.000+10:30

http://ideditor.com/ is pretty neat - it's a bit slower than potlatch, but try tracing the roof of a house.

Here's the full blog post. It certainly looks like the kind of editor you want for any kind of property data capture - imagine doing a kerbside valuation and pulling this up on an ipad to explore / add data.

Bugs and annoyances with the Adelaide Metro Card Site

2013-03-27T08:40:00.001+10:30

1. You cannot mark a card as lost or consolidate balances. I am now on my third card.

2. Add one item to the cart, it redirects you to the list view. Add a duplicate due to a spotty connection - say... found on a bus in motion - it silently redirects you to the referring page but still adds the item.

3. The site will charge you for multiple things which are identical.

4. Deleting an item from the cart results in a 'temporarily unavailable' action, and does not remove content from your card.

5. It is so painful to complete the act of purchasing, that I am twenty minutes in to the process. A bus driver laughed when I suggested I could recharge my expired card and reswipe it.

It seems like a fairly obvious set of things to cover during testing. I cant understand how a simple cart + cc gateway can take so long (Nov 2012 to now) and be so buggy.

Consuming SOAP services with Ruby / Rails

2013-03-21T09:38:00.000+10:30

Savon is a ruby soap library, and I mostly like it. I've been using the latest Savon to do integrations with SSRS at work.

Getting started
As practical advice, I strongly recommend you go and find a decent pre-built SOAP IDE. I've used XMLSpy and Oxygen before, which have a slightly wider focus than just pure SOAP; but in recent times I've come to like SoapUI quite a lot.

Compared to other tools, it will:

Take a WSDL
Generate a sample request for each action
Generate code for you in you happen to be writing in Java - and stub services
Allow you to easily set different endpoints (ie: prod, staging, dev) without stuffing around with WSDLs.
Allow you to develop a number of premade requests which will return expected responses.

In short, it's a heck of a tool with a low price point: free.

SSRS

If you aren't developing in C# or similar, MSDN isn't very helpful. Not having had to... enjoy... windows servers for some time, I found a lot of time was spent:

Working out if I was authenticating correctly (browser to web UI, browser to WSDL, SOAP UI to WSDL, SOAP UI to SOAP Action, Actual Code to Soap Action)
Blundering around looking for configuration files in a file manager, instead of using google to find a likely file path and using vim until success was achieved.
Finding all kinds of Authentication that should work, simply don't. In the end, if the SOAP MSDN articles had simply pointed me to Configure Basic Authentication on the Report Server, I think I have achieved more.
Discovering the docs were incorrect re what WSDLs were actually available vs what the server published, and what worked vs the configuration of 'Native Mode'.

Once you get over the successive hurdles of Auth, identifying the correct service, and guessing that ListChildren is the method you want to get a list of reports, it's fairly straight forward.

Gotchas with Savon
A few of the soap calls wanted specific soap headers. Savon only supports global soap headers, and once you've built the object it's hard to mutate it.

Tips:
1) To get more control, ditch the Savon::client() factory method - it's not really doing too much you can't do yourself.
https://github.com/savonrb/savon/blob/master/lib/savon.rb

2) If you need to monkey with the global options, you can often extend your client to make them accessible.
https://github.com/savonrb/savon/blob/master/lib/savon/client.rb

class ExtendedSoapClient < Savon::Client
attr_writable :globals
end

3) It's worth stealing from Javaland in most cases - for each configured Savon Client, do up a quick FooServiceProxy. Inject your client into the constructor, and for all of the management of WSDLs, go stick that in your environment config.
If needed, go build yourself a SSRSServiceFactory to handle all of this wiring up.

The primary advantage - you get a reasonably clean API that can return either hashes, or if you go a little further you can generate models for the SOAP responses and have those appropriately populated.

It's important to remember these proxy classes exist only to translate a SOAP request from your code, to the transport layer, and to return a simple, meaningful result to your code.

That means avoiding a lot of logic in favour of making an API explicit.

My recommended pathway into Ruby / Rails

2013-03-21T09:37:00.001+10:30

I was asked what I'd encourage a complete outsider to RoR to look into.

Here's my list - what would you add?

- Linux intro
- User accounts
- File permissions
- su / sudo
- Installing programs (apt)
- What's a distro? (Ubuntu, Debian, etc)
- Gnome desktop
- Alternatives
- Bourne Again SHell (BASH)
- cd, ls, touch, vim, top, kill, pgrep, pkill, mkdir, chmod, chown
- Basic bash scripts
- Git intro
- Cloning
- Commits
- Log
- Push
- Pull
- Fetch, Merge
- Branches
- Tags
- Revert
- Reset
- Ruby intro
- Hello world
- Functions, Control flow, Loops
- Classes
- Modules
- Blocks! Lambas! Currying!
- Chunky Bacon and Foxes!
- DSLs
- Libraries (gems)
- Require / include path
- Metaprogramming
- OOP Intro
- OOP: SOLID principles
- OOP: TDD
- HTML intro
- HTML5
- what is it?
- differences to HTML4/xhtml?
- Data attributes
- New inputs
- Javascript basics
- JQuery
- AJAX
- Backbone, Ember, etc
- D3
- CSS basics
- CSS3
- Responsive design
- SVG basics
- JSON! YAML! Data structures & serializations.
- Rails intro
- Generators
- Rake
- Organisation
- Scaffolds
- Convention, not configuration
- Rspec
- Sinatra intro
- SASS / Less
- What is it?
- Bootstrap/Compass
- Continuous Integration
- Jenkins
- Travis CI
- Post commit hooks
- Rake tasks
- DevOps!
- Chef
- Puppet

Adelaide Ruby Hack Day

2013-02-28T14:12:00.002+10:30

There's an effort to pull together a Ruby Hack Day

We're aiming to start up an open coding session on the Saturday following the monthly meetups. The main goal is to complement the presentations with a more hands on component in which some of the more experienced guys can make themselves available to help out newbies working their way into the Ruby / Rails world.
The guys at the Majoran Distillery have been fantastic and offered to host this, so we've locked in the first one for the 9th of March from 10am-1pm.

I'd like to go, but have travel planned.

A decent article on what tends to upset me about rails

2013-02-28T09:14:00.000+10:30

From http://collectiveidea.com/blog/archives/2012/06/28/wheres-your-business-logic/

I put some of the blame on Rails itself, which has guided developers to use Controllers, Models, or Libraries, and nothing else.

and

Are your tests slow (>1s to run a single unit test)? Do your tests have large ungainly setup? Are you using factory_girl in your unit tests? Are you mocking implementation instead of interfaces?

I'm only really into my 3rd or 4th rails application of appreciable size, and they all seem to have this kind of problem - the tight coupling of active record to the model, and the model mixing persistence with business logic is rampart.

Enter the Interactor. An Interactor handles a use case. It pulls together the models and libraries it needs to process a single business rule, and then it’s done. These objects are very easy to test and use and in proper OO fashion can be used anywhere the app needs to apply the use case or business rule.

I dont think you need to call it an Interactor, it's just a Model. Not the relational-persistance kind, but the Model of your business logic - expressly applying SRP.

Ruby, Currying

2013-02-26T10:13:00.001+10:30

One of the better, clearer explanations of currying.

http://khelll.com/blog/ruby/ruby-currying/

Obviously useful for math, or scenarios where you inject your own lambdas (ie: sorting).

Keyword arguments.

2013-02-25T13:55:00.002+10:30

Thank god.

# Ruby 1.9: # (From action_view/helpers/text_helper.rb) def cycle(first_value, *values) options = values.extract_options! name = options.fetch(:name, 'default') # ... end # Ruby 2.0: def cycle(first_value, *values, name: 'default') # ... end

More @ http://blog.marc-andre.ca/2013/02/23/ruby-2-by-example/

Tree structures in relational data stores

2013-02-19T12:54:00.000+10:30

http://fungus.teststation.com/~jon/treehandling/TreeHandling.htm

Handy to read, as it lets you run queries over collections when tree relationships dont matter as much. IE cascading security or calculating totals.

Hurray for modern javascript!

2013-02-05T14:12:00.000+10:30

[1, 2, 3].foreach(function(i) { alert(i); });

https://developer.mozilla.org/en-US/docs/JavaScript/Reference/Global_Objects/Array/forEach

So. Much. Yes.

2013-02-05T00:28:00.001+10:30

http://mattsears.com/articles/2011/11/16/nyan-cat-rspec-formatter

Capistrano frustrates me

2013-02-01T10:02:00.001+10:30

Here's why:

1) It defines its own class path. If you want to load a customised extension, good luck - it is seemingly unaware of gems you add via bundle; pointing at a git repository

2) Task definition and execution are not separated. require is replaced with load - and load means "load and run".

For example - https://github.com/capistrano/capistrano/blob/master/lib/capistrano/recipes/deploy/assets.rb
You cannot "just" load the code and do something like compile assets locally by drawing on commands generated by someone else.

3) The methods defined are too rigid. I would refactor every capistrano recipe out there into behaving in two ways. First, there's a lot of methods that generate commands to be run. Second, there's business logic which decides where those commands should be run.

Looking at the recipe above:

task :clean, :roles => assets_role, :except => { :no_release => true } do
run "cd #{latest_release} && #{rake} RAILS_ENV=#{rails_env} #{asset_env} assets:clean"
end

You get no control there - the run() method will execute that remotely.
More frustrating: the commands are dumb strings. You don't really have the input escaping or knowledge of what's valid to put into the command in the code itself.

There needs to be flexibility, I agree, but some kind of basic input checking or modelling of what's happening should be more widely used.

4) The recipes tend to have no idea of where they are - you might as well write shell scripts to do most of it. Every time you see 'cd {absolute path}' is a good indication of this - smarter recipes would likely check current working directory (pwd); or keep a pointer to that.

The only thing that capistrano does better than pure bash scripts:

There's a lot pre-written, so you don't reinvent the wheel.
It has a dependency solver (before this do that, after that do this)

Air con remote control success

2013-01-27T17:18:00.000+10:30

After yet more hacking, I've stuck together just enough code to think I want to go and find a proper home automation framework.

I've got weather-util installed on a cron job doing:
0 * * * * weather ypad ~/butler/weather.txt

This gets me a small text file:

$ cat weather.txt 
Current conditions at Adelaide Airport, Australia (YPAD) 34-56S 138-31E 4M
Last updated Jan 26, 2013 - 11:30 PM EST / 2013.01.27 0430 UTC
   Temperature: 73 F (23 C)
   Relative Humidity: 46%
   Wind: from the SW (230 degrees) at 14 MPH (12 KT)
   Sky conditions: mostly clear

The first argument is an airport identifier. Next, I've used Blather to pull together a quick XMPP bot. It supports commands like

!ac on
!ac cool
!ac temp 23
!ac off
!weather

!weather just pipes the weather.txt back to whomever messaged the bot.

The !ac commands invoke some of the code to send commands to the air conditioner I talked about earlier.

Finally, the bot checks the configured owners for all Auth, and if it hasn't seen you in 8 hours, says hello.

It's far from the cleanest code I've ever written, and Blather is good but needs that little bit more to become "the rails of XMPP". Most importantly here though, I'm a little bit of regex away from writing some simple rules (IFTTT style), and adding the relevant cronjobs. The rules are going to be fairly terse:

while temp > 30; turn on AC

while temp < 25; turn off AC

I'd say if I did much more I'd want to refactor a lot of stuff to be more componentized - IE: jabber notification service when a decision is made doesn't need to be right next to the jabber command service.

I can't decide if I'd want to bring in something like Whenbot, or deal with something as complicated seeming as rhouse. Rhouse (linuxMCE) has a lot of UPnP stuff built it, but annoyingly the samsung smart devices are only using a subset of it - my phone notifies, the air con notifies back; rather than properly using M-SEARCHs.

Having spent a lot of time on this, I suspect that would be quite annoying to work with.

The next step I'll probably do is a further cleanup of my nmap code - when my phone's mac address walks in the door, the butler-bot should message me a greeting / fire up the AC decision engine.

Finally, I have to stick this on it's production server - my new raspberry pi.

More than happy to share the code I've got with anyone interested, it's in a few private bitbucket repos for now - I figured there would be few people with the same setup as I have; so kept it private (at least till I can make it into a few gems)...

SMART Airconditioner, Part II

2013-01-26T00:49:00.001+10:30

I started trying to figure out my air conditioner, and have made some fairly good progress.

Unfortunately, though I've been able to do a simple proof of concept; it's ambiguous as to how I can really make use of the resulting code - the terms of service narrowly define only interfaces provided by Samsung can be used to access their services.

I'd wager that a restful service is "provided by Samsung" in all of the technical senses, but I don't really have the desire to explore the legal interpretations of that.

Anyway...

As a first step, you want to authenticate your restful client against the Samsung SSO interface.

I made use of Mechanize and providing my credentials to correctly generate an authorisation cookie for the samsung service; in an identical fashion to how chrome would do this step.

There are a few steps to this, and a lot of javascript going back and forth.

A post to https://account.samsung.com/account/check.do to get the login form (with a ServiceID and ActionID); followed by submitting the actual login form.
There's yet more javascript that changes the form submit to location, so you end up manually pointing your requests at https://account.samsung.com/account/startSignIn.do

A final form submit completes the sign in, and you have a cookie somewhere in there generated.
After that, a fairly overwhelming interface renders itself. A lot of javascript happens to dynamically load UI controls and so forth.

Rather than deal with this tag soup, I took my DUID (the mac address from before) and relied on the existing cookie to go forward.

Mechanize wasn't going to let me do much with POST and raw XML, and I tried a few different clients - I didn't find one I was happy with.

RestClient was close, but to be honest I would likely just use Net::HTTP directly in other situations.
After I had the authorisation, I sat in chrome and watched what was happening.
First, there's a call to understand your device's capabilities - selectDevice().

You tend to get a lot of data back, and it took me a while to figure out that this is probably just the same UPnP service description file I was looking for.

Interestingly, Samsung spell out all of the operations you can perform on the machine - it's certainly not a WSDL, but not too hard to see how they would create a number of dynamic UPnP control devices/services/UIs.


      <Power type="string">
        <AvailableList>
          <On/>
          <Off/>
        </AvailableList>
      </Power>
      <OperationMode type="string">
        <AvailableList>
          <Auto/>
          <Cool/>
          <Dry/>
          <Wind/>
          <Heat/>
        </AvailableList>
      </OperationMode>

Actually using the interface they provide, I saw requests to:

http://global.samsungsmartappliance.com/Communication/setControl

After a bit of probing, this looks very like the 'key value pair' style of controlling a UPnP device.

For example, to switch on your AC:

POST http://global.samsungsmartappliance.com/Communication/setControl

Cookie: (What you had before)

duid: (mac address)

The response to this is an acknowledgement and a UUID, known as a CommandId.

The website them proceeds to spam the hell out of the services - 1-2 requests a second, polling - just to see if your command worked at all; and to update the device status.

GET checkControl

CommandId: UUIDFromBefore


<rsp stat="ok"><ControlResult DUID="...">Processing</ControlResult></rsp>

GET checkControl

CommandId: UUIDFromBefore


<rsp stat="ok"><ControlResult DUID="...">Success</ControlResult></rsp>

Given the previous work sniffing packets, I would think I've just sent basically the UPnP packets to the AC, and global.samsung.com would be doing little to no transformation of them; beyond adding/removing from a queue.

At this point, I was ready to try controlling the AC with my restful code and start publishing it to github - unfortunately; I thought to read the ToS and found the narrowly worded agreement.

While I'd usually consider that hard to enforce, I don't really want to find out the ramifications of "unauthorised use of a computer" in Australian law - even if the end service is a Thing in My House that arguably doesn't need traffic going to Korea for it.

However, since I now have a pretty good idea of the other half of the conversation; I'm going to look closely at UPnP in the local environment again. If the services look similar, and it's just a matter of the same requests; but over UDP, I'm sorted!

How the Samsung SMART Air conditioners/appliances work remotely

2013-01-24T23:20:00.001+10:30

I bought a smart air conditioner, and am marvelling at the fact I can control it from my phone.

What I'm less happy about is the fact I can only control it from my phone; and that my local network has decided to stop letting me talk to the air con.

I fired up wireshark, and here's what I know so far:

Like most devices, the Air Conditioner is running UPnP services basically. There's packets being sent via http://en.wikipedia.org/wiki/Simple_Service_Discovery_Protocol for my Android phone and the air con to discover each other.

Here's my phone asking over HTTPU (HTTP UDP) for air conditioners via multicast:
NOTIFY * HTTP/1.1
LOCATION: 192.168.1.2
HOST: 239.255.255.250:1900
CACHE-CONTROL: max-age=20
SERVER: AIR CONDITIONER
MAC_ADDR: 98:0c:82:f4:d4:5c
SPEC_VER: MSpec-1.00
SERVICE_NAME: ControlServer-MLib
MESSAGE_TYPE: CONTROLLER_START

... and the air conditioner starts to respond:

NOTIFY * HTTP/1.1
LOCATION: http://192.168.1.4
NTS: ssdp:alive
CACHE_CONTROL: max-age=60
HOST: 255.255.255.255:1900
SERVER: SSDP,SAMSUNG-AC-BORACAY
MAC_ADDR: 7825AD103D06
SERVICE_NAME: ControlServer-MLib
SPEC_VER: MSpec-1.00
MESSAGE_TYPE: DEVICEDESCRIPTION
NICKNAME: 536D61727420412F432837383235414431303344303629
MODELCODE: SAMSUNG_DEVICE

Unfortunately, my air con actually lives at 192.168.1.7, the web application isn't smart enough to respond to changing IP addresses; even if the web server and everything else is.

I'm not entirely sure why, but my phone tends to flood packets out there - I presume the people who put the phone app together wanted to avoid dodgy local wireless problems; and this now explains why my air con is always discovered quickly, but the screen keeps 'loading'.

Strangely, for the local in home network stuff, it looks like my air conditioner is talking to a samsung owned service to give it status updates, even when I'm on my local network:
211.45.27.216

HTTP/1.1 200 OK
Cache-Control: private
Content-Length: 111
Content-Type: text/xml; charset=utf-8
Server: Microsoft-IIS/7.5
X-AspNet-Version: 2.0.50727
X-Powered-By: ASP.NET
Date: Thu, 24 Jan 2013 10:38:31 GMT

<rsp stat="ok"></rsp>
<device duid="7825AD103D06" message="Off"></device>

Attempt #2: There's an online service you can use.
Now, our clever friends at Samsung have done the right thing and made communication with their services via HTTPS - at least from the android to the soap service. Weirdly, the air conditioner does not speak HTTPS; and seems to be sending packets to the samsung control service, not my phone directly.

I looked at a few packets, but at this point, I was pretty much fed up - I'm not going to do a MITM attack via my router, and redirect traffic to a HTTP proxy I control; I'm not going to reroute traffic, and sticking the app in an android emulator to sniff the packets is annoyingly hard.

So, what's really left to do but... log in to the website and look at what's going on there?

This looks like paydirt - there are numerous javascript files which are hitting endpoints to execute commands.
It's probably a fairly safe assumption to think those endpoints are just RESTful wrappers around the underlying webservice.

Two are of quite a lot of interest, if ... dubious quality:
http://global.samsungsmartappliance.com/Content/js/communication/commMgr.js
http://global.samsungsmartappliance.com/Content/js/communication/Device/Device.js

Looking in firebug/chrome's console, you can see that the device traffic is going back and forth, with an incrementing token continually polling my air conditioner for status.

When you actually execute a command, a fairly simple request is sent:

Request URL:

http://global.samsungsmartappliance.com/Communication/setControl
Request Method:

POST
Status Code:

200 OK
Request Headersview source

Accept:

application/xml, text/xml, */*; q=0.01
Accept-Charset:

ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:

gzip,deflate,sdch
Accept-Language:

en-GB,en-US;q=0.8,en;q=0.6
Connection:

keep-alive
Content-Length:

122
Content-Type:

text/xml
Cookie:

ASP.NET_SessionId=y3akcf55bfxtcnzhd0hyz5ru; Language=en; Country=GB; LoginLanguage=en; iPlanetDirectoryPro=AQIC5wM2LY4Sfcy5aJdOInrRhyDKEyNfWseX44FQLVR9hZI%3D%40AAJTSQACMDIAAlMxAAIwNQ%3D%3D%23; GMT=10.5
Host:

global.samsungsmartappliance.com
Origin:

http://global.samsungsmartappliance.com
Referer:

http://global.samsungsmartappliance.com/
User-Agent:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.52 Safari/537.17
X-Requested-With:

XMLHttpRequest

Request Payload

Response Headersview source

Cache-Control:

private
Content-Length:

100
Content-Type:

text/html; charset=utf-8
Date:

Thu, 24 Jan 2013 12:40:11 GMT
Server:

Microsoft-IIS/7.5
X-AspNet-Version:

2.0.50727
X-AspNetMvc-Version:

1.0
X-Powered-By:

ASP.NET

Most of the other status information has included the MAC address of the air conditioner as the "DUID" - it's stripped of colons, but otherwise identical. While we can't tell what's in the session on the samsung side, I would not be surprised if it were a simple variable named DUID, which identifies where the command should be sent.

More interestingly, the requests can be replayed; so assuming you have a valid cookie that contains the right DUID, you can just send restful requests.

I'm going to assume that there's some kind of ESB or similar taking these requests and distributing them from my phone/browser to the aircon; as the traffic to the aircon is different to the endpoint my phone / browser speaks to.
There is a delay, perhaps 3-5 seconds, which is about what I would expect if there was a Giant Messaging Queue in use here; and to be honest the site seems to go down regularly - some service faceplanting it seems likely.

The fact it's jquery + xml being used also suggests there's an Enterprise Service Beast in play - though json is more common, the transformation between formats is a bit easier if you can hit it with an XSLT.

The use of a session would normally make things more secure... but the control to manipulate that value appears to be right out in the open.

Request URL:

http://global.samsungsmartappliance.com/Communication/selectDevice?_=1359032361539
Request Method:

GET
Status Code:

200 OK
Request Headersview source

Accept:

application/xml, text/xml, */*; q=0.01
Accept-Charset:

ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:

gzip,deflate,sdch
Accept-Language:

en-GB,en-US;q=0.8,en;q=0.6
Connection:

keep-alive
Cookie:

ASP.NET_SessionId=y3akcf55bfxtcnzhd0hyz5ru; Language=en; Country=GB; LoginLanguage=en; iPlanetDirectoryPro=AQIC5wM2LY4Sfcy5aJdOInrRhyDKEyNfWseX44FQLVR9hZI%3D%40AAJTSQACMDIAAlMxAAIwNQ%3D%3D%23; GMT=10.5
duid:

7825AD103D06
Host:

global.samsungsmartappliance.com
master_duid:
Referer:

http://global.samsungsmartappliance.com/
User-Agent:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.52 Safari/537.17
X-Requested-With:

XMLHttpRequest

I doubt you could hijack someone else's air con if you knew the mac address, but for devices associated with your account...

At this point I have two good avenues to explore:

Local network - I need to fix the IP addresses, and see what's going on after the SSDP phase. I had to authorise my phone, so I suspect that Samsung provides a third party service that verifies allowed MAC addresses prior to opening all of the communication flood gates.
Pretend to be a browser, and invoke the restful services. If I supply my credentials, I can get the session I need - then the API is just a few POST requests away.

I'm likely to do the latter, unless I can figure out how to power cycle my aircon properly, and get it understanding its IP address.

Forking Tipsy

2013-01-23T13:12:00.001+10:30

I like Tipsy, it's a great little plugin that works very well with D3 graphs to add tooltips.

Unfortunately, it's not very maintained - 51 pull requests open, and perhaps 2-3 ever merged.

As a result, I've forked tipsy and applied a good number of those fixes.
https://github.com/CloCkWeRX/tipsy

Hopefully the original author will just pull the combined changes from my branch back in and continue maintaining it, but if that doesn't happen I'll continue to apply any pull requests sent my way.

Activerelation, stop being so clever.

2013-01-17T15:41:00.004+10:30

I nearly flipped by lid today, when doing what should have been a simple activerelation query to get the 'last record'.

Foo.order('column')
SELECT * FROM foo ORDER BY column ASC;

Foo.order('column ASC').last
SELECT TOP 1 * FROM foo ORDER BY column DESC;

Foo.order('column DESC').last
SELECT TOP 1 * FROM foo ORDER BY column ASC;

Foo.order('column DESC').first
SELECT TOP 1 * FROM foo ORDER BY column DESC;

Foo.order('column ASC').last
SELECT TOP 1 * FROM foo ORDER BY column ASC;

Foo.order('column ASC').limit(1)
SELECT TOP 1 * FROM foo ORDER BY column ASC;

Foo.order('column DESC').limit(1)
SELECT TOP 1 * FROM foo ORDER BY column DESC;

Something tells me I'll never like SQL server's TOP 1 syntax, as I spent ages just staring at the end of the queries and wondering "why the heck isn't it choosing the last record, I told it to sort DESC!"

I miss my LIMITs.

I will settle for an Advertiser beat up of the 230 bus

2013-01-11T16:02:00.000+10:30

Everything's coming up Dan!

http://www.adelaidenow.com.au/south-australias-worst-bus-routes-are-located-in-the-inner-northern-and-southern-suburbs/story-e6frea6u-1226551464579

I can understand a late bus during the cricket...

2013-01-10T22:51:00.001+10:30

... but I really want to know where my no show 230 was at jeffcott st. over ten minutes later than published, no sign.

Caught a taxi. Actually want an apology or even better, recompense for the inconvience.

The shut down of the railway station seems ill timed and ill thought through.