Mark Aufflick's Weblog

AQXMLParser + OAuth

Thu, 02 Aug 2012 02:25:20 GMT

I've been enjoying using Jim Dovey's excellent AQXMLParser lately, and in fact am giving a talk on it tonight at Sydney CocoaHeads. To use it in a new project, I need to be able to authorise the outbound REST request with some OAuth 1 tokens etc.

OAuth can be a right pain, and the Cocoa implementations aren't awesome. gtm-oauth (the OAuth 1 library from the Google Toolbox for Mac) is a little odd in its implementation, but doing your own OAuth implementation would be so tedious it's one of the better options out there.

It's all pretty easy. Firstly, use gtm-oauth like normal to have your user authorise your app and receive a token.

Next, to make a request, gtm-oauth provides a way to authorise an NSMutableURLRequest, but AQXMLParser uses it's own HTTPMessage class (which wraps the lower level CFHTTPMessage from Core Foundation).

What I have done is extend HTTPMessage to provide a copy of itself as an NSURLRequest, and then write a category that auths that with gtm-oauth and then cribs the headers from that and applies them to the HTTPMessage. It's a little roundabout, but because of the need to sign the whole request and the way that gtm-oauth internals work, anything else would be very complicated. So in use, it looks like this:

#import "HTTPMessage+GTMOAuth.h" ... GTMOAuthAuthentication * oauth = ...; // setup gtm-oauth like normal HTTPMessage * myMessage = ...; // create the http request like normal [myMessage authorizeWithGTMOAuth:oauth]; // now create the parser and use the http message etc. like normal

For now the changes and category are only in my fork, but I have a pending pull request with Jim that I will work out the best way to incorporate them into his master repository. My fork also contains tweaks to work with the latest llvm and ARC (although note neither gtm-oauth nor HTTPMessage or AQXMLParser use ARC, so you'll have to add the -fno-objc-arc compiler flag where appropriate).

gtm-outh:�http://code.google.com/p/gtm-oauth/

My fork of aqtoolkit:�https://github.com/aufflick/aqtoolkit

supervisord supervisorctl cheat sheet

Sun, 25 Mar 2012 23:14:32 GMT

On my new server I made the switch from daemontools to supervisor. As if to make daemontools users feel at home the documentation for supervisorctl use is a bit thin :)
So far my biggest gotcha was that after adding a new .conf file in your includes (or to the main file itself, but you wouldn't do that would you?) is that you need to do *both* of the following:

> supervisorctl reread
new.app.conf: available
> supervisorctl update
new.app.conf: added process group
> supervisorctl
new.app � � � � RUNNING ...

�

haproxy

Unfortunately supervisor doesn't offer a way to provide a custom restart command, so doing an uninterrupted haproxy restart via supervisor isn't possible. Instead make sure you have autorestart set to unexpected (the default) rather than true (which is what I usually do) - that way if you want a clean haproxy restart you can just do it yourself at the commandline using the -sf option as normal.

SAPI Cocoa SDK

Sat, 24 Mar 2012 01:13:24 GMT

The Cocoa Sensis SAPI SDK is live, ready for this weekend's Sensis SAPI hackathon. https://github.com/pumptheory/SAPI-Cocoa-SDK

Stunnel + Thin + Rails for the world's easiest ssl hosting

Tue, 20 Mar 2012 01:52:39 GMT

These days I do most quick and dirty web stuff with Ruby on Rails, and any open source web stuff I install I prefer Rails based code for easy hacking.�Every so often you want to host something for internal or other small-volume usage, but it has to be SSL. Traditionally this is a pain in the backside, especially with Rails. Back in the day I did everything with OpenACS or plain AOLServer. This meant I had to install a bunch of stuff by hand (although debian packages are now available), but once it was done it was a simple config tweak to change the number of threads and/or add ssl etc.�With Rails it's kind of the opposite. Getting things up and running is incredibly easy, but scaling it or adding ssl usually means �Apache, or Nginx, or some other (relatively) heavy server plus other stuff. Doing this all as a non-super user is even more painful.

For a client I wanted to setup a Redmine instance for a wiki and browsing internal git repositories. It's a Rails 2.x app. Initial setup is easy. Using the latest trunk version of Redmine (which supports Ruby 1.9 and uses bundler) it's a simple matter of the following steps (which assume pretty much nothing is installed except base libraries like sqlite3):

checking out the svn trunk

svn co svn co http://redmine.rubyforge.org/svn/trunk redmine
cd �redmine

installing rvm,�setting up the right ruby & gem set

bash -s stable < <(curl -s https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer)
. ~/.bashrc
rvm install ruby-1.9.3
rvm use ruby-1.9.3@redmine --create
rvm install bundler

installing bundler and using it to prepare all the ruby gems (I am using sqlite and don't have mysql libraries installed)

gem install bundler
bundle install --without=mysql

follow the main redmine installation steps from step 3, but make the database adapter sqlite3 and the database db/production.sqlite
http://www.redmine.org/projects/redmine/wiki/RedmineInstall

You now have Redmine up and running, via the test webrick server, on port 3000. But that's no good for two reasons, one we obviously don't have ssl, and two webrick is a single process (and ruby is always a single processor thread) so it's not very scalable.

A nice simple, and yet fast, way to scale Ruby http servers is Thin. It's Ruby and C. It's only http though, so we need to put ssl in front some how. A great way of doing this is stunnel, which can tunnel just about anything to anything when it comes to ssl. So we run a couple of Thin instances, and stunner to proxy it to ssl. The great thing is we can arrange for the communication between the two to be via Unix sockets so we don't even need to reserve localhost TCP ports?so we can scale the number of instances without checking with anyone else.

Here are my steps for Thin and stunnel:

Install Thin:

gem install thin

Install stunnel:

# get the latest version of stunner from�http://www.stunnel.org/?page=downloads
tar zxf stunnel-4.xx.tar.gz
cd stunnel-4.xx
mkdir ~/usr�
./configure --prefix=~/usr
make
make install

You need a .pem certificate for stunnel. If you don't have a real certificate, you can create a self-signed one like so:

openssl req -new -x509 -days 365 -nodes -config stunnel.cnf -out stunnel.pem -keyout my_stunnel.pem

where the stunnel.cnf file is in the source distribution. Answer the questions appropriately.
stunnel needs a config file, something like this (in this case for three thin instances)

pid =
cert = /path/to/my_stunnel.pem

[redmine]
accept = 12345
connect = /path/to/my_rails_app/tmp/thin.0.sock

connect = /path/to/my_rails_app/tmp/thin.1.sock

connect = /path/to/my_rails_app/tmp/thin.2.sock

start up stunnel

~/usr/bin/stunnel /path/to/my/stunnel.conf

start up your rails app with Thin

thin start -s3 --socket /path/to/my_rails_app/tmp/thin.sock -e production

And that's it, your site running on https port 12345! All this has not required root access at any point. If you want to listen on a low port, eg. 443, the only thing that needs to run as root is stunnel.

But don't break out the champagne yet - we have a problem. Your rails app knows the hostname and port thanks to the Host: header, but it�has no idea that it's running behind stunnel and is therefore ssl. This is only a problem in the occasions where a full url is needed, and the primary case is for http redirect headers. I take care of this by rewriting the Location header in application_controller.rb:

after_filter :force_https_for_redirects
 def  force_https_for_redirects
  if response.headers.has_key?("Location")
 response.headers["Location"].sub!(/^http:/, 'https:')
  end
 end

You may have other cases that need fixing, like constructing urls for emails. Note that I wouldn't recommend mucking about with the X-Forwarded-Proto patches for stunnel. I just don't trust third party patches for security software, and in any case I'm not sure it would work with using Unix sockets for the intermediary transport.

Update: Here's how to monkey patch ActionController::UrlWriter::url_for() so your emails will have correct links. In Rails 2 (in my case for redmine), I did it in environment.rb - for Rails 3 I think you could do it in application.rb, but it doesn't really matter where you do it as long as it gets compiled after ActionController::UrlWriter is loaded.

module  ActionController
    module  UrlWriter
        alias  :original_url_for :url_for
        def  url_for(options = {})
      options[:host] = 'efserv.com' unless options[:host]
      options[:protocol] = 'https'  unless options[:protocol]
      options[:port] = 7022  unless options[:port]
 
      original_url_for(options)
        end
    end
end

�

PMPKVObservation ? Yet Another KVO-improvement project

Tue, 13 Mar 2012 18:49:13 GMT

If you?re reading this then you?re probably as frustrated by seemingly random KVO crashes and/or the pain of huge if/else blocks in your observers. I?ve never been happy with the other KVO-improvement classes I?ve seen, and never having had any bright insights myself I kept doing things the normal way. This becomes especially painful with things like view based NSTableViews when you are re-using objects and so need to observe a new object, being very careful to un-observe the prior object (unless it?s been released, which you need to either track yourself if you can, or retain it, which has its own problems).

It was clear that the dealloc swizzling approach of Mike Ash?s MAKVONotificationCenter was unavoidable, but I didn?t like the complexity. Recently Gwynne Raskind posted a somewhat updated MAKVONotificationCenter which sparked some discussion, including a comment discussing SPKVONotificationCenter by Joachim Bengtsson. Joachim?s brainwave was that observations should be modelled as standalone objects and simply managing the lifecycle of that object appropriately should remove the observation. Clean and simple.

Except it?s not quite that simple because you still need to swizzle the dealloc of the observed object since it can go away at any time. And as much as I love a good macro as much as the next hacker, Joachim?s $depends() macro looks about as much fun as a turing complete makefile.

Enter PMPKVObservation!

Include PMPKVObservation.m and .h in your project, then do something like:

#import "PMPKVObservation.h"
 
@interface  PMPKVOTests ()
@property (retain) PMPKVObservation * kvo;
@end
 
@implementation  PMPKVOTests
@synthesize kvo = _kvo;
 
- (void)observationFired:(PMPKVObservation *)observation changes:(NSDictionary *)changes;
{
    NSLog(@"observation fired for object: %@ keyPath: %@ changes: %@",
        observation.observee,
        observation.keyPath,
        changes);
}
 
- (void)selectorObservation
{
    self.kvo = [PMPKVObservation observe:anObjectToObserve
                                observer:self
                                selector:@selector(observationFired:changes:)
                                 keyPath:@"observeMe"
                                 options:0];
}
 
- (void)blockObservation
{
    self.kvo = [PMPKVObservation observe:anObjectToObserve
                                 keyPath:@"observeMe"
                                 options:0
                                callback:^(PMPKVObservation *observation, NSDictionary *changeDictionary) {
                                              NSLog(@"observation fired for object: %@ keyPath: %@ changes: %@",
                                                  observation.observee,
                                                  observation.keyPath,
                                                  changes);
                                 }];
}
 
- (void)dealloc
{
    [ _kvo release];
 
    [super dealloc];
}
 
@end

The options and change dictionary are as per normal KVO. You need to be wary of the block causing retain cycles of self as usual.

Unlike MAKVONotificationCenter and others, you are responsible for managing the lifecycle of the kvo object ? when it is dealloc-ed the observation will be removed. Like MAKVONotificationCenter, a minimum of magic is used to swizzle the dealloc method of the observed object and remove the observation if it still exists (tracked via an associated NSHashTable). If for some reason you want to find out if the kvo object you hold has been released early you can check the isValid accessor (which you can observe with KVO).

Caveats

While I am using this in a client project, that project has not yet shipped so I can?t promise wide-spread testing. Take a look at the tests in the test app and if you are using scenarios that you don?t thing are tested pull requests are welcome!
I haven?t tried this under ARC, but I think I?ve made PMPKVObservation.m so it will compile and work as-is (the same can?t be said for the test app). Please let me know how you go.
No attempt is made to stop you changing the observed object etc. after creation, but doing so isn?t going to be very effective?

Repository:�https://github.com/aufflick/PMPKVObservation

Apple Museum

Wed, 22 Feb 2012 21:10:49 GMT

After a rejig of my office, there is finally room for the Apple museum. More pieces to come!

Your eyes don't deceive you, that is a 5 1/4 inch drive - the processor direct slot has an Apple // card in it (which was designed for the LC, which my parents bought it with, but it works great in the Colour Classic).

Emacs Programmed Completion

Tue, 21 Feb 2012 05:36:05 GMT

Subtitle - how (completing-read) stole my afternoon.

I thought I'd post my experiences since completing-read is so poorly understood based on the forum and blog posts out there. When you write an interactive Elisp command, many will know that you can easily read a value from the user. Eg: this will ask you how you feel and log it:

(defun  mood (mood)

  "Logs your mood"

  (interactive "sHow are you feeling? ")

  (message mood))

But the string form of argument to interactive is just a shortcut. You can easily supply some code instead, with a static completion list:

(defun   mood (mood)

  "Logs your mood"

  (interactive (list
                (completing-read "How are you feeling? " '("happy" "sad")))

               (insert arg))

  (message mood))

If you want something more complex though, instead of supplying a completion list, you can supply a function and that function receives as one of it's arguments the string to be completed. But it's not as simple as that function just returning a list unfortunately, and that's where people come unstuck. Your function is now embedded in the guts of the Emacs completion mechanism and as such it has to answer a whole range of questions about completions. It is reasonably well documented (if a bit terse) in the Programmed Completion section of the Emacs manual. There is a second argument which contains an optional predicate function (I'm ignoring that), but the important bit is the third argument which effectively sets the mode in which your function is being called.

In fact, your function will usually be called multiple times for each completion tab press. Essentially, your function may be asked to provide the best completion string possible, if there string is an exact match, if there are no matches, or to provide a list of possible matches. There is one final thing that will be asked, and that is the substring your completion is for. So in the example you're about to see I am completing directories, so the completion being offered at any given time only applies to the part after the last forward slash.

An example is as good as a thousand or so words, so here's my interactive function cdsrc. Basically it's a convenience function to quickly change directory into one of my client projects, which may be one or two levels deep in my main company source directory. The hard work is done by cdsrc-completions and you can see in the (cond) at the end where it responds to the different modes discussed briefly above and in detail in Programmed Completion.

(setq cdsrc-completions-prefix "~/src/_Pumptheory/")



(defun safe-fill-common-string-prefix (s1 s2)

  (if (and s1 s2)

      (fill-common-string-prefix s1 s2)
    nil))



(defun cdsrc-completions (str pred mode)

  (let ((dirname (replace-regexp-in-string "[^/]+$" "" str))

        (filename (replace-regexp-in-string "^.*/" "" str))

        (slashpos (or (string-match "/.*$" str) 0)))



    (let ((completions (all-completions filename (delq nil 

                                                  (mapcar (lambda (x)

                                                            (and (file-directory-p (concat cdsrc-completions-prefix dirname x))

                                                                 (not (string-match "^\\." x))

                                                                 (concat x "/")))

                                                          (directory-files (concat cdsrc-completions-prefix dirname)))))))



      ;; return differently based on what mode we were called in
      ;; see http://www.gnu.org/software/emacs/manual/html_node/elisp/Programmed-Completion.html

      (cond

       ((not mode) (cond

                    ((and (eq 0 (length completions)) (eq 0 (length filename)))
                     't) ;; in our case, only report exact match when no nested dirs left

                    ((> (length completions) 1)

                     (concat dirname (or (reduce 'safe-fill-common-string-prefix completions) filename)))

                    ((eq (length completions) 1)

                     (concat dirname (car completions)))

                    ('t
                     nil)))



       ((eq mode 't) completions)



       ((eq mode 'lambda) (member (concat filename "/") completions))



       ('t (cons (list 'boundaries slashpos) (length filename)))) ;; let completion know our completions only apply after the last /
      )))



(defun cdsrc (path)

  "Changes to a subdir of cdsrc-completions-prefix in the current active interactive shell buffer"

  (interactive (list

                (let ((completion-ignore-case 't))

                  (completing-read

                   (concat "Enter subdir of " cdsrc-completions-prefix ": ")

                   'cdsrc-completions))))

  (let ((path2 (concat cdsrc-completions-prefix path)))

    (cd path2)

    (if (string-match mode-name "Shell")

        (progn (end-of-buffer)

               (comint-kill-input)

               (comint-send-string (current-buffer) (concat "cd " path2))

               (comint-send-input)))))

This is out of my emacs.d, specifically aufflick-shell-hooks.el.

Comments are broken

Fri, 10 Feb 2012 01:44:17 GMT

It seems that adding comments is broken after the server move. I'll get right on it... Update: Comments are now fixed, both for registered and anonymous persons.

New Server

Thu, 09 Feb 2012 07:09:38 GMT

The server that used to host this site, and a few others, has finally had one of it's raid drives fail after over 10 years of use. That's like 100 in server years! At it's age it's time to put it to bed, so I've moved this site over to linode. I upgraded the underlying Postgres, OpenACS and AOLServer instances at the same time. There are a few styling issues that need fixing, but if you notice anything actually broken please let me know. BTW, as well as the same hardware running for over 10 years, the same Linux installation (with security patches which eventually had to be backported after it stopped being supported) has been running that whole time. If I remember right, the maximum uptime was around 700 days - just short of two years. Unfortunately it had to be shut down then to move it to another power rail. I felt a certain nostalgia poking around it's directory structure looking for anything I wanted to keep. It's probably the last server I've had with /usr/local/playground in a nod to russm. And certainly the last server I will ever run qmail on - it was so awesome, but the massively scalable queue based mail handling just doesn't work with today's spam probes.

Monday Reader?The Adventures of Sherlock Holmes

Sun, 13 Nov 2011 23:26:22 GMT

This week's Monday Reader isn't a technical book, but I'd recommend all developers and support people read all the Sherlock Holmes novels. Not only are the collections of short stories fun and light night time reads, Sir Arthur Conan Doyle has a way of encouraging the reader to be more logical and insightful in his or her thinking:

To carry the art, however, to its highest pitch, it is necessary that the reasoner should be able to utilise all the facts which have come to his knowledge; and this in itself implies, as you will readily see, a possession of all knowledge, which, even in these days of free education and encyclopaedias, is a somewhat rare accomplishment. It is not so impossible, however, that a man should posses all knowledge which is likely to be useful to him in his work, and this I have endeavoured in my case to do. ? a man should keep his little brain-attic stocked with all the furniture that he is likely to use, and the rest he can put away in the lumbar-room of his library, where he can get it if he wants it.

Being out of copyright, you can get these for free in iBooks or elsewhere, but I prefer to read paper books?which I can then put away in the lumbar-room of my library :).