MRo Blog

Rank sql(ite) text search results

2011-08-09T20:55:27Z

When searching for text snippets in sql databases you might want to rank the results according to “how good did it match”. And: the ranking shouldn’t make the query slower.

Let’s take a simple example using the LIKE operator. (I know, FTS does a better job, but let’s stick to like for now).

Assume the search expression ‘a bc de’ and a table ‘my_table’ with text columns ‘title’ and ‘description’.

We want to find all rows with ‘title’ matching all three blank-separated parts of the search term:

SELECT rowid, title
FROM my_table
WHERE (title LIKE '%a%' AND title LIKE '%bc%' AND title LIKE '%de%')

To sort them, we apply a bonus for parts matching the column start:

SELECT rowid, title,
    -- column start bonus
    LIKE('a%', title) +
    LIKE('bc%', title) +
    LIKE('de%', title) +
0 AS bonus
FROM my_table
WHERE ((title LIKE '%a%') AND (title LIKE '%bc%') AND (title LIKE '%de%'))
ORDER BY bonus DESC, title ASC, rowid ASC

Next, we’d like to add a (somewhat smaller) bonus for word-starts:

SELECT rowid, title,
    -- column start bonus
    LIKE('a%', title) * 2 +
    LIKE('bc%', title) * 2 +
    LIKE('de%', title) * 2 +
    -- word start bonus
    LIKE('% a%', title) * 1 +
    LIKE('% bc%', title) * 1 +
    LIKE('% de%', title) * 1 +
0 AS bonus
FROM my_table
WHERE ((title LIKE '%a%') AND (title LIKE '%bc%') AND (title LIKE '%de%'))
ORDER BY bonus DESC, title ASC, rowid ASC

Rows matching the three terms in order get an even bigger bonus:

SELECT rowid, title,
    -- correct order bonus
    LIKE('%a%bc%de%', title) * 5 * 3 +
    -- column start bonus
    LIKE('a%', title) * 2 +
    LIKE('bc%', title) * 2 +
    LIKE('de%', title) * 2 +
    -- word start bonus
    LIKE('% a%', title) * 1 +
    LIKE('% bc%', title) * 1 +
    LIKE('% de%', title) * 1 +
0 AS bonus
FROM my_table
WHERE ((title LIKE '%a%') AND (title LIKE '%bc%') AND (title LIKE '%de%'))
ORDER BY bonus DESC, title ASC, rowid ASC

And finally adding the match on ‘description’ secondary:

SELECT rowid, title, description,
    -- title is primary match:
    -- correct order bonus
    LIKE('%a%bc%de%', title) * 50 * 3 +
    -- column start bonus
    LIKE('a%', title) * 20 +
    LIKE('bc%', title) * 20 +
    LIKE('de%', title) * 20 +
    -- word start bonus
    LIKE('% a%', title) * 10 +
    LIKE('% bc%', title) * 10 +
    LIKE('% de%', title) * 10 +
    -- description is secondary match:
    -- correct order bonus
    LIKE('%a%bc%de%', description) * 5 * 3 +
    -- column start bonus
    LIKE('a%', description) * 2 +
    LIKE('bc%', description) * 2 +
    LIKE('de%', description) * 2 +
    -- word start bonus
    LIKE('% a%', description) * 1 +
    LIKE('% bc%', description) * 1 +
    LIKE('% de%', description) * 1 +
0 AS bonus
FROM my_table
WHERE ((title       LIKE '%a%') AND (title       LIKE '%bc%') AND (title       LIKE '%de%'))
OR    ((description LIKE '%a%') AND (description LIKE '%bc%') AND (description LIKE '%de%'))
ORDER BY bonus DESC, title ASC, description ASC, rowid ASC

You get the idea.

Funny thing is – the whole ranking logic doesn’t hit performance (at least for small texts in the two columns)!

So, key is:

scan the table only once to find match candidates using the LIKE operator,
use the LIKE function(!) plus weighting-factors to compute a bonus for each hit,
evtl. add secondary matching columns.

P.S.: This post was inspired by a chat with Deesa on the way home riding False Creek Ferry.

Ruby: Simple Fast Fourier Transform

2011-04-05T00:40:11Z

by far not as powerful as the Fastest Fourier Transform in the West but maybe sometimes useful for a quick data analysis or de-noising. Reads stdin and writes to stdout.

Algorithm taken from Meyberg, Vachenauer: Höhere Mathematik II and ported to plain ruby myself.

< ![CDATA[ 
#!/usr/bin/env ruby
require 'complex'
 
class Array
	# DFT and inverse.
	# 
	# Algorithm from 
	# 'Meyberg, Vachenauer: Hoehere Mathematik II, Springer Berlin, 1991, page 332'
	#
	# See http://blog.mro.name/2011/04/simple-ruby-fast-fourier-transform/ 
	#
	def fft doinverse = false
		src = self
		# raise ArgumentError.new "Expected array input but was '#{src.class}'" unless src.kind_of? Array
		n = src.length
		nlog2 = Math.log( n ) / Math.log( 2 )
		raise ArgumentError.new "Input array size must be a power of two but was '#{n}'" unless nlog2.floor - nlog2 == 0.0
		n2 = n / 2
		phi = Math::PI / n2
		if doinverse
			phi = -phi
		else
			src.collect!{|c| c /= n.to_f}
		end
 
		# build a sine/cosine table
		wt = Array.new n2
		wt.each_index { |i| wt[i] = Complex.new Math.cos(i * phi), Math.sin(i * phi) }
 
		# bit reordering
		n1 = n - 1
		j = 0
		1.upto(n1) do |i|
			m = n2
			while j >= m
				j -= m
				m /= 2
			end
			j += m
			src[i],src[j] = src[j],src[i] if j > i
		end
 
		# 1d(N) Ueberlagerungen
		mm = 1
		begin
			m = mm
			mm *= 2
			0.upto(m - 1) do |k|
				w = wt[ k * n2 ]
				k.step(n1, mm) do |i|
					j = i + m
					src[j] = src[i] - (temp = w * src[j])
					src[i] += temp
				end
			end
			n2 /= 2
		end while mm != n
		src
	end
end
 
class String
	# parse Complex.new.to_s
	def to_c
		m = @@PATTERN.match self
		return nil if m.nil?
		Complex.new m[1].to_f, m[2].to_f
	end
private
	# float_pat = /(-?\d+(?:\.\d+)?(?:e[+-]?\d+)?)/
	@@PATTERN = /^[ \t\r\n]*(-?\d+(?:\.\d+)?(?:e[+-]?\d+)?)?\s*((?:\s+|[+-])\d+(?:\.\d+)?(?:e[+-]?\d+)?i)?[ \t\r\n]*$/
end
 
values = []
$stdin.each_line do |l|
	c = l.to_c
	if c.nil?
		$stderr.puts "unmatched '#{l}'"
	else
		values < < c
	end
end
INVERSE = ARGV[0] == 'inverse'
$stderr.puts INVERSE ? 'inverse' : 'forward'
 
values.fft(INVERSE).each {|i| puts "#{i.real} #{i.image}i"}
]]>

WordPress messes up the angle brackets as usual, but there’s a gist for that.

Randnotiz: Die bayerische Frühgotik und der Islam

2011-03-22T23:10:46Z

Neulich war ich auf der Burg zu Burghausen¹ und entdeckte auf einem der ältesten Bilder (eine Schutzmantelmadonna) im Palas eine merkwürdige Aufschrift:

Das Bild kam unter einem Stifterbild der Hedwigskapelle¹ zum Vorschein und ist eine Rekonstruktion. Offiziell datiert ist das Bild auf

1507

Mir fiel jedoch die verblüffende Ähnlichkeit mit der Zahl 1258 in arabischer Schreibweise²

١٢٥٨

auf, was mir und meiner kunstgeschichtlichen Expertenbegleiterin keine Ruhe ließ. Ergibt eine solche Jahreszahl einen Sinn? Könnte ein Künstler in Burghausen die in Arabien gebräuchlichen Ziffern gekannt haben? Und wieso benutzte er nicht die in Europa (heute) übliche Schreibweise?

Eine zackige Wikipedia Recherche ergab:

1202 vollendete Leonardo Fibonacci³ sein Liber Abbaci³, in dem er laut Wikipedia “unter anderem die indischen Ziffern vorstellt und diese in der Tat als »indische Ziffern« und nicht als »arabische Ziffern« bezeichnet.“²,
1235 empfängt Otto II., Vater und Vorgänger von Herzog Heinrich XIII.⁴, “den aus Italien heimkehrenden Friedrich II. in Burghausen.“⁵ — Friedrich II.⁶ war immerhin Kaiser des Heiligen Römischen Reiches und hatte selbst evtl. arabische Grundkenntnisse⁷ ,
1255 entsteht die Elisabethkapelle im Palas der Burg zu Burghausen: “Die innere Schlosskapelle aus diesem Jahr gilt als die älteste frühgotische Kirche im südbayerischen Raum.”¹ – gebaut von Herzog Heinrich XIII.

Fazit

Spannend wäre eine Verbindung zwischen Burghausen und Fibonacci abzuklopfen,
- rege Verbindungen nach Italien bestanden allemal und lassen sich sicher durch kunsthistorische Analysen oder Wirtschaftsbücher überprüfen,
- eine Eingrenzung des Künstlermilieus und Untersuchung ob’s Verbindungen zu Algebra oder Geometrie aus dem Liber Abbaci gibt,
zumindest zur bayerischen Frühgotik gehört islamische Gelehrsamkeit allemal, das sagt der Laptop,

aber so posaunt’s aus der Lederhose

Quellen

Seite „Burg zu Burghausen“. In: Wikipedia, Die freie Enzyklopädie.
Bearbeitungsstand: 11. Januar 2011, 12:48 UTC.
URL: http://de.wikipedia.org/w/index.php?title=Burg_zu_Burghausen&oldid=83748961
(Abgerufen: 21. März 2011, 16:00 UTC)
Seite „Indische Zahlendarstellung“. In: Wikipedia, Die freie Enzyklopädie.
Bearbeitungsstand: 15. März 2011, 10:46 UTC.
URL: http://de.wikipedia.org/w/index.php?title=Indische_Zahlendarstellung&oldid=86473530
(Abgerufen: 21. März 2011, 16:03 UTC)
Seite „Leonardo Fibonacci“. In: Wikipedia, Die freie Enzyklopädie.
Bearbeitungsstand: 19. Februar 2011, 17:59 UTC.
URL: http://de.wikipedia.org/w/index.php?title=Leonardo_Fibonacci&oldid=85499344
(Abgerufen: 21. März 2011, 16:05 UTC)
Seite „Heinrich XIII. (Bayern)“. In: Wikipedia, Die freie Enzyklopädie.
Bearbeitungsstand: 14. Februar 2011, 17:08 UTC.
URL: http://de.wikipedia.org/w/index.php?title=Heinrich_XIII._(Bayern)&oldid=85290895
(Abgerufen: 21. März 2011, 16:08 UTC)
Seite „Geschichte der Stadt Burghausen“. In: Wikipedia, Die freie Enzyklopädie.
Bearbeitungsstand: 4. März 2011, 10:16 UTC.
URL: http://de.wikipedia.org/w/index.php?title=Geschichte_der_Stadt_Burghausen&oldid=86028680
(Abgerufen: 22. März 2011, 09:02 UTC)
Seite „Friedrich II. (HRR)“. In: Wikipedia, Die freie Enzyklopädie.
Bearbeitungsstand: 21. März 2011, 06:52 UTC.
URL: http://de.wikipedia.org/w/index.php?title=Friedrich_II._(HRR)&oldid=86720244
(Abgerufen: 22. März 2011, 09:07 UTC)
Siehe hier

P.S.: Sämtliche Zitate und Quellen sind zwar nur aus Wikipedia, aber als solche ausgewiesen und verlinkt – das hier ist schließlich ein seriöser Blog und keine juristische Dissertation.

Vortrag: Index Suche mit CoreData und SQLite

2011-01-27T09:39:35Z

Gestern gab’s einen Mini-Vortrag von mir bei den CocoaHeads München:

Die Folien dazu.

AlcoCalc iPhone App online!

2011-01-13T14:04:31Z

Hurra! Vor einigen Tagen ist die erste Version des Promillerechners im erschienen. Die App merkt sich was Du wann getrunken hast und schätzt nach der Widmark Formel grob den aktuellen und fallenden Promillewert ab:

High-Res UIImage remote load

2010-11-08T10:24:22Z

Loading UIImages automatically in high-resolution works fine for locally stored images – but if you want to fetch them via remote URL you have to code yourself.

A simple, blocking but backward compatible (iOS >= 3.0, maybe even 2.0 but untested) implementation could look like this:

@implementation UIImage (MRORemote)
 
// add the @2x filename suffix
+(NSURL *)url2x:(NSURL *)url
{
  NSString *path = url.path;
  NSAssert(path != nil, @"");
  NSString *last = path.lastPathComponent;
  NSString *ext = path.pathExtension;
  NSAssert(last != nil, @"");
  NSAssert(ext != nil, @"");
  NSString *part = [last substringToIndex:MAX(0, last.length - ext.length - 1)];
  return [NSURL URLWithString:[NSString stringWithFormat:@"%@@2x.%@", part, ext] relativeToURL:[url absoluteURL]];
}
 
+(UIImage *)imageWithContentsOfURL:(NSURL *)url probe2x:(BOOL)probe2x
{
  if ( url == nil )
    return nil;
  UIScreen *screen = [UIScreen mainScreen];
  const CGFloat scale = [screen respondsToSelector:@selector(scale)] ? [screen scale] : 1.0f;
  if ( probe2x && 2.0f == scale && [UIImage respondsToSelector:@selector(imageWithCGImage:scale:orientation:)] ) {
    UIImage *img = nil;
    NSData *raw = [NSData dataWithContentsOfURL:[UIImage url2x:url]];
    if ( raw != nil )
      img = [UIImage imageWithData:raw];
    if ( img != nil )
      img = [UIImage imageWithCGImage:img.CGImage scale:scale orientation:img.imageOrientation];
    if ( img != nil )
      return img;
    NSAssert(raw == nil && img == nil, @"");
  }
  // MRLogD(@"loading %@", [url absoluteURL]);
  NSData *raw = [NSData dataWithContentsOfURL:url];
  if ( raw == nil )
    return nil;
  return [UIImage imageWithData:raw];
}
@end

Use at your will but without any warranty.

High-Res Artwork Management Automation

2010-10-24T16:45:48Z

The iPhone4 comes with a super high-res display and to leverage that encourages App Developers to provide all artwork twofold – once “normal” and once in double resolution named equally with a “@2x” suffix.

To ease my designers’ life and avoid confusion (and designers are easily confused I found) I ask them to provide the high-res artwork only and I scale it down myself. And as this is a reoccuring task, I automated via a Makefile like this:

#!/usr/bin/make
# Make help: http://www.gnu.org/software/make/manual/html_node/Phony-Targets.html#Phony-Targets
 
# Requires ImageMagick, Installation per macport: $ sudo port install imagemagick +no_x11
CONVERT	:=	convert
 
# Where are the images?
ASSETS_DIR := .
 
# Which ones? All @2x.png plus twins without @2x.png
ASSETS_HIGH	:=	$(wildcard $(ASSETS_DIR)/*@2x.png)
ASSETS_LOW	:=	$(patsubst %@2x.png,%.png,$(ASSETS_HIGH))
 
# The scaling command
%.png: %@2x.png
	convert $< -resize 50% $@
 
assets: $(ASSETS_LOW)
 
clean:
	-rm $(ASSETS_LOW)

Upgrade iOS SDK 4.0 -> 4.1 with custom location

2010-10-24T18:50:13Z

again, for my custom install location I need to prepare:

cleanly uninstall and remove cruft:

$ dir=/Users/Developer.SnowLeopard
$ sudo sh $dir/Library/uninstall-devtools
$ sudo rm -r $dir/*
$ sudo mv /Developer /Developer.deleteme

then do the custom-location install and

finally restore some hotfix softlinks:

$ dir=/Users/Developer.SnowLeopard
$ sudo ln -s $dir/Platforms /Developer/Platforms
$ sudo ln -s $dir/SDKs /Developer/SDKs
$ sudo ln -s $dir/Applications/Xcode.app /Developer/Applications/Xcode.app

Not removing the cruft will get you this quite terse error message:

XCode 3.2.4 install without prior cleaning

Vortrag: Parser mit Ragel

2010-09-22T22:02:46Z

Parser mit Ragel – komplizierte Grammatiken und rattenschnelles XML.
Warum Parser bauen?
Was spricht gegen Ad-Hoc Parsing (a.k.a. Gefrickel)?
Quelltext Impression
Wie komme ich drauf?
Anschnallen: Die Bausteine
Beispiel: SVG Path Parser
Beispiel: XML Parser
Ausblick: Zustandsautomaten
Vielen Dank

Die Folien vom Vortrag am 22. September bei den CocoaHeads München.

UILabel with a (custom) CGFont

2010-11-08T09:59:12Z

UILabel’s font property accepts UIFonts – but strange enough there’s no way to get a custom loaded CGFont (from a ttf or otf file) converted into such an UIFont. You’re stuck with the iPhone’s pre-installed fonts – at least when you have to support iOS 3.0 devices.

After googling a bit and searching Stackoverflow I found the solutions presented there not ideal or great, but too heavy weight.

So I inherited UILabel with a very lean custom class UILabelWithCGFont and overloaded it’s drawTextInRect: method like this:

-(void)drawTextInRect:(CGRect)rect
{
	MRLogD(@"(%f,%f) (%f,%f)", rect.origin.x, rect.origin.y, rect.size.width, rect.size.height);
	if ( _CGFont == NULL ) {
		[super drawTextInRect:rect];
		return;
	}
	NSAssert(_mapping != NULL, @"Mapping function pointer not set.");
 
	// prepare the target graphics context.
	const CGContextRef ctx = UIGraphicsGetCurrentContext();
	CGContextSaveGState(ctx);
	{
		// prepare the glyphs array to draw
		const NSString *txt = self.text;
		const size_t glyphCount = txt.length;
		CGGlyph glyphs[glyphCount];
		{
			// turn the string txt into glyphs (indices into the font):
			// give non-allocating unicode character retrieval a try:
			const UniChar *raw_unichars = CFStringGetCharactersPtr( (CFStringRef)txt );
			const UniChar *unichars = raw_unichars == NULL ? malloc( glyphCount * sizeof(UniChar) ) : raw_unichars;
			NSAssert(unichars != NULL, @"unichars not allocated");
			if ( raw_unichars == NULL )
				CFStringGetCharacters( (CFStringRef)txt, CFRangeMake(0, txt.length), (UniChar *)unichars );
			for ( int i = glyphCount - 1; i &gt;= 0; i-- )
				glyphs[i] = _mapping(unichars[i]);
			if ( raw_unichars == NULL )
				free( (void *)unichars );
		}
 
		CGContextSetFont(ctx, _CGFont);
		CGContextSetFontSize(ctx, self.font.pointSize);
		CGContextSetTextMatrix( ctx, CGAffineTransformMake(1.0, 0.0, 0.0, -1.0, 0.0, 0.0) );
 
		// first print 'invisible' to measure size:
		CGContextSetTextDrawingMode(ctx, kCGTextInvisible);
		const CGPoint pre = CGContextGetTextPosition(ctx);
		CGContextShowGlyphs(ctx, glyphs, glyphCount);
		const CGPoint post = CGContextGetTextPosition(ctx);
		// restore text position
		CGContextSetTextPosition(ctx, pre.x, pre.y);
 
		// centered horizontal + vertical:
		NSAssert( (int)rect.origin.x == 0, @"origin.x not zero" );
		NSAssert( (int)rect.origin.y == 0, @"origin.y not zero" );
		NSAssert(self.baselineAdjustment == UIBaselineAdjustmentAlignCenters, @"vertical alignment not 'center'");
		NSAssert(self.textAlignment == UITextAlignmentCenter, @"horizontal alignment not 'center'");
		const CGPoint p = CGPointMake( ( rect.size.width - (post.x - pre.x) ) / 2, (rect.size.height + self.font.pointSize + pre.y) / 2 );
 
		// finally render it to the graphics context:
		CGContextSetTextDrawingMode(ctx, kCGTextFill);
		CGContextSetFillColorWithColor(ctx, self.textColor.CGColor);
		CGContextShowGlyphsAtPoint(ctx, p.x, p.y, glyphs, glyphCount);
	}
	CGContextRestoreGState(ctx);
}

Usage: Just turn the UILabel instances in Interface Builder into UILabelWithCGFont and implement the UIViewController::viewDidLoad method like this:

CGGlyph unicode2glyphDeutscheDruckschrift(UniChar c)
{
	if ( '0' < = c && c <= '9' )
		return c + (16 - '0');
	if ( 'A' <= c && c <= 'Z' )
		return c + (32 - 'A');
	if ( 'a' <= c && c <= 'z' )
		return c + (58 - 'a');
	return 0;
}
 
-(void)viewDidLoad
{
	[super viewDidLoad];
	...
	[fontLabel setFontFromFile:@"DeutscheDruckschrift" ofType:@"ttf" mapping:unicode2glyphDeutscheDruckschrift];
	...
}

See this github gist for the complete implementation.

The mapping from Unicode character codes to glyph indices (inside the font description) currently is done via a C mapping function you have to provide a function pointer for. A later implementation could map the unicode character code to the glyph name and leverage CGFontGetGlyphWithGlyphName and render the custom mapping function obsolete.

Visualise macports dependencies

2011-01-25T09:42:36Z

to clean up your installed macports and remove cruft you need to uninstall them in the correct order – according to their dependencies.

A graphical visualisation might help doing so:

Call

$ ./port-deps2dot.rb | dot -Tpdf -o port-deps.pdf ; open port-deps.pdf

with the ruby script port-deps2dot.rb (github gist) as follows:

#!/usr/bin/ruby -w
 
# visualize macports dependencies.
# pipe the result through graphviz, e.g.
# $ ./port-deps2dot.rb | dot -Tpdf -o port-deps.pdf ; open port-deps.pdf
 
def scan_deps
	pat = /^([^:]+):(.+)$/
	name = ''
	deps = []
	IO.popen('port info --name --pretty --depends installed') do |f|
		f.each_line do |l|
			case l
				when /^--$/
					yield name, deps
					name = ''
					deps = []
				when /^([^:]+):(.+)$/
					if 'name' == "#$1"
						name = "#$2".strip
					else
						deps.concat("#$2".split(/\s*,\s*/))
					end
				else
					raise "Fall-through for '#{l}'"
			end
		end
	end
end
 
all = {}
 
scan_deps do |name,deps|
	d = all[name]
	all[name] = d = [] if d.nil?
	deps.collect! {|i| i.strip}
	d.concat deps
	d.sort!
	d.uniq!
end
 
head = < <END_OF_STRING
#!/usr/bin/dot -Tpdf -o port-deps.pdf
/*
	See http://www.graphviz.org/Documentation.php
*/
digraph "port deps" {
	rankdir=LR;
    label="port deps";
    node [style=filled,fillcolor=lightblue,shape=ellipse];
    top_level [shape=point];
END_OF_STRING
 
puts head
 
all.keys.sort.each do |name|
	deps = all[name]
	if deps.count > 0
		deps.each {|d| puts "\t\"#{name}\" -> \"#{d}\";" }
	else
		puts "\t\"#{name}\";"
	end
end
 
foot = < <END_OF_STRING
}
END_OF_STRING
 
puts foot

Develop with iOS 4 SDK on an OS 3.0 Device

2010-06-23T11:32:27Z

After upgrading to iOS 4.0 SDK, iPhone OS 3.0 is no longer a valid “Base SDK”. Naively switching to iPhone 3.2 prevents deployment to a 3.0 device.

But such “Project -> Edit Project Settings” work out fine:

iPhoneOS 3.0 development device with iOS 4.0 SDK

“Deployment Target” hint found here.

iOS 4 Simulator crash (when installed to custom folder)

2010-06-23T11:23:41Z

just downloaded and installed the iOS 4 SDK and as my root OS X partition is rather (too) small, I put it into a custom location /Users/Developer.Snowleopard/.

This causes the iPhone Simulator to crash and compiling gives an error like:

ibtool failed with exception: Interface Builder encountered an error communicating with the iPhone Simulator. If you choose to file a crash report or radar for this issue, please check Console.app for crash reports for "Interface Builder Cocoa Touch Tool" and include their content in your crash report.
...
dyld: Library not loaded: /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator4.0.sdk/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
  Referenced from: /Users/Developer.SnowLeopard/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator4.0.sdk/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
  Reason: image not found

The cure – a softlink:

$ ln -s /Users/Developer.SnowLeopard/Platforms /Developer/Platforms

XCode: missing “deprecated” warnings

2010-06-23T09:23:34Z

when developing for long-term use, you want to use APIs that aren’t likely to be removed soon, a.k.a. “deprecated”.

So, don’t use downward compatible calls below a point you really aim for.

XCode helps with compiler warnings about “deprecated” calls – if “Project -> Edit Project Settings -> GCC_WARN_ABOUT_DEPRECATED_FUNCTIONS” is set:

Compiler setting: Warn about deprecated calls

But be careful, this complains e.g. here only once while initWithFrame: is also deprecated (compiling for iPhone OS 3.0):

No "deprecated" warning on initXY

As Kevin Ballard pointed out at Stackoverflow, this is because [AnyClass alloc] returns a type id – which doesn’t know about it’s interface.

To get this kind of compiler warnings, you have to type-cast the [AnyClass alloc] like this:

[(AnyClass*)[AnyClass alloc] initXY];

Maybe time for a macro?

The macro could look like

// http://blog.mro.name/2010/06/xcode-missing-deprecated-warnings/
#define alloc(c)	((c*)[c alloc])

Migrate your codebase via XCode “Edit -> Find in Project” with search pattern \[\s*([^\[\]]+)\s+alloc\s*\] and replacement pattern alloc(\1).

Simple HTTP Access Authorisation

2010-06-14T10:56:49Z

sometimes you may want to lock down RESTful APIs or plain HTTP GET resources for authorised access by your own client software only, without requiring authentication. You don’t know who (not authenticated), but you know she may access (is authorised).

If the server has a valid SSL certificate based on a root certificate pre-installed on the iPhone among the simplest ways to do it are:

HTTP Basic Authentication with static username + password. This requires just a .htaccess configuration setting and you’re done.

send a custom HTTP Request Header with a secret token, also just a .htaccess rewrite setting required:

RewriteEngine On
RewriteCond %{HTTP:My-Secret-Token} !=WRdsWXwwTZjEIRrgD5tODVf0U
RewriteRule ^.*$ - [forbidden,last]
# Test: $  curl --header "My-Secret-Token:WRdsWXwwTZjEIRrgD5tODVf0U" http://myserver.example.com/demo/

Decompiling an App may raise the bar high enough though hard-coded secrets surely aren’t bulletproof Secret Service grade quality. If you don’t want the password or secret token as literal string inside the App, synthesize it at runtime.

If your transport channel isn’t confidential (e.g. plain HTTP, not HTTPS) you might think about Digest Authentication or a custom implemented CRAMish mechanism which I will not go into in this post.

P.S.: Here are some really nice .htaccess examples.

Automatic gzip compression for Apache2 Webservers

2010-06-13T11:55:25Z

after failing and failing again in the last months, I finally got it with the help of http://www.debian-administration.org/articles/137

The .htaccess configuration

AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css application/javascript

requires Apache’s mod_deflate enabled via

$ a2enmod deflate
Module deflate installed; run /etc/init.d/apache2 force-reload to enable.

Check the result with http://www.gidnetwork.com/tools/gzip-test.php

Caution: There seems to be an If-Modified-Since/Last-Modified HTTP 304 bug in mod_deflate, so better don’t use it for large, rarely changing files.

iPhone: libxml2 & RELAX NG validation

2010-06-14T12:37:40Z

Having a validating parser in place can reduce the required code to parse XML a lot – you know very well what you actually get. As mentioned in my last post about RELAX NG & trang, I prefer RELAX NG over W3C XML Schema – which doesn’t matter anyway because Apple’s suggested XML parser doesn’t validate at all.

So we have to go one level deeper and have a look at libxml2.

Apple’s example “XmlPerformance” helped to get started, but didn’t do the trick because libxml2 allows validation for xmlDocPtr or xmlTextReader but not for SAX parsers as used in the example.

The libxml2 examples didn’t help me too much either, but luckily there’s xmllint available in source (OSS just rocks) which does almost what we want. It first parses the XML into a xmlDocPtr and validates afterwards – and it does so for a reason:

You can have a validating xmlTextReader (via xmlTextReaderRelaxNGSetSchema), but it won’t detect IDREFs missing their referred to ID and the error messages lack the name of the failing item. BTW – when validating against a W3C schema this ID/IDREF check isn’t available yet.

I finally discarded streaming XML parsing in favour of validation and “push” parsing (nice for data coming in over the wire) and did:

load the RELAX NG regular form schema (watch out for the assignment of relaxngschemas) – similar to xmllint schema loading,
push the raw XML data into a xmlDocPtr (xmlCreatePushParserCtxt) exactly like xmllint,
validate the in-memory document (xmlRelaxNGValidateDoc),
turn it into a xmlTextReader,
process the reader.

Wrap up:

if you want full RELAX NG validation with libxml2 v2.7.3, forget about streamed parsing,
wrap the document into a xmlTextReader if you want a SAXish programming model.

I may prepare and publish a MroLibxml2Parser inheriting NSXMLParser and firing it’s callbacks in order to easily switch validating and non-validating parser implementations, but this has to wait a bit. Stay tuned.

XML Toolbox: RELAX NG & trang

2011-02-12T18:02:33Z

e.g. when handling RESTful APIs you may want to validate the response XML – a custom one in most cases.

I typically use tools already installed on every Mac and fire a http GET request with curl and immediately check it with xmllint like

$ curl http://www.heise.de/newsticker/heise-atom.xml | xmllint --format --schema myschema.xsd -

But I just don’t like to create and edit W3C XML Schemas – the notorious angle brackets hurt my eyes and the redundant element names hide the real stuff in tons of ever same text. Neither do I like to click through graphical schema editors and getting lost hunting for hidden settings and property dialogs.

A minimal and naive schema validating the above example Atom feed (and simply created from the feed itself with trang, see below) as W3C Schema looks like this:

Naive Atom W3C Schema

Here comes in RELAX NG, especially it’s “compact form“, which is just what I like – a concise, BNF-ish syntax. It was designed by Murata Makoto and James Clark, Technical Lead of the XML Working Group back when XML was created and father of the famous expat parser.

The very same schema as above as RELAX NG boils down to ½ the lines and about ⅓ of the characters without a single angle bracket:

default namespace = "http://www.w3.org/2005/Atom"
 
start =
  element feed {
    title,
    element subtitle { text },
    link+,
    updated,
    element author {
      element name { text }
    },
    id,
    element entry { title, link, id, updated }+
  }
title = element title { text }
link =
  element link {
    attribute href { xsd:anyURI },
    attribute rel { xsd:NCName }?
  }
updated = element updated { xsd:dateTime }
id = element id { xsd:anyURI }

And as libxml2 and therefore xmllint supports RELAX NG, you can use the regular syntax to validate like in the beginning, but with a much more editable schema:

$ curl http://www.heise.de/newsticker/heise-atom.xml | xmllint --format --relaxng myschema.rng -

trang

is a schema converter for RELAX NG written in Java which I wrapped inside a bash script:

#!/bin/sh
java -jar `dirname $0`/trang-20090818/trang.jar $@

Writing a new schema from scratch can be much more convenient if you have a bunch of XML files you can feed into trang:

$ trang *.xml myschema.rnc

then refine the resulting schema in compact form and finally turn it into the regular form:

$ trang myschema.rnc myschema.rng

Trang also serves me as a schema indenter by converting from compact to regular and back.

BUT: trang converts RELAX NG into W3C but not vice versa.

Deep validation

Validating XML documents shouldn’t stop with elements and attributes but rather leverage XML Schema Datatypes and apply e.g. regular expressions

  element uuid {
    xsd:string {
 
      ## A UUID
      pattern =
        "[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}"
    }
  }

or range constraints

        element year {
          xsd:unsignedShort { minInclusive = "1900" maxInclusive = "2100" }
        }

P.S.: For a more complete Atom RELAX NG schema see here or ask your search engine of choice.

TextWrangler + tidy

2010-05-21T21:59:42Z

as I didn’t get TidyService to work correctly with UTF8 umlauts, I created a UNIX Shell Script wrapper for html tidy as it comes with OS X that does the job at least for TextWrangler:

open TextWranglers “Unix Filters Folder”
create a file named e.g. “Tidy Html.sh”,

paste the following lines into the file and save it:

#!/bin/sh
# run "tidy" on the file given as 1st (and only) parameter.
#
/usr/bin/tidy -utf8 -asxhtml -indent -wrap 100 -quiet "$1" 2> /dev/null

now you can run tidy on files opened in TextWrangler, even remote files.
assign a keyboard shortcut (I used CTRL-T in the shot above) via the “Unix Filters” Palette:

Immo Betrug / Scam

2010-05-12T10:43:26Z

auf Wohnungssuche begegnet man lustigen Sachen. Da war dieses extrem billige Angebot (2 Zi, 80m² für 550€) mit spannender Adresse, aber fragen kostet ja nix.

Die erste Mail der Anbieterin war dann:

Hi,

Thanks for your interest. The apartment is still available. I moved recently with my job in London, United Kingdom, but the rent is high here, I decided to offer to rent the apartment I have in Munchen . I rented the apartment for maximum 10 years … This is the period that I have a contract here, but I can rent it for a shorter period also. I own the ground and is exactly as in the photos. The rent for 1 month is 550 EUR and does not include all the utilities you (water, electricity, Internet, cable). You can enter the apartment on the same day when receiving the keys .. The only problem is that i had to move with my job to United Kingdom, London where i am now and i left the keys and contract already signed by me at a company called Rent.com and they will handle the payment and delivery for us. If you want to know more about how this deal can work please get back to me ASAP and i will send you the details step by step. Thank you and hope to hear back from you.

My German is limited at just a few words so i wish a lot to continue our conversation in English.

dann ging’s 2x per Mail hin und her,

Hello,

If we decide to proceed with this transaction, I will have to contact Rent.com and provide them all the necessary information, so they can start the process right away. I will need your full name and address for that. You will receive a notification from Rent.com shortly after that, together with all the instructions to follow and the invoice as well.

Regarding the payment, you will be instructed to deposit the money to a Rent.com account. They will hold and insure your money until you check the apartment and decide if you want take it or not. That is how their buyer protection policy works. As far as my concerns, I will be glad to know that Rent.com has the possession of the money during the delivery period. That is my insurance.

As soon as the funds have been deposited into their account, they will immediately start the shipping process.

The keys and rental contract will be delivered at your address in no more than 3 working days. You will be given a ten days inspection period from the day you receive the keys and contract at home. If you decide to hold the apartment, then you will have to authorize Rent.com to release the funds to me, and the transaction will be completed. If you will not be satisfied with the apartment, you will be able to send the keys and contract back through the same service and ask Rent.com to return the funds to you.

Through I am sure you will love the apartment, it is good to know that you do have this second option available. If you wish to proceed with renting the apartment, please provide me your full name and address so I can initiate the deal through Rent.com right away.

I am looking forward to hearing from you.

Regards,

die Anbieterin wollte nicht anrufen aber eine Zahlungsanweisung kam, die mich dann richtig stutizg machte, weil

der ganze Ablauf eh schon recht seltsam war,
deren Absenderemail “rentcom@consultant.com” nicht mal von rent.com kam,
rent.com außerhalb USA gar nix macht,
die Einlieferungs-IP der Zahlungsanweisung die gleiche wie die der Mails von der Anbieterin war,
ja sogar “In-Reply” auf die vorherigen Mails,
das Geld an jemanden anderen gehen sollte,
an der Wohnungs-Adresse wohl gar keine Wohnungen sind,
ein Spezl mich auf die Warnungen bei rent.com und einen Bericht bei der BBC hinwies.

Nach meiner Frage nach dem Namen des Hausmeisters war dann Funkstille in London.

Zu guter letzt hat mich Immoscout24 heute nochmal per Mail vor Betrügern gewarnt.