<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" media="screen" href="/~d/styles/rss2full.xsl"?><?xml-stylesheet type="text/css" media="screen" href="http://feeds.feedburner.com/~d/styles/itemcontent.css"?><rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:wfw="http://wellformedweb.org/CommentAPI/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:sy="http://purl.org/rss/1.0/modules/syndication/" xmlns:slash="http://purl.org/rss/1.0/modules/slash/" xmlns:feedburner="http://rssnamespace.org/feedburner/ext/1.0" version="2.0">
<channel>
<title>Home of Emanuel Ferm</title>

<link>http://eferm.com</link>
<description>Website of Emanuel Ferm, Oxford grad student, where he blogs about ideas, observations and technology.</description>
<lastBuildDate>Tue, 28 Feb 2012 22:55:54 +0000</lastBuildDate>
<language>en</language>
<sy:updatePeriod>hourly</sy:updatePeriod>
<sy:updateFrequency>1</sy:updateFrequency>
<generator>http://wordpress.org/?v=3.3.2</generator>
<atom10:link xmlns:atom10="http://www.w3.org/2005/Atom" rel="self" type="application/rss+xml" href="http://feeds.feedburner.com/eferm" /><feedburner:info uri="eferm" /><atom10:link xmlns:atom10="http://www.w3.org/2005/Atom" rel="hub" href="http://pubsubhubbub.appspot.com/" /><item>
<title>Machine Learning cheat sheet</title>
<link>http://feedproxy.google.com/~r/eferm/~3/8XnMHMldBzA/</link>
<comments>http://eferm.com/machine-learning-cheat-sheet/#comments</comments>
<pubDate>Wed, 04 May 2011 23:04:59 +0000</pubDate>
<dc:creator>Emanuel</dc:creator>
<category>
<![CDATA[Uncategorized]]>
</category>
<category>
<![CDATA[linkedin]]>
</category>
<guid isPermaLink="false">http://eferm.com/?p=2612</guid>
<description>
<![CDATA[For a recently taken course in Machine Learning, a substantial part involved learning and applying linear classifiers and clustering algorithms on smaller data sets. In order to summarise the most important material, I created a cheat sheet in LaTeX. I figured someone else might appreciate it as well, so why not make it available for [...]]]>
</description>
<content:encoded><![CDATA[
<p>For a recently taken course in <a href="http://www.comlab.ox.ac.uk/teaching/courses/2010-2011/machinelearning/">Machine Learning</a>, a substantial part involved learning and applying linear classifiers and clustering algorithms on smaller data sets. In order to summarise the most important material, I created a cheat sheet in LaTeX. I figured someone else might appreciate it as well, so why not make it available for more people than myself?</p>
<p><img src="http://static.eferm.com/wp-content/uploads/2011/05/cheat2.png" alt="cheat sheet preview" title="cheat" width="600" height="425" class="alignnone size-full wp-image-2641" /></p>
<p><code>.pdf</code> can be downloaded <a href='http://static.eferm.com/wp-content/uploads/2011/05/cheat3.pdf'>here</a>.<br />
<code>.tex</code>-file is on Github <a href="https://github.com/Emore/mlcheatsheet">here</a>; feel free to modify or add information. Please let me know if you find mistakes!</p>
<p><em>Note that his document was really only created for my own study purposes, and hence might be of limited use for others. Hopefully not, though.</em></p>
<p>EDIT: Discussion on Hacker News: <a href="http://news.ycombinator.com/item?id=2515612">http://news.ycombinator.com/item?id=2515612</a></p>
 <p><a href="http://eferm.com/?flattrss_redirect&amp;id=2612&amp;md5=d61663a7753ae5c6ef206837fb89d829" title="Flattr" target="_blank"><img src="http://eferm.com/wp-content/plugins/flattr/img/flattr-badge-large.png" alt="flattr this!"/></a></p>
<img src="http://feeds.feedburner.com/~r/eferm/~4/8XnMHMldBzA" height="1" width="1"/>]]></content:encoded>
<wfw:commentRss>http://eferm.com/machine-learning-cheat-sheet/feed/</wfw:commentRss>
<slash:comments>18</slash:comments>
<atom:link rel="payment" href="https://flattr.com/submit/auto?user_id=Emore&amp;popout=1&amp;url=http%3A%2F%2Feferm.com%2Fmachine-learning-cheat-sheet%2F&amp;language=en_GB&amp;category=text&amp;title=Machine+Learning+cheat+sheet&amp;description=For+a+recently+taken+course+in+Machine+Learning%2C+a+substantial+part+involved+learning+and+applying+linear+classifiers+and+clustering+algorithms+on+smaller+data+sets.+In+order+to+summarise+the+most...&amp;tags=linkedin%2Cblog" type="text/html" />
<feedburner:origLink>http://eferm.com/machine-learning-cheat-sheet/</feedburner:origLink></item>
<item>
<title>Implementing durability for in-memory databases, on SSDs</title>
<link>http://feedproxy.google.com/~r/eferm/~3/0G9O3URqKIc/</link>
<comments>http://eferm.com/implementing-durability-for-in-memory-databases-on-ssds/#comments</comments>
<pubDate>Tue, 12 Apr 2011 15:18:01 +0000</pubDate>
<dc:creator>Emanuel</dc:creator>
<category>
<![CDATA[Uncategorized]]>
</category>
<category>
<![CDATA[linkedin]]>
</category>
<guid isPermaLink="false">http://eferm.com/?p=2516</guid>
<description>
<![CDATA[As the examination for a recently completed course in Database Systems Implementation, students had to implement a durable, high-throughput, in-memory key/value database for strings, coincidently the same problem as this year&#8217;s SIGMOD programming contest. I thought I&#8217;d present aspects of my own implementation of durability, focusing on problems I encountered and how I solved them. [...]]]>
</description>
<content:encoded><![CDATA[
<p>As the examination for a recently completed course in <a href="http://www.comlab.ox.ac.uk/teaching/courses/2010-2011/databasesystemsimplementation/">Database Systems Implementation</a>, students had to implement a durable, high-throughput, in-memory key/value database for strings, coincidently the same problem as this year&#8217;s <a href="http://db.csail.mit.edu/sigmod11contest/">SIGMOD programming contest</a>. I thought I&#8217;d present aspects of my own implementation of durability, focusing on problems I encountered and how I solved them. I also relate parts of my solution to existing <a href="http://en.wikipedia.org/wiki/NoSQL_(concept)">NoSQL</a> databases as well as how SSD disks can reduce the sophistication needed for a good-enough solution.</p>
<p>Perhaps the main problem of the programming contest lies in implementing the in-memory data structure itself, with issues such as concurrency control and efficient string comparisons. I will, however, assume such an arbitrary data structure, and focus this post on implementing durability.</p>
<p><strong>Disclaimer</strong><br />
This is by no means efficient, beautiful or advanced, and ideas should probably not be copied without first dissecting them with experienced eyes. I learned as I proceeded, and I have essentially no prior knowledge of databases up until this course and assignment. This post is written for unexperienced enthusiasts like myself; for others there is probably little new.<br />
<span style="float:right; text-align:right;"><span id="more-2516"></span></span></p>
<p><br/></p>
<h2>Definitions</h2>
<p>First, a couple of loose definitions, as this post shouldn&#8217;t require too much prior knowledge within databases:</p>
<p><strong>Durability</strong>: the D in <a href="http://en.wikipedia.org/wiki/ACID">ACID</a>. For our purposes, an update to the database is physically written to persistent storage (hard disks or SSDs), and thus recoverable in case of a system crash.</p>
<p><strong>Throughput</strong>: a measure of performance, to be maximised. Requests served per second.</p>
<p><strong>In-memory database</strong>: the database resides fully in memory, meaning its size is small enough to be contained in RAM. (Conversely, most traditional databases are built for persistent storage, due to the larger available capacity of hard drives.)</p>
<p><strong>Key/value database</strong>: only a key and its corresponding value is stored, as opposed to the relational model which stores data as tables with multiple attributes.</p>
<p><strong>SSD</strong>: Solid State Drive, flash-based storage without moving parts. Faster in every aspect but much more expensive than hard drives.</p>
<p><br/></p>
<h2>A naive in-memory only solution</h2>
<p>If durability is no requirement, an in-memory database is simply any data structure mapping string keys to string values, which also supports lexicographical iteration over elements. For example, Java&#8217;s <a href="http://download.oracle.com/javase/6/docs/api/java/util/TreeMap.html"><code>TreeMap&lt;K,V&gt;</code></a> would be perfectly fine. The database is then simply an adapter containing an instance of TreeMap, mapping inserts to <code>put(K,V)</code> etc. The only potentially difficult part involves concurrency. </p>
<p><strong>Concurrency</strong><br />
Any Java person has probably been thrown a <code>ConcurrentAccessException</code> at some point, and if you code C you probably just get seg faults if your data structure is not thread safe. In a highly concurrent environment, in which a database lives, updates to the memory structure need to support concurrent access.</p>
<p>Three broad strategies exist: </p>
<ol style="margin-bottom:10px;">
<li>associate a <a href="http://en.wikipedia.org/wiki/Mutual_exclusion">mutex</a> with the whole structure, only allowing access to one thread at a time; </li>
<li>implement a more fine-grained locking strategy, for example with upgradeable read/write locks as used in traditional <a href="http://en.wikipedia.org/wiki/B-tree">B trees</a>; </li>
<li>use a lock free data structure, such as <a href="http://download.oracle.com/javase/6/docs/api/java/util/concurrent/ConcurrentSkipListMap.html"><code>ConcurrentSkipListMap&lt;K,V&gt;</code></a>.</li>
</ol>
<p><strong>Transactions</strong><br />
With upport for concurrent access, a transaction for an in-memory only data structure roughly follows the following pseudo code:</p>
<p><code>&nbsp;&nbsp;&nbsp;&nbsp;lock(data_structure)</code><br />
<code>&nbsp;&nbsp;&nbsp;&nbsp;update(data_structure)</code><br />
<code>&nbsp;&nbsp;&nbsp;&nbsp;unlock()</code><br />
<code>&nbsp;&nbsp;&nbsp;&nbsp;return</code></p>
<p>That is, each update is synchronous within each thread, and each operation is considered a transaction of its own.</p>
<p>To make things simple, I used <a href="http://fallabs.com/kyotocabinet/">Kyoto Cabinet</a>, a <a href="http://en.wikipedia.org/wiki/Dbm">DBM</a> with an excellent C/C++ API,  for the in-memory data structure and concurrency control. More specifically, I used a B tree with node-level read/write locks, provided by Kyoto&#8217;s GrassDB class. This way, multiple threads may read simultaneously from the data structure.</p>
<p><br/></p>
<h2>High-level problem analysis</h2>
<p>With full durability as a requirement, each update to the in-memory data structure needs to be written to disk as well, before success or failure is returned. Furthermore, each transaction is synchronous which poses a surprisingly interesting constraint. With the goal of maximised throughput, it is probably desirable to decouple writing to disk from accessing the in-memory data structure &#8212; something which I initially failed at, as illustrated in the following two invalid solutions. (Each disk write is assumed to be atomic.)</p>
<p><strong>Solution 1</strong><br />
This solution is not valid since the database does not maintain <a href="http://en.wikipedia.org/wiki/Consistency_(database_systems)">consistency</a>: while a slow disk write is under way in thread 1, other threads may read the update from memory even though thread 1 has not committed (returned &#8216;success&#8217; or &#8216;failure&#8217;):</p>
<p><code>&nbsp;&nbsp;&nbsp;&nbsp;lock(data_structure)</code><br />
<code>&nbsp;&nbsp;&nbsp;&nbsp;update(data_structure)</code><br />
<code>&nbsp;&nbsp;&nbsp;&nbsp;unlock()</code><br />
<code>&nbsp;&nbsp;&nbsp;&nbsp;update(disk)</code><br />
<code>&nbsp;&nbsp;&nbsp;&nbsp;return</code></p>
<p><strong>Solution 2</strong><br />
This is essentially as fast as updating in-memory only, but not valid since the disk update occurs asynchronously in a separate thread, &#8216;success&#8217; being returned before writing to disk has finished:</p>
<p><code>&nbsp;&nbsp;&nbsp;&nbsp;lock(data_structure)</code><br />
<code>&nbsp;&nbsp;&nbsp;&nbsp;update(data_structure)</code><br />
<code>&nbsp;&nbsp;&nbsp;&nbsp;unlock()</code><br />
<code>&nbsp;&nbsp;&nbsp;&nbsp;dispatch_new_thread(update(disk))</code><br />
<code>&nbsp;&nbsp;&nbsp;&nbsp;return</code></p>
<p>A naive solution to both scenarios above associates an exclusive lock with <em>both</em> updating memory and updating disk:</p>
<p><code>&nbsp;&nbsp;&nbsp;&nbsp;exclusive_lock()</code><br />
<code>&nbsp;&nbsp;&nbsp;&nbsp;update(data_structure)</code><br />
<code>&nbsp;&nbsp;&nbsp;&nbsp;update(disk)</code><br />
<code>&nbsp;&nbsp;&nbsp;&nbsp;unlock()</code><br />
<code>&nbsp;&nbsp;&nbsp;&nbsp;return</code></p>
<p>This, however, has several drawbacks: 1) sophisticated concurrency control for the chosen data structure is not leveraged; 2) no read may be served from memory while any disk write is in progress. </p>
<p>Instead, I chose to write to disk <em>before</em> updating the memory, solving the consistency problem <em>and</em> allowing for concurrent access to memory. This way, the updating of memory and disk can be optimised separately. </p>
<p><code>&nbsp;&nbsp;&nbsp;&nbsp;update(disk)</code><br />
<code>&nbsp;&nbsp;&nbsp;&nbsp;lock(data_structure)</code><br />
<code>&nbsp;&nbsp;&nbsp;&nbsp;update(data_structure)</code><br />
<code>&nbsp;&nbsp;&nbsp;&nbsp;unlock()</code><br />
<code>&nbsp;&nbsp;&nbsp;&nbsp;return</code></p>
<p>Notice the similarity to <a href="http://en.wikipedia.org/wiki/Write-ahead_logging">Write Ahead Logging</a>. Curiously, this is also the same strategy used by Facebook&#8217;s Cassandra (see Lakshman &#038; Malik, <a href="http://www.cs.cornell.edu/projects/ladis2009/papers/lakshman-ladis2009.pdf">Cassandra &#8212; A Decentralized Structured Storage System</a>). An additional logging step should be added before returning, confirming the update to memory in order to guarantee atomicity; if the system crashes in-between writing to disk and updating memory, the key/value pair written to disk will be restored to memory. </p>
<p>The problem of implementing durability is now fully decoupled from whatever in-memory data structure is used, and can be optimised independently on its own.</p>
<p><br/></p>
<h2>File organisation for SSDs</h2>
<p>Since the database resides fully in memory, no individual pages are read to and from disk. The need for block-sized pages, clustered files and other such traditional database considerations are thus not needed; a separate recovery feature need not be very efficient. A simple transaction log is sufficient for maintaining a persistent state of the database.</p>
<p>Due to the large rotational delay and seek time of hard drives, transaction logs are typically append-only, minimising the time to write an update, leveraging the fast sequential write speed of hard drives. This is coupled with periodical checkpointing of the database state, so that the logs don&#8217;t keep growing ad infinitum &#8212; if everything was a log, deletes would simply <em>add</em> data rather than remove it.</p>
<p>SSDs, however, have no rotational delay and have seek times orders of magnitude smaller than those of hard drives. Furthermore, the random write speed of a state-of-the-art SSD is much faster than even the sequential write speed of the fastest hard drives. AnandTech offers great <a href="http://www.anandtech.com/show/4159/ocz-vertex-3-pro-preview-the-first-sf2500-ssd">insight and graphs</a> on this.</p>
<p>For SSDs, a complex architecture involving both logging and checkpointing can be substituted by in-place writing of updates, by maintaining a slot directory of free (deleted) positions within a persistent file, as well as a lookup table mapping keys to offsets in the file. When writing to file, the persistence layer first checks these data structures, before appending to the end:</p>
<p><script src="https://gist.github.com/915563.js?file=persistence.cc"></script></p>
<p>This way, the file &#8216;garbage collects&#8217; itself, and all free space is hopefully reused (depending on the sizes of keys and values). One mechanism that could be added, though, is some sort of garbage collection of large chunks of free space, since the file-size is never reduced and is bounded by the largest amount of data that has resided in the database at any point in time. Nevertheless, the file organisation leverages the inherent advantages of SSDs and provides a simple, good-enough solution to persistent writing.</p>
<p><br/></p>
<h2>Issues with durability</h2>
<p>In C, I/O usually involves <code>write()</code> and <code>read()</code> operations. For my purposes, apart from a recovery phase, only writing to disk is necessary as read operations are served directly from memory. However, simply invoking <code>write()</code> does not guarantee durability, since a system crash may occur before the disk cache is flushed to disk. Any update that is reported as complete may then, in fact, be lost given some bad luck. </p>
<p>To alleviate this, there exists the database programmers&#8217; favourite function, <code>fsync()</code>. It essentially forces file updates to physical disk, sacrificing performance for guaranteed durability. The time for a <code>write()</code> is negligible compared to an <code>fsync()</code>, which typically takes tens- to hundreds of milliseconds to complete. </p>
<p><strong>Dealing with <code>fsync()</code> is thus crucial to the overall performance of the database.</strong></p>
<p>I considered three strategies for making the updates to memory persistent:</p>
<ol style="margin-bottom:10px;">
<li>Use Kyoto Cabinet&#8217;s <code>PolyDB::OAUTOSYNC</code> mode for letting Kyoto handle physical file writing as well as file organisation. This was, unfortunately, way too slow. </li>
<li>Implement my own file organisation, and open the persistent file in <code>O_DIRECT</code> mode, bypassing any buffer. Linus Torvalds convinced me this <a href="http://lkml.org/lkml/2007/1/10/233">wasn&#8217;t such a good idea</a>:<br />
<blockquote><p>
This is your brain: O<br />
This is your brain on O_DIRECT: .</p>
<p>Any questions?
</p></blockquote>
</li>
<li>Implement my own file organisation, but <code>fsync()</code> manually. This is what I opted for. </li>
</ol>
<p><strong>A comment regarding the cost of an <code>fsync()</code></strong><br />
A simple, linear formula for the cost of an <code>fsync()</code> is given in a <a href="http://tokutek.com/2010/01/high-insertion-rates-into-a-tokudb-table-with-durable-transactions/">blog post</a> at Tokutek:</p>
<p><code>&nbsp;&nbsp;&nbsp;&nbsp;fsync() time = N/R + K</code></p>
<blockquote><p>where <code>N</code> is the amount of dirty data that needs to by written to disk, <code>R</code> is the disk write rate, and <code>K</code> is a constant time defined by the storage system, such as disk seek time.</p></blockquote>
<p>Consider what happens if each individual write operation is followed by an <code>fsync()</code>. With <code>X</code> clients each writing an update of 1 unit, the total time for all to finish is:</p>
<p><code>&nbsp;&nbsp;&nbsp;&nbsp;X(1/R + K) = X/R + XK</code></p>
<p>That is, the constant is multiplied by the number of clients. Furthermore, since each <code>write()</code> is followed by an <code>fsync()</code>, the total performance is bounded by each individual <code>fsync()</code>. This means that the system as a whole can never perform faster than the number of writes multiplied by the cost of an <code>fsync()</code>.</p>
<p>Instead, I consider batching updates together, and issuing one <code>fsync()</code> for the whole batch. Again, for <code>X</code> clients each writing 1 unit of data, this results in improved performance according to the formula:</p>
<p><code>&nbsp;&nbsp;&nbsp;&nbsp;X/R + K</code></p>
<p>The more updates in the batch, the better, as the goal is < 1 <code>fsync()</code> for each update.</p>
<p><strong>Batching writes together</strong><br />
There are a couple of strategies in use for minimising the amount of <code>fsync()</code>s:</p>
<ol style="margin-bottom:10px;">
<li>Keep a buffer of fixed byte-size in memory, and issue <code>fsync()</code>s whenever it becomes full. Worst-case data loss is the number of updates that fit inside the buffer. This is a strategy used by multiple DBMSs. </li>
<li>Issue an <code>fsync()</code> periodically, every x milliseconds. Worst-case data loss is the amount of data written under a period of x milliseconds. For example, <a href="http://redis.io/topics/persistence">Redis</a> can be set to <code>fsync()</code> once every second. </li>
</ol>
<p>NoSQL stores are quick to sacrifice persistence for performance, for example <a href="http://stackoverflow.com/questions/3736533/why-doesnt-mongodb-use-fsync">MongoDB</a> doesn't <code>fsync()</code> at all (!) None of these strategies is fully durable, and thus they don't fulfil the requirement given in the exam. On the other hand, <code>fsync()</code>'ing each write doesn't scale and is painfully slow.</p>
<p><br/></p>
<h2>A scalable and fully durable solution</h2>
<p>The strategy I came up with considers batching updates together based on <em>concurrent access</em>. This way, writing to disk scales well with concurrent clients, and the worst-case data loss is zero -- adhering to the constraints given.</p>
<p>I realise this by implementing a <a href="http://en.wikipedia.org/wiki/Producer-consumer_problem">consumer/producer pattern</a>, by letting each client -- the producers -- add their update to a queue. A dedicated thread (the consumer), responsible for writing to persistent file, empties the queue and writes the updates to disk, issuing one <code>fsync()</code> for the batch. The result is that while the writes and disk synchronisation is in progress, <em>other clients will fill up the queue</em>. Batch-wise writing is thus implemented, with more writes combined as concurrent clients increase. The size of the buffer is bounded by the number of writing jobs added during the time to write the previous batch, plus the time to issue the <code>fsync()</code>. </p>
<p>Recall that each operation as a whole needs to be synchronous. However, by adding an update to the queue and letting a separate thread write, the function does not wait for the write to finish. I solved this problem by packaging each update into a struct, attaching a reference to a boolean which is set to true by the writing thread:</p>
<p><script src="https://gist.github.com/915526.js?file=persistence.cc"></script></p>
<p>The code for the writing thread:</p>
<p><script src="https://gist.github.com/915534.js?file=persistence.cc"></script></p>
<p>With a conditional variable, each client then waits for a <code>notify_all()</code> issued by the writing thread, checks the boolean, and then returns:</p>
<p><script src="https://gist.github.com/915530.js?file=persistence.cc"></script></p>
<p><br/></p>
<h2>Results</h2>
<p>I benchmarked my solution against the provided code for the SIGMOD contest. It performs around 4 million random inserts, reads and deletes for 8 byte keys and 8 byte values. Below is a plot of the performance of the index as a whole, with- and without durability. The y-axis shows throughput in requests/second, while the x-axis shows number of threads. Bear in mind that the throughput depends on the sizes of keys and values.</p>
<p><img style="background: none; padding: none;" src="http://static.eferm.com/wp-content/uploads/2011/04/concurrency.png" alt="concurrency plot" title="concurrency plot" width="600" height="244" class="alignnone size-full wp-image-2581" /></p>
<p>The same data, plotted on a logarithmic y-scale:</p>
<p><img style="background: none; padding: none;" src="http://static.eferm.com/wp-content/uploads/2011/04/concurrency-log.png" alt="concurrency-log plot" title="concurrency-log plot" width="600" height="243" class="alignnone size-full wp-image-2582" /></p>
<p>As a reference point, without the consumer/producer pattern the performance remained constant at ~300&nbsp;requests/second -- the same as for 1 thread. Instead, for highly concurrent access the durable B tree outperforms the in-memory only B tree. While this is likely due to reduced contention around nodes in the B tree, a stable state around 12000 requests/second seems to be reached which probably reflects the number of writes possible during an <code>fsync()</code>. The next step for optimising the solution would be to figure out ways to speed up writes in a single thread.</p>
<p><br/></p>
<h2>Conclusion</h2>
<p>I built an in-memory database using Kyoto Cabinet for the data structure, and a hand-rolled persistence layer. A simple file organisation is provided for SSD disks, and I optimise persistent disk writes by batching together concurrent writes, thus scaling well with concurrent access. But most importantly, I learned a lot regarding databases and programming techniques!</p>
<p><br/><br />
EDIT: Discussion on Hacker News: <a href="http://news.ycombinator.com/item?id=2440406">http://news.ycombinator.com/item?id=2440406</a><br />
EDIT 2: I'm pleased to know that this assignment received a distinction. </p>
 <p><a href="http://eferm.com/?flattrss_redirect&amp;id=2516&amp;md5=384385d5ad71f174de040ce72471b712" title="Flattr" target="_blank"><img src="http://eferm.com/wp-content/plugins/flattr/img/flattr-badge-large.png" alt="flattr this!"/></a></p>
<img src="http://feeds.feedburner.com/~r/eferm/~4/0G9O3URqKIc" height="1" width="1"/>]]></content:encoded>
<wfw:commentRss>http://eferm.com/implementing-durability-for-in-memory-databases-on-ssds/feed/</wfw:commentRss>
<slash:comments>0</slash:comments>
<atom:link rel="payment" href="https://flattr.com/submit/auto?user_id=Emore&amp;popout=1&amp;url=http%3A%2F%2Feferm.com%2Fimplementing-durability-for-in-memory-databases-on-ssds%2F&amp;language=en_GB&amp;category=text&amp;title=Implementing+durability+for+in-memory+databases%2C+on+SSDs&amp;description=As+the+examination+for+a+recently+completed+course+in+Database+Systems+Implementation%2C+students+had+to+implement+a+durable%2C+high-throughput%2C+in-memory+key%2Fvalue+database+for+strings%2C+coincidently+the+same+problem+as+this...&amp;tags=linkedin%2Cblog" type="text/html" />
<feedburner:origLink>http://eferm.com/implementing-durability-for-in-memory-databases-on-ssds/</feedburner:origLink></item>
<item>
<title>Dumb pipes — a (young) developer’s perspective</title>
<link>http://feedproxy.google.com/~r/eferm/~3/LTCcbwDAS08/</link>
<comments>http://eferm.com/dumb-pipes-a-young-developers-perspective/#comments</comments>
<pubDate>Thu, 21 Oct 2010 14:31:32 +0000</pubDate>
<dc:creator>Emanuel</dc:creator>
<category>
<![CDATA[Uncategorized]]>
</category>
<guid isPermaLink="false">http://eferm.com/?p=2395</guid>
<description>
<![CDATA[The openness of the internet is no doubt the major contributing factor for its success. Through the simplicity of dumb pipes, and with timeless principles such as the end-to-end principle and network neutrality, the internet grew and became what it is today: relevant for everyone, everywhere. At least this is my opinion. Others, mainly telecom [...]]]>
</description>
<content:encoded><![CDATA[
<p>The openness of the internet is no doubt the major contributing factor for its success. Through the simplicity of <a href="http://en.wikipedia.org/wiki/Dumb_pipe">dumb pipes</a>, and with timeless principles such as the <a href="http://en.wikipedia.org/wiki/End-to-end_principle">end-to-end principle</a> and <a href="http://en.wikipedia.org/wiki/Network_neutrality">network neutrality</a>, the internet grew and became what it is today: relevant for everyone, everywhere.</p>
<p>At least this is my opinion. Others, mainly telecom companies, argue that a more regulated and &#8220;smart&#8221; internet is necessary for its future expansion. Through &#8220;smart pipes&#8221; a better internet can be constructed, where the idea seems to be that intelligent wiring allows for more features and markets to be exploited, mainly by telecoms. </p>
<p>However, I believe there are important issues to be raised, and for relevant arguments to be heard, in favor for the pipes to remain dumb. </p>
<p>First and foremost, the fact that no smart pipes were needed to build today&#8217;s state-of-the-art should be a strong argument. On the other hand, the established internet is different today than it was during its build-up phase. The question is, then, whether the future expansion demands <a href="http://en.wikipedia.org/wiki/Deep_packet_inspection">deep-packet inspection</a> and other such technologies?</p>
<p>This question can only be answered with business cases detailing important end-user problems that would be solved, should these technologies be implemented. Examples where smart pipes are falsely claimed to solve problems are spam protection (already solved for end-customers; I haven&#8217;t seen spam in 5+ years), location (already solved by GPS, or location-databases such as Skyhook) or streaming (my YouTube works fine, thank you very much). </p>
<p>Regarding tiered pricing, service-specific connectivity or any such regulatory &#8220;innovation&#8221;, I&#8217;m afraid they will produce <a href="http://en.wikipedia.org/wiki/Walled_garden_(technology)">walled gardens</a> that are, at best, only slightly cheaper for the end-customer &#8212; a discount not worth paying for. </p>
<p>In the future, three scenarios may emerge. Hopefully smart pipes will be used for solving some problem in need of real innovation. Alternatively, they will simply produce software-services by companies whose core-business is not software &#8212; harmless but useless. Finally, and most dangerously, they might contribute to a radical change in end-user connectivity. This scenario is frightening because there is nothing I, as a developer, dislike more than walled gardens.</p>
<p>The vast majority of the creative and profitable internet is the product of young programmers&#8217; ideas, and therefore the industry&#8217;s continued growth is largely dependent on these people remaining innovative. Dumb openness is the way to go &#8212; because creativity does not thrive within walls.</p>
 <p><a href="http://eferm.com/?flattrss_redirect&amp;id=2395&amp;md5=0a60ec1ba0d6c2b5bd6875db05ad5d72" title="Flattr" target="_blank"><img src="http://eferm.com/wp-content/plugins/flattr/img/flattr-badge-large.png" alt="flattr this!"/></a></p>
<img src="http://feeds.feedburner.com/~r/eferm/~4/LTCcbwDAS08" height="1" width="1"/>]]></content:encoded>
<wfw:commentRss>http://eferm.com/dumb-pipes-a-young-developers-perspective/feed/</wfw:commentRss>
<slash:comments>5</slash:comments>
<atom:link rel="payment" href="https://flattr.com/submit/auto?user_id=Emore&amp;popout=1&amp;url=http%3A%2F%2Feferm.com%2Fdumb-pipes-a-young-developers-perspective%2F&amp;language=en_GB&amp;category=text&amp;title=Dumb+pipes+%E2%80%94+a+%28young%29+developer%26%238217%3Bs+perspective&amp;description=The+openness+of+the+internet+is+no+doubt+the+major+contributing+factor+for+its+success.+Through+the+simplicity+of+dumb+pipes%2C+and+with+timeless+principles+such+as+the+end-to-end+principle...&amp;tags=blog" type="text/html" />
<feedburner:origLink>http://eferm.com/dumb-pipes-a-young-developers-perspective/</feedburner:origLink></item>
<item>
<title>That big magical goal</title>
<link>http://feedproxy.google.com/~r/eferm/~3/tSNY4pX1N-I/</link>
<comments>http://eferm.com/that-big-magical-goal/#comments</comments>
<pubDate>Tue, 14 Sep 2010 01:37:13 +0000</pubDate>
<dc:creator>Emanuel</dc:creator>
<category>
<![CDATA[Uncategorized]]>
</category>
<guid isPermaLink="false">http://eferm.com/?p=2383</guid>
<description>
<![CDATA[There is a fine concept called “fuck you money” (which proves that it comes from America) that essentially means a large enough amount of money that allows you to completely decide over your lifestyle. It&#8217;s enough so that you can say “fuck you” to anyone, anywhere, because you don&#8217;t need to reassure your wealth. While [...]]]>
</description>
<content:encoded><![CDATA[
<p>There is a fine concept called “fuck you money” (which proves that it comes from America) that essentially means a large enough amount of money that allows you to completely decide over your lifestyle. It&#8217;s enough so that you can say “fuck you” to anyone, anywhere, because you don&#8217;t need to reassure your wealth.</p>
<p>While those amounts of money are rather hard to make, there is a much finer achievement of equal or greater significance that anyone can reach. Call it the “fuck you goal”, or simply that big magical goal.</p>
<p>You know it by certain characteristics, and it&#8217;s completely personal; only you yourself know what it might be. When you reach it, though, it will define you from there on &#8212; because you will have proven something to yourself that no one can take away from you. You will then, only then, realize your full capacity and further potential.</p>
<p>Reaching that goal is, however, really really hard. Today&#8217;s modern society, especially in Sweden with all its social safety nets, has almost completely removed the requirement for <em>effort</em> needed to accomplish anything. There are government agencies who find jobs for you. You apply to university through at most three clicks &#8212; no letters of recommendation needed, no interviews or even a letter of your own. And you most definitely don&#8217;t need to kill your own food each day.</p>
<p>While fantastic from a welfare state point of view, the complete removal of effort from everyday life and the simplicity of survival leaves many people disillusioned and merely half alive. When you don&#8217;t have to do much but still have everything you need to go about your day, you never really get to know your human capacity.</p>
<p>Philippe Petit, known as the “<a href="http://en.wikipedia.org/wiki/Philippe_Petit">Man on Wire</a>”, says it best:</p>
<blockquote><p>“I am not advocating to live in danger, but at the same time, to force birds to carry a leash is to kill the idea of what a bird is.”</p></blockquote>
<p>That big magical goal is exactly what will remind you of what you <em>can</em> do. It will create an inner calm, with an added assurance that whatever you do thereafter doesn&#8217;t really matter. Not so much because your achievement can&#8217;t be topped (on the contrary &#8212; it will enable you to go much farther), but because you already hit the upper plateau. Once you&#8217;ve reached it, it takes huge effort to fall back down again. And once you&#8217;re up there, you no longer have the stress of proving yourself to anybody, because you did something of real meaning &#8212; to yourself. That&#8217;s worth more than fuck you money.</p>
 <p><a href="http://eferm.com/?flattrss_redirect&amp;id=2383&amp;md5=608c94c557690428c896f14c2be363d0" title="Flattr" target="_blank"><img src="http://eferm.com/wp-content/plugins/flattr/img/flattr-badge-large.png" alt="flattr this!"/></a></p>
<img src="http://feeds.feedburner.com/~r/eferm/~4/tSNY4pX1N-I" height="1" width="1"/>]]></content:encoded>
<wfw:commentRss>http://eferm.com/that-big-magical-goal/feed/</wfw:commentRss>
<slash:comments>2</slash:comments>
<atom:link rel="payment" href="https://flattr.com/submit/auto?user_id=Emore&amp;popout=1&amp;url=http%3A%2F%2Feferm.com%2Fthat-big-magical-goal%2F&amp;language=en_GB&amp;category=text&amp;title=That+big+magical+goal&amp;description=There+is+a+fine+concept+called+%E2%80%9Cfuck+you+money%E2%80%9D+%28which+proves+that+it+comes+from+America%29+that+essentially+means+a+large+enough+amount+of+money+that+allows+you+to+completely...&amp;tags=blog" type="text/html" />
<feedburner:origLink>http://eferm.com/that-big-magical-goal/</feedburner:origLink></item>
<item>
<title>White space time management</title>
<link>http://feedproxy.google.com/~r/eferm/~3/X5N9bQP_NA0/</link>
<comments>http://eferm.com/white-space-time-management/#comments</comments>
<pubDate>Sun, 31 Jan 2010 16:51:17 +0000</pubDate>
<dc:creator>Emanuel</dc:creator>
<category>
<![CDATA[Uncategorized]]>
</category>
<guid isPermaLink="false">http://eferm.com/?p=1993</guid>
<description>
<![CDATA[The naïve approach to graphic design is usually to include as much information as possible on a given canvas. White space is nothing more than unused space, ready to be filled with more graphics and copy. However, this is obviously a bad strategy since all it does is confuse the viewer and obscure the message. [...]]]>
</description>
<content:encoded><![CDATA[
<p>The naïve approach to graphic design is usually to include as much information as possible on a given canvas. White space is nothing more than unused space, ready to be filled with more graphics and copy. However, this is obviously a bad strategy since all it does is confuse the viewer and obscure the message. This is why Google became king of search, and why Apple keeps being awesome.</p>
<p>Similarly, the naïve time management strategy is to fit in as much work and meetings in a schedule as possible. The person with the busiest calendar is clearly very good at managing his time and responsibilities. Or is he? </p>
<p>There is a difference between busy following a schedule, and busy solving a problem. This is essentially the same as a design busy presenting information and a design conveying a message. </p>
<p>Strangely, the concept of simplicity is never the natural state but always the result of carefully considered choices. Just as a design process should be about removing clutter until the bare essence is left, time management should be about removing appointments. Not adding new ones where they fit. </p>
<p>White space is a powerful element, necessary for creating dynamics between the essential components of a design. In the same way, free time is essential for effective time management. Leisure time works in your favor when alternated with creative problem solving. Mindlessly adding more work is thus nothing else than adding clutter. </p>
<p>True productiveness is just as much a product of free time, as of hard work. This is why the most creative people always tend to have hobbies, read books, write and travel &#8212; while the mediocre majority complain about being busy. </p>
 <p><a href="http://eferm.com/?flattrss_redirect&amp;id=1993&amp;md5=c2b666849f78139277851bfb19331c54" title="Flattr" target="_blank"><img src="http://eferm.com/wp-content/plugins/flattr/img/flattr-badge-large.png" alt="flattr this!"/></a></p>
<img src="http://feeds.feedburner.com/~r/eferm/~4/X5N9bQP_NA0" height="1" width="1"/>]]></content:encoded>
<wfw:commentRss>http://eferm.com/white-space-time-management/feed/</wfw:commentRss>
<slash:comments>3</slash:comments>
<atom:link rel="payment" href="https://flattr.com/submit/auto?user_id=Emore&amp;popout=1&amp;url=http%3A%2F%2Feferm.com%2Fwhite-space-time-management%2F&amp;language=en_GB&amp;category=text&amp;title=White+space+time+management&amp;description=The+na%C3%AFve+approach+to+graphic+design+is+usually+to+include+as+much+information+as+possible+on+a+given+canvas.+White+space+is+nothing+more+than+unused+space%2C+ready+to+be...&amp;tags=blog" type="text/html" />
<feedburner:origLink>http://eferm.com/white-space-time-management/</feedburner:origLink></item>
<item>
<title>The Game 3.0</title>
<link>http://feedproxy.google.com/~r/eferm/~3/zEAvZOyF-ys/</link>
<comments>http://eferm.com/the-game-3-0/#comments</comments>
<pubDate>Mon, 30 Nov 2009 15:52:59 +0000</pubDate>
<dc:creator>Emanuel</dc:creator>
<category>
<![CDATA[Uncategorized]]>
</category>
<guid isPermaLink="false">http://eferm.com/?p=1874</guid>
<description>
<![CDATA[Yet another game is now complete and over. I&#8217;d have to say this was one of the better ones we&#8217;ve made! It&#8217;s always very difficult to judge the quality and difficulty of tasks beforehand, but based on player response the conclusion is that it was successful. I thought I&#8217;d share some of the problems that [...]]]>
</description>
<content:encoded><![CDATA[
<p>Yet another game is now complete and over. I&#8217;d have to say this was one of the better ones we&#8217;ve made! It&#8217;s always very difficult to judge the quality and difficulty of tasks beforehand, but based on player response the conclusion is that it was successful. I thought I&#8217;d share some of the problems that were in the game. For more about the game me and <a href="http://futu-risk.com/">Fredrik</a> make, see <a href="http://eferm.com/the-game">The Game</a> and <a href="http://eferm.com/the-game-2-0">The Game 2.0</a>.<br />
<span style="float:right; text-align:right;"><span id="more-1874"></span></span></p>
<p><strong>Stage 1, problem 7 &#8212; <a href="http://static.eferm.com/wp-content/uploads/2010/08/say.mp3">The Kidnapping</a><br />
</strong></p>
<p>That&#8217;s all the clues you got. Notice that the title is actually a link to an .mp3 file. Three weird voices reciting numbers can be heard on it. However, this was a red herring; the real clue was in the album art image in the meta data of the file.</p>
<p><object height="81" width="100%"><param name="movie" value="http://player.soundcloud.com/player.swf?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F15005078"></param><param name="allowscriptaccess" value="always"></param> <embed allowscriptaccess="always" height="81" src="http://player.soundcloud.com/player.swf?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F15005078" type="application/x-shockwave-flash" width="100%"></embed></object><br />
<em>The sound file linked to in the title.</em></p>
<p><script src="http://gist.github.com/566024.js" type="text/javascript"></script><br />
<em>People on Macs can run this command in the Terminal to produce the voices recorded in the .mp3 file.</em></p>
<p><img alt="album-art" src="http://static.eferm.com/wp-content/uploads/2010/09/artwork.png" title="Album art" class="aligncenter" width="266" height="200" /><br />
<span class="aligncenter" style="text-align: center;"><em>The album art of the .mp3 file linked to.</em></span></p>
<p>The picture was taken near the south entrance of the venue. If a contestant searched the place displayed in the photo, he or she found a note underneath the counter with a phone number written on it. When calling the number, morse code for the word &#8220;<a href="http://sound.dhgame.eu">sound.dhgame.eu</a>&#8221; could be decoded. On this webpage yet another sound file could be found.</p>
<p><object height="81" width="100%"><param name="movie" value="http://player.soundcloud.com/player.swf?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F15005077"></param><param name="allowscriptaccess" value="always"></param> <embed allowscriptaccess="always" height="81" src="http://player.soundcloud.com/player.swf?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F15005077" type="application/x-shockwave-flash" width="100%"></embed></object><br />
<em>The last message from the kidnapped girl. The voice is actually me, somewhat edited :)</em></p>
<p>On the second .mp3 file a scared female voice was whispering landmarks she saw while being dragged away by unknown people. Her ill fate ended by what she described as &#8220;&#8230;roses &#8230;five stones&#8230;&#8221; This was a description of a fountain outside of the venue. If the players identified it, ran over there and searched the place, they found a note with the password to the next level.</p>
<p><strong>Stage 2, problem 4 &#8212; Exploded Encryption</strong></p>
<blockquote><p><span style="font-size: 11px;">571225901882778912997980282894594610215406780706531189904448031184<br />08314869947760328114059751957706621912851280200677377971352196615</span></p></blockquote>
<p>For this task I implemented an encryption that I came up with, which I call Exploded Encryption. Basically, it converts a string to ASCII codes and hides these sparsely (&#8220;exploded&#8221;) within the decimals of pi. This was, however, too hard for the players to backtrack without clues. But the intended way of solving it was to first realize that the numbers seem completely random. But since there still must be <em>something</em> hidden in there an external key should be able to provide it. This key I used was <strong>pi</strong>. </p>
<p><script src="http://gist.github.com/566019.js" type="text/javascript"></script><br />
<em>The explode() function I wrote to generate the string. It&#8217;s on Github <a href="http://github.com/Emore/Exploded-Encryption">here</a>.</em></p>
<p>The interesting information within the string can be found by overlaying it with pi, and extracting the matching digits at the same indices.</p>
<p><span style="font-size:11px;"><br />
3141592653589<span style="font-weight:bold;color:red;">7</span>93238462643383279502884<span style="font-weight:bold;color:red;">1</span>971<span style="font-weight:bold;color:red;">6</span>939937<span style="font-weight:bold;color:red;">5</span>105<span style="font-weight:bold;color:red;">8</span>2097<span style="font-weight:bold;color:red;">4</span>94459230<br />7<span style="font-weight:bold;color:red;">8</span>16<span style="font-weight:bold;color:red;">4</span>0<span style="font-weight:bold;color:red;">6</span>28620899862803482<span style="font-weight:bold;color:red;">5</span>3421170<span style="font-weight:bold;color:red;">6</span>798214808651328230664<span style="font-weight:bold;color:red;">7</span>093844<span style="font-weight:bold;color:red;">6</span>09<span style="font-weight:bold;color:red;">5</span></span><br />
<span style="font-size: 11px;"><br />
5712259018827<span style="font-weight:bold;color:red;">7</span>89129979802828945946102<span style="font-weight:bold;color:red;">1</span>540<span style="font-weight:bold;color:red;">6</span>780706<span style="font-weight:bold;color:red;">5</span>311<span style="font-weight:bold;color:red;">8</span>9904<span style="font-weight:bold;color:red;">4</span>48031184<br />0<span style="font-weight:bold;color:red;">8</span>31<span style="font-weight:bold;color:red;">4</span>8<span style="font-weight:bold;color:red;">6</span>99477603281140597<span style="font-weight:bold;color:red;">5</span>1957706<span style="font-weight:bold;color:red;">6</span>219128512802006773779<span style="font-weight:bold;color:red;">7</span>135219<span style="font-weight:bold;color:red;">6</span>61<span style="font-weight:bold;color:red;">5</span></span><br />
<em>Pi, and the given string, with the corresponding digits highlighted.</em></p>
<p>The resulting string is 71658484656765. The skilled eye notices that each pair of numbers are between 65 and under 100, something that should make the hunter think of ASCII codes. Indeed, converted to text the string yields &#8220;GATTACA.&#8221;</p>
<p>If every random sequence of numbers is overlaid with pi from now on, I am probably to blame. :)</p>
 <p><a href="http://eferm.com/?flattrss_redirect&amp;id=1874&amp;md5=ca568cfc077b3afb8e828a0e1bd074a1" title="Flattr" target="_blank"><img src="http://eferm.com/wp-content/plugins/flattr/img/flattr-badge-large.png" alt="flattr this!"/></a></p>
<img src="http://feeds.feedburner.com/~r/eferm/~4/zEAvZOyF-ys" height="1" width="1"/>]]></content:encoded>
<wfw:commentRss>http://eferm.com/the-game-3-0/feed/</wfw:commentRss>
<slash:comments>4</slash:comments>
<enclosure url="http://static.eferm.com/wp-content/uploads/2010/08/say.mp3" length="273621" type="audio/mpeg" />
<enclosure url="http://static.eferm.com/wp-content/uploads/2010/08/say.mp3" length="273621" type="audio/mpeg" />
<atom:link rel="payment" href="https://flattr.com/submit/auto?user_id=Emore&amp;popout=1&amp;url=http%3A%2F%2Feferm.com%2Fthe-game-3-0%2F&amp;language=en_GB&amp;category=text&amp;title=The+Game+3.0&amp;description=Yet+another+game+is+now+complete+and+over.+I%26%238217%3Bd+have+to+say+this+was+one+of+the+better+ones+we%26%238217%3Bve+made%21+It%26%238217%3Bs+always+very+difficult+to+judge+the+quality+and...&amp;tags=blog" type="text/html" />
<feedburner:origLink>http://eferm.com/the-game-3-0/</feedburner:origLink></item>
<item>
<title>The Game 2.0</title>
<link>http://feedproxy.google.com/~r/eferm/~3/L22zTw83Tq0/</link>
<comments>http://eferm.com/the-game-2-0/#comments</comments>
<pubDate>Thu, 19 Nov 2009 10:11:00 +0000</pubDate>
<dc:creator>Emanuel</dc:creator>
<category>
<![CDATA[Uncategorized]]>
</category>
<guid isPermaLink="false">http://eferm.com/?p=1849</guid>
<description>
<![CDATA[Last week I went back to the UK with Fredrik, for another game. There turned out to be less participants, but we did our best to create some interesting problems anyway. Here are two of them. Stage 1, problem 6 &#8212; Retrieve me treasure&#8230; &#8230;or reel th&#8217; plank! Clues: Yer booty is t&#8217; be found [...]]]>
</description>
<content:encoded><![CDATA[
<p>Last week I went back to the UK with <a href="http://futu-risk.com">Fredrik</a>, for another <a href="http://eferm.com/the-game">game</a>. There turned out to be less participants, but we did our best to create some interesting problems anyway. Here are two of them.</p>
<p><strong>Stage 1, problem 6 &#8212; Retrieve me treasure&#8230;</strong></p>
<blockquote><p>
&#8230;or reel th&#8217; plank!<br />
<a href="http://static.eferm.com/wp-content/uploads/2010/09/map.png"><img src="http://static.eferm.com/wp-content/uploads/2010/09/map.png" alt="map" class="alignnone" width="420" /></a><br />
<strong>Clues:</strong><br />
Yer booty is t&#8217; be found on &#8220;Indoor Island&#8221;.<br />
Me treasure is me weap&#8217;n!
</p></blockquote>
<p>The key to solve this task is to realize that the map does not depict some South Sea island, but instead a location at the venue where the game took place. To help finding the right place, some clues were given: the compass shows the relative orientation and &#8220;Indoor Island&#8221; indicates that it&#8217;s not outside. Through the event&#8217;s website, this seating map for one particular floor can be found:</p>
<p><a href="http://static.eferm.com/wp-content/uploads/2010/09/floorplan.png"><img alt="" src="http://static.eferm.com/wp-content/uploads/2010/09/floorplan.png" title="Floorplan" class="aligncenter" height="300" /></a></p>
<p>The basic characteristics are similar, and the actual interior design corresponds to objects in the map. For example, &#8220;West Port&#8221; and &#8220;East Pier&#8221; indicate entrances, and &#8220;Underground Gorge&#8221; is the escalator from the floor below. If the participants went to the &#8220;<strong><span style="color:red;">X</span></strong>&#8220;, they found a poem containing a reference to the password: the poor pirate&#8217;s <strong>gold plated dagger</strong>.</p>
<p><strong>Stage 2, problem 5 &#8212; Jazz</strong></p>
<blockquote><p>
U+100AA</p>
<p>o hrhi ng odvm xutz grfwor nideiyy jiuz’x oxrivhi vm colxh phse, rzh cxxhrbw ux to cartxx crdiq. bl tymx jty tigi ux sujf lnok fvxx gagt yq lnw rojf xux ulu ieef coixh, ctod r tmta vrzoi shx lzhmaz zof xsaz cikt e fbtgcq hexgm. yq qhlz hrhi yhukvp yc tz ae grstsicuee lqy ktvbnmh wdmtazeeurt ekamqw ngj syuzrkkd re lr yuuep autz a xdsgxyqlq xubtg r dsfx os rzh uhc rri xux yuexmtaz wre ycht tyq wptxcvxc pkkakqh tkgsj. m rrp coixh, ztzeiuey potyayg ukies vrtr, wyqvr iuoi slblzs, sdinmnies hexgmj xmxx gii, pvvyzeu rsemaikayfee asayg&#8230;eokv flnm gsyqr, stttrexvv lixgvr zriuurt muwrdh ubs tydshzn tyq ezhxpyayf mxeve.
</p></blockquote>
<p>The easiest method to solve this task was to identify the string &#8220;U+100AA&#8221; as a Unicode code for the Linear B ideogram &#8220;Garment.&#8221; By intuition or by evaluating alternative encryption methods, the larger paragraph of text was found to be a Vigénere cipher. &#8220;Garment&#8221; was thus the keyword. If deciphered, and Googled for, the text could be identified as an excerpt from Fitzgerald&#8217;s <strong>The Great Gatsby</strong>. The really literate participants would, of course, recognize the text immediately and also that &#8220;Jazz&#8221; refers to The Jazz Age &#8212; the period in American history in which The Great Gatsby took place.</p>
<p>All in all it was lots of fun creating and interacting with the competing teams! Next game is in only 2 weeks, something I&#8217;m greatly looking forward to.</p>
 <p><a href="http://eferm.com/?flattrss_redirect&amp;id=1849&amp;md5=4cf545dcd0e6eba1f415402f45310cc7" title="Flattr" target="_blank"><img src="http://eferm.com/wp-content/plugins/flattr/img/flattr-badge-large.png" alt="flattr this!"/></a></p>
<img src="http://feeds.feedburner.com/~r/eferm/~4/L22zTw83Tq0" height="1" width="1"/>]]></content:encoded>
<wfw:commentRss>http://eferm.com/the-game-2-0/feed/</wfw:commentRss>
<slash:comments>3</slash:comments>
<atom:link rel="payment" href="https://flattr.com/submit/auto?user_id=Emore&amp;popout=1&amp;url=http%3A%2F%2Feferm.com%2Fthe-game-2-0%2F&amp;language=en_GB&amp;category=text&amp;title=The+Game+2.0&amp;description=Last+week+I+went+back+to+the+UK+with+Fredrik%2C+for+another+game.+There+turned+out+to+be+less+participants%2C+but+we+did+our+best+to+create+some+interesting+problems...&amp;tags=blog" type="text/html" />
<feedburner:origLink>http://eferm.com/the-game-2-0/</feedburner:origLink></item>
<item>
<title>An idea denied</title>
<link>http://feedproxy.google.com/~r/eferm/~3/2UySYIKA2RA/</link>
<comments>http://eferm.com/an-idea-denied/#comments</comments>
<pubDate>Thu, 08 Oct 2009 17:56:51 +0000</pubDate>
<dc:creator>Emanuel</dc:creator>
<category>
<![CDATA[Uncategorized]]>
</category>
<guid isPermaLink="false">http://eferm.com/?p=1810</guid>
<description>
<![CDATA[Why are successful people turned into instruments of unique opportunity? Why are their stories told as circumstances of chance? There are many examples of this kind of mystification. Isaac Newton suddenly discovered gravity when he observed an apple fall from a tree. A composer found inspiration and then wrote an amazing symphony. An entrepreneur had [...]]]>
</description>
<content:encoded><![CDATA[
<p>Why are successful people turned into instruments of unique opportunity? Why are their stories told as circumstances of chance?</p>
<p>There are many examples of this kind of mystification. Isaac Newton suddenly discovered gravity when he observed an apple fall from a tree. A composer found inspiration and then wrote an amazing symphony. An entrepreneur had a brilliant idea and made a fortune through his business.</p>
<p>In countless of stories such as these, the triggering factor to a significant event is always made up to be <em>external</em>, and thus out of reach for the person&#8217;s influence. </p>
<p>But this is not reality: authors write nine to five regardless of any &#8220;inspiration&#8221;, they rewrite the same passage countless of times and they most often produce their best work when they are old. Newton didn&#8217;t discover gravity due to some falling apple; he had studied physics, mathematics and astronomy more than most people ever did.</p>
<p>I dare say that the real factor to success is always, and has always been, <em>damn hard work</em> and <em>many failed attempts</em>.</p>
<p>This is, however, a dangerous idea. People go to extreme lengths to deny it. For example the woman playing the lottery, believing it is the only way for her to become rich. Or the failed author blaming his empty pages on not having found the right inspiration &#8212; &#8220;yet.&#8221;</p>
<p>There is really no excuse for not having the capacity to achieve one&#8217;s own success. No talent? Talent is not nearly as important as the time you spend with a task, as described by Malcolm Gladwell in his <em>Outliers</em> (in which he also describes the importance of opportunity). No money? University is free, study loans are accessible for anyone and it doesn&#8217;t cost anything to start a company. Not &#8220;ready&#8221;? <a href="http://eferm.com/the-illusion-of-mental-prerequisites">Get over it</a>.</p>
<p>Why is the idea that anyone can become anything through one&#8217;s own ambition not embraced, but instead denied? Because it means complete responsibility for oneself, and the unlimited possibilities are easier shrugged off than capitalized. By some reason people are much more likely to confine themselves to a state where they believe they can&#8217;t affect their own destiny, rather than to embrace freedom. But still everyone knows that they <em>could</em> do anything, if they only tried. The guilt reminds them of this.</p>
<p>This is why successful people are mystified. By admitting that the causes for success were indeed equally accessible for everybody, the guilt is unbearable. So the choice is simple &#8212; deny the possibilities, stop thinking about it and blame everything on lucky circumstances.</p>
 <p><a href="http://eferm.com/?flattrss_redirect&amp;id=1810&amp;md5=a7976827e55ccb2b49b9633a8acbca8f" title="Flattr" target="_blank"><img src="http://eferm.com/wp-content/plugins/flattr/img/flattr-badge-large.png" alt="flattr this!"/></a></p>
<img src="http://feeds.feedburner.com/~r/eferm/~4/2UySYIKA2RA" height="1" width="1"/>]]></content:encoded>
<wfw:commentRss>http://eferm.com/an-idea-denied/feed/</wfw:commentRss>
<slash:comments>8</slash:comments>
<atom:link rel="payment" href="https://flattr.com/submit/auto?user_id=Emore&amp;popout=1&amp;url=http%3A%2F%2Feferm.com%2Fan-idea-denied%2F&amp;language=en_GB&amp;category=text&amp;title=An+idea+denied&amp;description=Why+are+successful+people+turned+into+instruments+of+unique+opportunity%3F+Why+are+their+stories+told+as+circumstances+of+chance%3F+There+are+many+examples+of+this+kind+of+mystification.+Isaac+Newton...&amp;tags=blog" type="text/html" />
<feedburner:origLink>http://eferm.com/an-idea-denied/</feedburner:origLink></item>
<item>
<title>Do stuff</title>
<link>http://feedproxy.google.com/~r/eferm/~3/okebM3o1yi8/</link>
<comments>http://eferm.com/do-stuff/#comments</comments>
<pubDate>Wed, 30 Sep 2009 21:57:50 +0000</pubDate>
<dc:creator>Emanuel</dc:creator>
<category>
<![CDATA[Uncategorized]]>
</category>
<guid isPermaLink="false">http://eferm.com/?p=1791</guid>
<description>
<![CDATA[Here&#8217;s a graph illustrating a model of the relationship between doing a lot of stuff, and the amount of fun it results in: With stuff I refer to anything productive and rewarding, such as a course at school, a qualified job or a project of your own. And with fun I refer to the feeling [...]]]>
</description>
<content:encoded><![CDATA[
<p>Here&#8217;s a graph illustrating a model of the relationship between doing a lot of stuff, and the amount of fun it results in:</p>
<p><img alt="Productivity graph" src="http://static.eferm.com/wp-content/uploads/2010/09/467785111c.png" title="model" class="aligncenter" width="300" height="240" /></p>
<p>With <em>stuff</em> I refer to anything productive and rewarding, such as a course at school, a qualified job or a project of your own. And with <em>fun</em> I refer to the feeling of satisfaction and purpose that is the result of doing meaningful things.</p>
<p>So, what can this model tell us?</p>
<ul>
<li>Do more, create something, engage in productiveness and a great feeling will follow.
</li>
<li>Eliminate wasted time to give room for more personal projects, sports or arts.
</li>
<li>Being overworked removes all the joy from what you&#8217;re doing.
</li>
<li>Make sure to find out your personal &#8220;maximum workload constant&#8221;, to know the feeling of when there&#8217;s simply too many things going on. You&#8217;ll never want to end up getting burned again.
</li>
<li>Remember where your limit is, and carefully balance your workload to stay just below the threshold.
</li>
<li>The &#8220;maximum workload constant&#8221; is not constant: it can be extended to allow for an increased capacity.
</li>
<li>Having too little to do is far better than being overworked.
</li>
<li>When you have &#8220;almost too much to do&#8221; it&#8217;s really just the right amount of work!
</li>
<li>If you&#8217;re engaged in stuff you like, and you are filling your time with it, you&#8217;ll hopefully experience <a href="http://en.wikipedia.org/wiki/Flow_(psychology)">flow</a>.
</li>
</ul>
<p>Or as the Ruby hacker <a href="http://en.wikipedia.org/wiki/Why_the_lucky_stiff">_why</a> said:</p>
<blockquote><p>
when you don&#8217;t create things, you become defined by your tastes rather than ability. your tastes only narrow &#038; exclude people. so create.
</p></blockquote>
<p>After all, it&#8217;s a thousand times more interesting to talk to someone that fills his time with interesting work and projects of his own, rather than someone completely defined by his music taste or belief. Experiences come through interaction with the real world, and they don&#8217;t create themselves &#8212; they need to be obtained through hard work. And a few leaps of faith.</p>
 <p><a href="http://eferm.com/?flattrss_redirect&amp;id=1791&amp;md5=beff17f95704c0f418d8068de4df69b6" title="Flattr" target="_blank"><img src="http://eferm.com/wp-content/plugins/flattr/img/flattr-badge-large.png" alt="flattr this!"/></a></p>
<img src="http://feeds.feedburner.com/~r/eferm/~4/okebM3o1yi8" height="1" width="1"/>]]></content:encoded>
<wfw:commentRss>http://eferm.com/do-stuff/feed/</wfw:commentRss>
<slash:comments>4</slash:comments>
<atom:link rel="payment" href="https://flattr.com/submit/auto?user_id=Emore&amp;popout=1&amp;url=http%3A%2F%2Feferm.com%2Fdo-stuff%2F&amp;language=en_GB&amp;category=text&amp;title=Do+stuff&amp;description=Here%26%238217%3Bs+a+graph+illustrating+a+model+of+the+relationship+between+doing+a+lot+of+stuff%2C+and+the+amount+of+fun+it+results+in%3A+With+stuff+I+refer+to+anything+productive...&amp;tags=blog" type="text/html" />
<feedburner:origLink>http://eferm.com/do-stuff/</feedburner:origLink></item>
<item>
<title>The Game</title>
<link>http://feedproxy.google.com/~r/eferm/~3/J0_901w-Nw0/</link>
<comments>http://eferm.com/the-game/#comments</comments>
<pubDate>Thu, 13 Aug 2009 01:28:09 +0000</pubDate>
<dc:creator>Emanuel</dc:creator>
<category>
<![CDATA[Uncategorized]]>
</category>
<guid isPermaLink="false">http://eferm.com/blog/?p=277</guid>
<description>
<![CDATA[Twice a year me and Fredrik create The Dreamhack Game (DHG), at the Dreamhack computer festival. Earlier this summer we got an email from the Multiplay staff, who arrange UK&#8217;s largest LAN parties, inviting us over to create what is now the i-Hunt. Apparently they knew of what we do in Sweden, and liked it [...]]]>
</description>
<content:encoded><![CDATA[
<p>Twice a year me and <a href="http://futu-risk.com/">Fredrik</a> create The Dreamhack Game (DHG), at the Dreamhack computer festival. Earlier this summer we got an email from the Multiplay staff, who arrange UK&#8217;s largest LAN parties, inviting us over to create what is now the <a href="http://ihunt.multiplay.co.uk">i-Hunt</a>. Apparently they knew of what we do in Sweden, and liked it enough to fly us over to their own event. Quite cool indeed, and of course we made the most out of it. I&#8217;d say my first international &#8220;business&#8221; trip was a success. </p>
<p>The game is advertised as a &#8220;contest of intellect, lateral thinking and logical skill&#8221;. No special skills or knowledge are required, only the ability to figure out <em>what</em> to do and <em>how</em> to obtain necessary and relevant information. The tasks usually include elements of code breaking, <a href="http://en.wikipedia.org/wiki/Alternate_reality_game">alternate reality gaming</a>, <a href="http://en.wikipedia.org/wiki/Geocaching">geocaching</a>, deciphering, <a href="http://en.wikipedia.org/wiki/Treasure_hunt_(game)">treasure hunting</a> and various puzzles. Advanced Google skills are fundamental to solving the game, and so are endurance and thoroughness. Once a problem is solved, you move on to the next level. You usually compete in teams, but you can never know the current position of your competitors. This makes it a competition of intelligence cloaked in mystery and with a touch of psycological warfare. </p>
<p>There were a few tasks we made for this event that I&#8217;m a little extra proud of. I&#8217;ll explain them here, as they give a very good picture of what The Game is really about. But if the reader feels like giving the game &#8211; and these problems in particular &#8211; a try, then head over to the <a href="http://ihunt.multiplay.co.uk/">i-Hunt website</a> (which will be up until November 09), and register to play. An answer sheet is also available if you get stuck and just want to try the next problem.</p>
<p><strong>Stage 1, problem 5 &#8212; The Shameful One</strong></p>
<blockquote><p>
An outbound coast<br />
Surrounds or embraces? The city<br />
Hamilton&#8217;s (NZ) antipode.<br />
<a href="http://static.eferm.com/wp-content/uploads/2010/09/shame.png"><img alt="The Shameful One" src="http://static.eferm.com/wp-content/uploads/2010/09/shame.png" title="The Shameful One" class="alignnone" width="425" /></a>
</p></blockquote>
<p>There are two clues here &#8211; the haiku poem and the maze image. Both can give the answer on their own, but also in combination with another. The line that is traced through the maze when solved, can be identified by the hunter as the south coast of Spain, and part of Portugal. This is what is referred to in the poem as &#8220;An outbound coast&#8221;. Furthermore, the location of the square dot indicates &#8220;The city, Hamilton&#8217;s (NZ) antipode&#8221; &#8211; which is the city of Córdoba in Spain. An <a href="http://en.wikipedia.org/wiki/Antipodes">antipode</a> is a complete opposite geographical location, a rare property that Córdoba and Hamilton in New Zealand share. <strong><strong>Córdoba</strong></strong> is thus the password.</p>
<p><strong>Stage 2, problem 3 &#8212; Who&#8217;s coming to visit today?</strong></p>
<blockquote><p>
Have you been paying attention to the local tv-station?<br/>
</p></blockquote>
<p>In this problem, the contestants had to realize that &#8220;the local tv-station&#8221; referred to the event-specific daily Youtube broadcasts, mainly the Saturday one found <a href="http://www.youtube.com/watch?v=cJTnBxXRpFQ">here</a>. The careful watcher will notice the announcement of a new sponsor &#8211; Tentacle Technology. However, this company is not to be found on the internet, nor did they actually show up at the venue.  Instead, if the URL <a href="http://tentacletechnology.com">http://tentacletechnology.com</a> was thought of and followed, a really fancy website was found. The password &#8220;<strong>puppet</strong>&#8221; could be found if downloading the latest press release.</p>
<p>Smell something fishy here? That&#8217;s because me and Fredrik made it all up; in two hours we had set up the <a href="http://tentacletechnology.com">company website</a>, got fake sponsorship deals, marketed ourselves, staged a 31 year old corporate history, stolen product descriptions from IBM and even put together a catchy mission statement. This kind of ARG-inspired problem is one of my favourites.</p>
<p><strong>Stage 2, problem 5 &#8212; Think inside the box</strong></p>
<blockquote><p>
<a href="http://static.eferm.com/wp-content/uploads/2010/09/cards.png"><img alt="Cards" src="http://static.eferm.com/wp-content/uploads/2010/09/cards.png" title="Think inside the box" class="alignnone" width="425" /></a>
</p></blockquote>
<p>In this last problem, the first realization to make is that the cards are actually not part of some unspecified card game. Instead it&#8217;s a sudoku, and when solved the numbers revealed in order are 174143214192. For a hunter, this is immediately identified as the IP address 174.143.214.192, and one of the first things to do with an IP is to HTTP it. When done so, a simple website containing the following image was found:</p>
<p><a href="http://static.eferm.com/wp-content/uploads/2010/09/ihunt.jpg"><img alt="ihunt" src="http://static.eferm.com/wp-content/uploads/2010/09/ihunt.jpg" title="Think inside the box" class="alignnone" width="425" /></a></p>
<p>Again, the experienced contenstant would identified the erect and fallen cans as morse code. When deciphered, the final password was &#8220;<strong>wey</strong>&#8220;.</p>
<p>* * *</p>
<p>At Dreamhack we attract ~600 players, and at the i-Hunt we managed to get 200 registered players which should be considered good, since it was the first game in the UK for us. However, it looks like I&#8217;ll be going back there soon, as I&#8217;ll probably be arranging the i-Hunt three times a year in total &#8211; which feels great! I&#8217;m looking forward to get to know Britain more, as well as to work with creating and further evolving The Game.</p>
 <p><a href="http://eferm.com/?flattrss_redirect&amp;id=277&amp;md5=69f3752989a4188b52b7e169374ca82a" title="Flattr" target="_blank"><img src="http://eferm.com/wp-content/plugins/flattr/img/flattr-badge-large.png" alt="flattr this!"/></a></p>
<img src="http://feeds.feedburner.com/~r/eferm/~4/J0_901w-Nw0" height="1" width="1"/>]]></content:encoded>
<wfw:commentRss>http://eferm.com/the-game/feed/</wfw:commentRss>
<slash:comments>6</slash:comments>
<atom:link rel="payment" href="https://flattr.com/submit/auto?user_id=Emore&amp;popout=1&amp;url=http%3A%2F%2Feferm.com%2Fthe-game%2F&amp;language=en_GB&amp;category=text&amp;title=The+Game&amp;description=Twice+a+year+me+and+Fredrik+create+The+Dreamhack+Game+%28DHG%29%2C+at+the+Dreamhack+computer+festival.+Earlier+this+summer+we+got+an+email+from+the+Multiplay+staff%2C+who+arrange+UK%26%238217%3Bs...&amp;tags=blog" type="text/html" />
<feedburner:origLink>http://eferm.com/the-game/</feedburner:origLink></item>
</channel>
</rss><!-- Performance optimized by W3 Total Cache. Learn more: http://www.w3-edge.com/wordpress-plugins/

Minified using disk: basic
Page Caching using disk: enhanced
Database Caching 1/45 queries in 0.973 seconds using disk: basic
Object Caching 905/1005 objects using disk: basic
Content Delivery Network via Amazon Web Services: CloudFront: static.eferm.com

Served from: eferm.com @ 2012-05-18 12:45:11 -->

