Aquariumtap

Palantir's bet against artificial intelligence

aquariumtap — Fri, 13 Jan 2012 17:23:58 +0000

Palantir's analytic platform is based on a critique of artificial intelligence: no algorithm can replace the effectiveness of the human mind against a data set moved cunningly by other humans. The video I've posted begins with two subjects. The first is the failure of computers to become world champions of chess, a claim that began in the 1950's and publicly died out when IBM's Deep Blue lost to Kasparov in 1996. If the brute force of 200 million evaluations per second cannot establish a clear victory, there must be more to "cunning" than great foresight in positioning symbols against formalized knowledge and rules.

What is cunning? I won't discuss that here, but I can highly recommend a book by Hubert Dreyfus called What Computers Still Can't Do: A Critique of Artificial Reason.

The second subject of the introduction is PayPal's approach to fraud protection. Initially PayPal followed a standard strategy of evaluating all records against a criteria for suspicious activity, then flagged matches for manual review by a human. Certainly, they caught people and went after them, but each time they did, it would reveal part of their strategy. The entrepreneurial adversary -- in the case of PayPal, a network of Russian hackers -- performed a diagnostic upon each failure, deduced why they were caught, and made alterations accordingly. This will always be the case. Anytime one team beats out another, there's a shift in strategy. The question is, how do you keep up with an agile enemy?

PayPal knew that computers were not as good as human analysts in detecting and accounting for those shifts, but there was an organizational flaw in their anti-fraud team that made the analyst (the person making sense of data) too dependent on the computer scientist (the person querying the data). While other banking systems were using analysts to tell computer scientists how to improve their detection, PayPal set out to create tools to augment the analyst's reach into the data. The scientist was moved from high level querying to low to mid level pattern detection, then presented that partially crunched data to the analyst for independent review. These patterns were nodal connections between yellow flag transactors, rather than transactions that contained untrustworthy attributes.

A former CTO of Amazon tells an anecdote that comes to mind: a large telephone company performed a marketing campaign based on two approaches. The first approach used sophisticated segmentation models designed by doctorates in mathematics. People were treated as symbols with attributes, and that was the basis for their campaign's targeting rules. In the second approach, the company began with a pool of previously successful conversions and targeted their social network based on call activity. The frequency of incoming and outgoing calls between two people informed the strength of a nodal connection. The story goes that the second campaign, the one based on actions taken (phone calls forming a social graph), had a 5x adoption rate over the segmented models. It was also far less expensive to produce.

How to have comma-separated tags in Drupal 7

aquariumtap — Sat, 01 Oct 2011 19:27:24 +0000

If you'd like taxonomy tags separated by commas instead of the default series of divs output by default, you need to override the default theme function.

What is the theme() function? It's Drupal's way of turning raw data into rendered, themed output -- in most cases, HTML markup. All requests for rendered output are passed through theme(). A menu, a block, the title of a page: these UI components are stored in the database, passed through theme(), referenced against a registry, and finally routed to a function that can work and loop through the data to render markup. This happens with taxonomy tags, although in somewhat abstracted way.

In Drupal 7, taxonomy terms are handled through the Field API. Drupal doesn't think of tags as something quite so specific; it's thought of as a type of field. What that means is the theme function for tags is the same as the theme function for every other field, such as "title" or "body". Take a look at theme_field(), which can be found in /modules/field.module. Every time a field is rendered in Drupal, it's run through that same function.

If you're not satisfied with the default markup, you can tell theme() to route to your function, instead of to the Drupal default.

To override the default, copy and paste the entire theme_field() function into your theme's template.php file. The goal here is to register your theme's implementation of "field", so instead of using theme_field(), Drupal will use YOURTHEME_field() to render output.

1) Rename the copied function from theme_field to YOURTHEME_field.
2) Clear your Drupal cache.

That's it. The theme registry will know to look to your function to render output for fields. Now you can tweak the way field_tags in particular is rendered. This is done within YOURTHEME_field(). The snippet below will output tags a comma-delimited list.

/**
 * Override of theme('field').
 */
function YOURTHEME_field($variables) {
 
  $output = '';
 
  // Render the label, if it's not hidden.
  if (!$variables['label_hidden']) {
    $output .= '<div class="field-label"' . $variables['title_attributes'] . '>' . $variables['label'] . ':&nbsp;</div>';
  }
 
  // Render the items.
  $output .= '<div class="field-items"' . $variables['content_attributes'] . '>';
 
  if ($variables['element']['#field_name'] == 'field_tags') {
    // For tags, concatenate into a single, comma-delimitated string.
    foreach ($variables['items'] as $delta => $item) {
      $rendered_tags[] = drupal_render($item);
    }
    $output .= implode(', ', $rendered_tags);
  } else {
    // Default rendering taken from theme_field().
    foreach ($variables['items'] as $delta => $item) {
      $classes = 'field-item ' . ($delta % 2 ? 'odd' : 'even');
      $output .= '<div class="' . $classes . '"' . $variables['item_attributes'][$delta] . '>' . drupal_render($item) . '</div>';
    }
  }
  $output .= '</div>';
 
  // Render the top-level DIV.
  $output = '<div class="' . $variables['classes'] . '"' . $variables['attributes'] . '>' . $output . '</div>';
 
  return $output;
}

Tags:

How to display just a template in Drupal 7

aquariumtap — Tue, 30 Aug 2011 23:49:19 +0000

Web developers from a MVC background (Model View Controller) are likely keen on this pattern: do your business logic in a script, collect everything into a data object, then send that data object to a template file for display.

That separation exists in Drupal, but it's mitigated by theme().

This much is hammered to death in the docs: your view layer (or theme layer in drupalero-speak) should always be invoked by the theme() function. Why? Your module or theme can implement a default presentation layer, and by using theme(), other modules and themes will have the opportunity to override it. The most easily accessible themer tutorials seem to focus on the task of overriding templates provided by Drupal core.

As an example, blocks in Drupal are handled by block.module, and the block module provides a default template called block.tpl.php. To change the way all blocks look on your site, you can override block.tpl.php inside of your theme by copying and pasting the file into your theme's directory. Alternatively, you can override the block module's template file using a function called {my_module}_block(). No matter how you slice it, the same data will be sent to function or theme that rises to the top of the theme() hierarchy. Google around for "drupal theme suggestions" for more info on this topic, which is covered fairly well by existing documentation and blog posts.

So what if you have a module called custom and all you want is for mysite.com/custom to display a new file of your creation, custom.tpl.php?

Yes, use theme(). Here's what the documentation says, which I find misleading (but read it anyway, it's important and relevant):

All requests for themed output must go through this function. It examines the request and routes it to the appropriate theme function or template, by checking the theme registry.

The first argument to this function is the name of the theme hook. For instance, to theme a table, the theme hook name is 'table'. By default, this theme hook could be implemented by a function called 'theme_table' or a template file called 'table.tpl.php', but hook_theme() can override the default function or template name.

Why do I find it misleading? Because calling theme('custom') will not load theme('custom.tpl.php') unless the hook "custom" has been registered in the theme registry. The theme() documentation doesn't emphasize that or explain how it's done.

The registry is a performance necessity. Drupal's hook-heavy architecture means it needs to scan an indefinite amount of extensions (modules + themes) to look for implementations of hooks, template files, etc. This info is stored in a registry to avoid expensive I/O per page load. For Drupal to recognize a new implementation, you need to clear your Drupal cache and update the registry.

Back to the original problem: a custom template is loaded for a particular path using these steps. From a high level:

Implement hook_menu() in your custom module so a system path (eg: somedomain.com/somepath) is tied to a function of your choosing.
Choose theme(). The string 'theme" is the value for the key "page callback", and the "page argument" is the name of a custom theme hook. For more info on page callback and page argument, see the hook_menu() docs.
Register both your theme hook and its template in the theme registry by implementing hook_theme().

Once it's put into code, it's pretty simple, but it's very drupaly idiosyncratic / not obvious / hard to parse by just reading the docs. Here's a real example. My module is called fusiontap. (Whatever.)

/**
 * Implements hook_menu().
 */
function fusiontap_menu() {
  $items['aquariumtap/test'] = array(
    'title' => 'Sample Title',
    'page callback' => 'theme',
    'page arguments' => array('sample_template'),
    'access arguments' => array('access content'),
    'type' => MENU_NORMAL_ITEM,
  );
  return $items;
}
 
function fusiontap_theme($existing, $type, $theme, $path) {
  return array(
    'sample_template' => array(
      'template' => 'my_template',
    ),
  );
}

The function fusiontap_theme() registers the theme hook "sample_template" into the registry. Additionally, it tells Drupal to look for a template called "my_template". Once the search happens, the extension will automatically be added based on the loaded template engine -- by default, that'll be .tpl.php, which is used by PHPTemplate.

Next, make sure the file my_template.tpl.php exists in the same directory as your module. It doesn't have to be exactly there, but you can get creative with file structure on your own time. ;)

Finally, make sure to clear your Drupal cache to register your new functions and template file into the registry. Voilà.

Tags:

Three tips to speed up OS X Lion

aquariumtap — Fri, 05 Aug 2011 16:08:35 +0000

There's an understandable performance hit immediately after the OS X Lion upgrade: Spotlight is reindexing the hard drive. But several days later I was still running up against mediocre to poor performance, and I had to take a few extra steps to make my Mid 2010 MacBook Pro snappy again. Here are some tips.

Upgrade your hard drive to 7200RPM. I'm not 100% certain that Lion is heavier on I/O than Snow Leopard, but I suspect it is. The performance improvement after my upgrade was dramatic! Launch times for certain applications went from 30 seconds to 5 seconds. Searching through Mail is hugely improved. Spotlight no longer staggers. System startup time was cut in half. I chose this particular hard drive, which includes 4 GB of solid state memory: Seagate Momentus XT 500 GB 7200RPM SATA 2.5 Inch Solid State Hybrid Drive The most frequently used files will be stored in solid state, so subsequent application loads will improve over time. Very cool.
Clear your OSX cache. This is done by starting up your computer while holding the left shift key, letting it boot up completely, then restarting it normally (without holding the shift key). This improved performance for me, and also corrected a major issue I had with Finder locking up or displaying files incorrectly.
Chrome + MAMP + domains that end in ".local" are problematic. It felt like a DNS issue, so I renamed my virtual hosts from websitename.local to websitename.dev. The speed increase was substantial.

As an aside, the DigitalColor Meter in Lion no longer displays hex values. I've replaced it with the $5 ColorSnapper application, which is well worth the money. I find myself using it all the time.

Tags:

lion

performance

tips

How to change your content every X hours in Drupal

aquariumtap — Tue, 14 Jun 2011 14:13:00 +0000

Scenario: you have a list of three featured articles on your front page. You'd like to shuffle them every x hours. Here are a couple of configuration strategies:

Random sort order + time based cache

If your content is generated using Views, time-based caching will store the results of your query until it expires after a set interval of time. Combine that with "Global: Randomize", and you have a View that randomizes each time the cache is cleared, but not with every page refresh.

Nodequeue + Nodequeue Randomizer

If you'd like to hand-pick the pool from which your three featured articles are drawn, the nodequeue module allows creation of arbitrary list of nodes that can be weighted manually. Views can use the content and weights from a nodequeue to pull your three featured articles. To set up shuffling, install the Nodequeue Randomizer module. It randomizes the weights of your nodequeue at set intervals.

Tags:

How to share templates between Drupal Views

aquariumtap — Mon, 13 Jun 2011 14:37:02 +0000

Two separate Views with the same front end

Scenario: you're running a gaming blog. In the sidebar you have a block generated by Views that lists the 5 most popular games. It looks perfect because you've custom templated the hell out of it using the template suggestions listed under "Theming Information". For example, this is the list of templates available for the style output:

views-view-unformatted.tpl.php, 
views-view-unformatted--featured-games.tpl.php,  
views-view-unformatted--fringe.tpl.php, 
views-view-unformatted--default.tpl.php, 
views-view-unformatted--featured-games--default.tpl.php

The template in bold is the one in use.

One frisky fall morning you decide to create a new section of your gaming blog called "Fringe Games". You no longer want to display the five most popular games; instead you want a random selection, but the front end should look exactly the same as the other block.

You follow your instincts and clone the View, change the sort order, and hope to call it a day. But the front end doesn't look the same anymore because your custom templates refer to the machine name of the original view.

views-view-unformatted.tpl.php, 
views-view-unformatted--featured-games-fringe.tpl.php, 
views-view-unformatted--news.tpl.php, 
views-view-unformatted--default.tpl.php, 
views-view-unformatted--featured-games-fringe--default.tpl.php

You want to use views-view-unformatted--featured-games.tpl.php, but that's not an option. Now what?

Sharing templates between Views using Displays

The best option is to create a new Display on the original View instead of cloning. For something as minor as a fork in sort order, a new View is not necessary.

Your new Display will have the same template suggestions as the original Display, but you can override some parts of the query such as the filter or sort order.

Sharing templates between Views using Tags

If your Views are so different that overriding is too cumbersome or counter-productive, you can tag the two separate Views with the same keyword, and that'll give them an overlapping template suggestion. In the suggestion list below, views-view-unformatted--fringe.tpl.php is based on the View's tag.

views-view-unformatted.tpl.php,
views-view-unformatted--featured-games.tpl.php, 
views-view-unformatted--fringe.tpl.php,
views-view-unformatted--default.tpl.php, 
views-view-unformatted--featured-games--default.tpl.php

Many thanks to Lynette Miles for the tip about templates and tags! I highly recommend the book she and merlinofchaos wrote together, Drupal Building Blocks

Tags:

Mongo DB for Drupal 7 is an option for write-heavy sites

aquariumtap — Mon, 16 May 2011 00:04:27 +0000

MySQL replication can only support a single write database, and its speed is comparably slow. It's not always the right fit, particularly if transactions are not required.

Mongo DB is a document object store and alternative to a relational database system such as MySQL or PostgreSQL. Some heavy-hitters in the Drupal community have invested time into the mongodb module, which is the product of migrating examiner.com from Cold Fusion to D7. Examiner is an extremely high traffic site with about 25 page hits per second. Its social features are too important and disseminated throughout the site to leverage a reverse-proxy cache layers like Varnish, which is the optimal solution for many sites receiving anonymous (read-only) traffic.

The Mongo DB a partial replacement for RDMS in the areas of Drupal where it makes sense: fields, blocks, cache, session and the watchdog.

The performance gain can be substantial. Speaking only about Mongo as an data engine for Fields, Moshe Weitzman reported a 5x increase in write performance during informal benchmarks on his laptop. In combination with a smoother process for sharding, I imagine Mongo DB will become a viable option for community-driven and write-intensive sites where page caching is not an option.

There's a D6 version, but it's less supported.

Hilarious media:

Useful media:

Tags:

mongo

Context module terminology

aquariumtap — Sun, 15 May 2011 18:03:25 +0000

The Context module was initially an API without any user interface. As it matured, the developers of the module (Development Seed) added a very nice UI component to configure Contexts. In figuring out how to use the module, I found the terminology to be inconsistant and I wasn’t clear on how the UI and API related to one another. Here are some of my notes. If you see any errors, please correct me in the comments and I’ll update this.

Context terminology

attribute (API): See name.
context (UI, API): This sometime refers to the name field in the UI, which is called the attribute in the API, and key in the API.TXT. Other times it refers to a configuration of conditions and reactions; for example, a context has a condition of being a “blog” content type, and has a reaction of displaying a block.
condition (UI, API): Conditions are toggled and configured through the UI, but there is an API to extend available conditions. Instead of configuring conditions through the UI, it’s possible, for example, to call context_set() inside of a control statement inside your module’s implementation of hook_nodeapi.
description (UI): No programmatic purpose. It is analogous to a code comment. It provides a field for notation about the context in the UI.
global context (Forums): I’ve seen people talk about a “global” context. In the UI, this is referred to as a sitewide context. It’s a context with conditions that are always met (i.e., are sitewide, or global).
name (UI): It is a unique string used as a key in the registry to index a context. Also known as attribute or key.
namespace (API): A unique string, generally specific to your project, which is used to avoid name collisions with contrib modules. The namespace of a context created through the UI is always “context” (source).
reaction (UI, API): An event that is triggered by a set ofcontext conditions.
sitewide context (UI): A context with conditions that are always met (i.e., are sitewide, or global). It is created by selecting sitewide context from the conditions drop-down in the UI, and checking the checkbox.
tag (UI): This is analogous to tags in the Views module. Sites with large numbers of contexts can group them together logically by tagging them. They have no effect on context behavior.
value (API): This is only available through the API, and provides a valuable method of messaging between components in Drupal. When setting a context using context_set(), any valuable can be stored along with the context. A common use-case is for a module to save the loaded node ID into a context, which can later be retrieved by a block.

Additional reading:

What is the “attribute name”?
Class Hierarchy (doxygen docs by Treehouse Agency)
Context Everywhere (by Treehouse Agency)
Context Module Abridged (by Affinity Bridge)