Tech Advent Calendars

[perl] So here it is Merry Christmas

Mark Fowler — Thu, 25 Dec 2014 05:00:00 +0000

Another year of Perl Advent had drawn to a close, and what a year it's been. I hope you've enjoyed reading all the articles as much as I have.

Everybody's having fun

Of course, just because Advent is over for another year, doesn't mean the gift of Perl modules has to end. As I write this there are one hundred and forty one thousand seven hundred and forty nine modules on the CPAN and each year we only get to publish twenty four advent articles. So where can you read about the rest of these modules and other exciting things happening in Perl?

You can search for Perl modules at metacpan where you can also see a list of recently published modules
CPAN Ratings offers a ratings and reviews of Perl modules. It's like a democatic advent calendar each day of the year!
The Perl Weekly provides one small succinct email a week with a summary of the interesting things happing in Perl that week, including new modules, interesting articles and upcoming events.
blogs.perl.org hosts a vast number of Perl blogs if you're looking for more articles to read throughout the year. The Perl Ironman Challenge links to a lot more blogs as part of the updating your Perl blog weekly challenge
You're reading the original programming article for advent advent calendar, but if you like the format there's a whole collection of them available dedicated to other aspects of Perl (e.g. Catalyst, Dancer, Perl 6, Seoul.pm, MSTPAN, and perlancer.) Other languages have also adopted the format and Advent Planet has a meta calendar that links to each entry for that day for over twenty calendars
Of course, if you want more Perl Advent articles, there's always the previous fourteen years of advent calendar to go through. Happy reading!

The Perl Advent Calendar wouldn't be possible without the hard work of all the article authors. This year we owe thanks to Alex Balhatchet, Augustina Ragwitz, Dave Cross, Graham Ollis, John SJ Anderson, Legolas Greenleaf, Marcus Ramberg, Mark Allen, Mark Fowler, Neil Bowers, Nova Patch, Olaf Alders, Paul "LeoNerd" Evans and Ricardo Signes. These people not only devoted untold time writing and formatting their articles, but put up with me repeatedly hassling them with my self imposed deadlines and bugging them with copy corrections. Thank you for putting up with me.

Thanks also to everyone who submitted corrections, helped organize, setup hosting, debugged software issues, and otherwise did the unglamorous things that made the advent calendar happen.

Look to the future now

As a Perl programmer reading the calendar chances are you have an article in you for publishing next year even if you don't know what it is yet. I encourage you to subscribe to the mailing list where we organize all of this and simply drop us an email saying you'd be interested in writing (you'll get a follow up email from me in about eight months when we start working out scheduling for 2015.)

It's only just begun

Perl does a lot for you. It keeps you entertained, it makes your life easier, and chances are it keeps you employed. In this season of giving you might consider making a small donation to The Perl Foundation who do important work, including sponsoring developers to work full time on improving the core of Perl.

It's Christmas!

Merry Christmas, one and all.

[java] Merry Christmas everyone!

Olimpiu Pop — Thu, 25 Dec 2014 03:04:00 +0000

This is the first year of the Java Advent Project and I am really grateful to all the people that got involved, published articles, twitted, shared, +1ed, shared etc. etc.
It was an unbelievable journey and all the glory needs to go to the people that took some time from their loved ones to give us their wisdom. As they say, the Class of 2014 of Java Advent is comprised of (in the order of publishing date):

Attila-Mihály Balázs, Transylvania JUG member
Florin Bunau
Petri Kainulainen
Oleg Shelajev from ZeroTurnaround
The perfect plumbr for your web application, Nikita Salnikov-Tarnovski
If you want to hear out your developers, Trisha Gee, can help you do it properly
If you like to measure the performance in nanoseconds Peter Lawrey (stackoverflow) and OpenHFTcan help you with it!
Markus Eisele from Eisele.net
Rene Jansen
Mite Mitreski from mitemitreski.com
Jens Schauder
jOOQs papa - Lukas Eder(linkedin, twitter)
Mani Sarkar
Marcin Grzejszczak from Too much coding
Rudiger Moller
Martijn Verburg from jClarity
Alexander Turner from Nerds Central
And yours truly Olimpiu POP

Thank you girls and guys for making it happen yet once more. And sorry for stressing and pushing you out. Also, last but not least thanks to Voxxed editors Lucy Carey and Mite Mitreski.

[sysadvent] Day 25 - Windows has Configuration Management?!?

Christopher Webber — Thu, 25 Dec 2014 00:00:00 +0000

Written by: Steven Murawski (@stevenmurawski)
Edited by: William Shipway (@shipw)

Windows Server administration has long been the domain of “admins” mousing their way through a number of Microsoft and third party management UIs (and I was one of them for a while). There have always been a stalwart few who, by hook or by crook, found a way to automate the almost unautomateable. But this group remained on the fringes of Windows administration. They were labeled as heretics and shunned, until someone needed to do something not easily accomplished by a swipe of the mouse.

The sea winds have shifted and over the past seven or eight years, Microsoft released PowerShell and began focusing on providing a first class experience to the tool makers and automation-minded. The earlier group of tool makers and automators gained traction and began to develop a larger following, as more Microsoft and third party products added support for PowerShell. That intrepid group of early automators formed the core of the PowerShell community and began welcoming new converts - whether they were true believers or forced into acceptance by the lack of some capability in their comfortable management UIs. Now, most Windows Server administrators have delved into the command line and have begun to succumb to the siren call of automation.

Just as the PowerShell community’s evangelism was reaching a fevered pitch, Microsoft added another management tool - Desired State Configuration. The tool-makers and automators were stunned. Cries of “what about my deployment scripts?” and “but, I already built my VM templates!” echoed through the halls. Early adopters of PowerShell v3 lamented “isn’t this what workflows were for?”. Some had already begun to explore the dark arts of configuration management using tools like Chef and Puppet to bring order to their infrastructure management. With the help of those in the community who blazed a trail in implementing configuration management on Windows, those cries of dismay began to turn into rabid curiosity and even envy. The administrators began to read books like the Phoenix Project and hear stories from companies like Stack Exchange, Etsy, Facebook, and Amazon about this cult of DevOps. They wanted access to this new realm of possibilities, where production deployments don’t mean a week of late nights in the office and requests for new servers don’t go to the bottom of the pile to sit for a month to “percolate”.

Read on, dear reader to understand the full story of Desired State Configuration and its place in the new DevOps world where Windows Server administrators find themselves.

An Introduction to Desired State Configuration

With the release of Windows Server 2012 R2 and Windows Management Framework 4, Microsoft introduced Desired State Configuration (DSC). DSC consists of three main components: the Local Configuration Manager, a configuration Domain Specific Language (DSL), and resources (with a pattern for building more). DSC is available on Windows Server 2012 R2 and Windows 8.1 64 bit out of the box and can be installed on Windows Server 2012, Windows Server 2008 R2, and Windows 7 64 bit with Windows Management Framework 4. There is an evolving ecosystem around Desired State Configuration, including support for a number of systems management and deployment projects. To me, one of the most important benefits of the introduction of Desired State Configuration is the awakening of the Windows administration community to configuration management concepts.

A Platform Play

The inclusion of Desired State Configuration may seem like a slap in the face to existing configuration management vendors, but that is not the case. Desired State Configuration is a platform level capability similar to PerfMon or Event Tracing for Windows. DSC is not intended to wholesale replace other configuration management platforms, but to be a base which other platforms can build on in a consistent manner.

The Evolution of DSC

One of the major knocks against administering Windows servers in the past has been the horrendous story around automation. Command-line tools were either lacking coverage or just plain missing. The shell was in a sorry state.

Then, shortly before Windows Server 2008 shipped, PowerShell came about. Initially, PowerShell had relatively poor native coverage for managing Windows, but it worked with .NET, WMI, and COM, so it could do just about anything you needed.

More coverage was introduced with each release of Windows Server. Windows Server 2012 had an explosion of coverage via native PowerShell commands for just about everything on the platform.

PowerShell appeared to be the management API for configuring Windows servers. The downside of a straight PowerShell interface is that PowerShell commands aren’t necessarily idempotent. Some like Add-WindowsFeature are, and do the right thing if the command is run repeatedly. Others are not, like New-Website, which will throw errors if the site already exists.

DSC was introduced to provide a common management API that offers consistent behavior. Under the covers, it is mostly PowerShell that is running, but the patterns the resources follow ensure that only the work that needs to be done is done, and when a resource is in the proper state, that it is left alone.

Being a platform feature means that there is a consistent, supported mechanism for customers and vendors to manage and evolve the configured state of Windows servers.

Standards Based

Desired State Configuration was built using standards already supported on the Windows platform - CIM and WSMAN.

CIM, Common Information Model, is the DMTF standard that WMI is based upon and provides structure and schema for DSC.

WSMAN, WS-Management, is a web services protocol and DMTF standard for management traffic. WinRM and PowerShell remoting are built on this transport as well.

While these might not be the greatest standards in the world, they do provide a consistent manner for interacting with the Desired State Configuration service.

An Evolving API

Though Windows Management Framework (WMF) was just recently introduced (it has been released for just over a year), WMF 5 development is well under way and includes many enhancements and bug fixes. One major change is to make the DSC engine’s API more friendly to use by third-party configuration management systems.

There was also a recent rollup patch for Server 2012 R2 (KB3000850) that contains a number of bugfixes and some tweaks for ensuring compatibility with changes coming in WMF 5.

Diving In

Now that we’ve got a bit of history and rationale for existence out of the way, we can dig in to the substance of Desired State Configuration.

The Local Configuration Manager

The engine that manages the consistency of a Windows server is the Local Configuration Manager (LCM). The LCM is exposed as a WMI (CIM) class (MSFT_DscLocalConfigurationManager) in the Root/Microsoft/Windows/DesiredStateConfiguration namespace.

The LCM is responsible for periodically checking the state of resources in a configuration document. This agent controls

whether resources are allowed to reboot the node as part of a configuration cycle
how the agent should treat deviance from the configuration state (apply and never check, apply and report deviance, apply and autocorrect problems)
how often consistency checks should be run
and more…

It has a plugin/extension point with the concept of Download Managers. Download Managers are used for Pull mode configurations. There are two download managers that ship in the box, one using a simple REST endpoint to retrieve configurations and one using a SMB file share. As it currently stands, these are not open for replacement by third parties (but it could be made so - please weigh in to the PowerShell team about that before WMF 5 is done!).

A Quick Note - Push vs. Pull

DSC configurations can be imperatively pushed to a node (via the Start-DscConfiguration cmdlet or directly to the WMI API), or if a Download Manager is configured it can pull a configuration and resources from a central repository (currently either SMB file share or REST-based pull server). If a node is in PULL mode, when a new configuration is retrieved, it is parsed to find the various modules required for the configuration to be applied. If any of the requisite modules and versions are not present on the local node, the pull server can supply those.

DSC Resources

Resources are the second major component of the DSC ecosystem, and are what make things happen in the context of DSC. There are three ways of creating DSC resources: They can be written in PowerShell, as WMI classes, or in Windows Management Framework 5, as PowerShell classes. As PowerShell class-based resources are still an experimental feature and the level of effort to create WMI based resources is pretty high, we’ll focus on PowerShell-based resources here.

DSC resources are implemented as PowerShell modules. They are hosted inside another PowerShell module under a DSCResources folder. The host module needs to have a module metadata file and have a module version defined in order for it to host DSC resources.

The resources themselves are PowerShell modules that expose three functions or cmdlets:

Get-TargetResource
Test-TargetResource
Set-TargetResource

Get-TargetResource returns the currently configured state (or lack thereof) of the resource. The function returns a hashtable that the LCM converts to an object at a later stage.

Test-TargetResource is used to determine if the resource is in the desired state or not. It returns a boolean.

Set-TargetResource is responsible for getting the resource into the desired state. Set-TargetResource is only executed after Test-TargetResource.

The Configuration DSL

Also introduced with Desired State Configuration are some domain specific language extensions on top of PowerShell. Actually, Windows Management Framework 4 added some public extension points in PowerShell for creating new keywords, which is what DSC uses.

Stick with me here, as it may get a bit confusing - I’ll be using “configuration” in two contexts. First is the configuration script. This is defined in PowerShell and can be defined in a script file, a module, or an ad hoc entry at the command line. The second use of “configuration” is in the context of the configuration document. This is the final serialized representation of the configuration for a particular machine or class of machines. This document is in Managed Object Format (MOF) and is how CIM classes are serialized.

The first keyword defined is configuration. The configuration keyword indicates that the subsequent scriptblock will be a configuration document and should be parsed differently. All your standard PowerShell constructs and commands are valid inside of a configuration, as are a few new keywords. There are two static keywords and a series of dynamic keywords in a configuration document.

The first two static keywords are node and Import-DscResource. I’ll deal with the latter first, since it seems very oddly named. Import-DscResource looks in name like a cmdlet or function, but is a keyword that is valid only in a configuration document and only outside of the context of a node. Import-DscResource identifies custom and third-party modules to make available in a configuration document. By default, only DSC resources in modules located at $pshome/modules (usually c:\windows\system32\windowspowershell\v1.0\modules) can be used without using Import-DscResource and specifying which modules to make resources available from. The second static keyword is the node keyword. Node is used to identify the machine or class of machines that the configuration is targeted at. Resources are generally assigned inside node declarations.

The configuration also includes a number of potential dynamic keywords which represent the DSC resources available for the configuration.

An example configuration script looks something like:

configuration SysAdvent
{
    Import-DscResource -ModuleName cWebAdministration

    node $AllNodes.where({$_.role -like 'web'}).NodeName
    {
      windowsfeature IIS
      {
        Name = 'web-server'
      }

      cWebsite FourthCoffee
      {
        Name = 'FourthCoffee'
        State = 'Started'
        ApplicationPool = 'FourthCoffeeAppPool'
        PhysicalPath = 'c:\websites\fourthcoffee'
        DependsOn = '[windowsfeature]IIS'
      }

    }
}

The above configuration script, when run, creates a command in the current PowerShell session called SysAdvent. Running that command will generate a configuration document for every server in a collection that has the role of a web server. The configuration command has a common parameter of ConfigurationData which is where AllNodes comes from (more on that in a bit). The result of this command will be a MOF document describing the desired configuration for every node identified as a web server.

MOF documents created by the command are written in a folder (of the same name as the configuration) created in the current working directory. Files are named for the node they represent (e.g. server1.mof). You can specify a custom output location. Here is our newly created MOF document:

/*
@TargetNode='localhost'
@GeneratedBy=Administrator
@GenerationDate=12/22/2014 04:12:56
@GenerationHost=ARMORY
*/

instance of MSFT_RoleResource as $MSFT_RoleResource1ref
{
SourceInfo = "::7::7::windowsfeature";
 ModuleName = "PSDesiredStateConfiguration";
 ModuleVersion = "1.0";
 ResourceID = "[WindowsFeature]IIS";
 Name = "web-server";

 ConfigurationName = "SysAdvent";

};
instance of PSHOrg_cWebsite as $PSHOrg_cWebsite1ref
{
ResourceID = "[cWebsite]FourthCoffee";
 PhysicalPath = "c:\\websites\\fourthcoffee";
 State = "Started";
 ApplicationPool = "FourthCoffeeAppPool";
 SourceInfo = "::12::7::cWebsite";
 Name = "FourthCoffee";
 ModuleName = "cWebAdministration";
 ModuleVersion = "1.1.1";

DependsOn = {

    "[windowsfeature]IIS"};

 ConfigurationName = "SysAdvent";

};
instance of OMI_ConfigurationDocument
{
 Version="1.0.0";
 Author="Administrator";
 GenerationDate="12/22/2014 04:12:56";
 GenerationHost="ARMORY";
 Name="SysAdvent";
};

Other Tidbits

There are a few other things one should know in preparation for digging into DSC.

ConfigurationData and AllNodes

Configurations have support for a convention-based approach to separating environmental data from the structural configuration. The configuration script represents the structure or model for the machine, and the environmental data (via ConfigurationData) fleshes out the details.

ConfigurationData is represented by a hashtable with at least one key - AllNodes. AllNodes is an array of hashtables representing the nodes that should have configurations generated and becomes an automatic variable that can be referenced in the configuration (like in the example above). The value provided in $ConfigurationData is also referenced and you can create custom keys and reference those in your configuration document. The PowerShell team reserves the right to use any key in the ConfigurationData hashtable that is prefixed with PS.

Example:

$ConfigurationData = @{
  AllNodes = (
      @{NodeName = '*', InterestingData = 'Every node can reference me.'}
      @{NodeName = 'Server1'; Role = 'Web'},
      @{NodeName = 'Server2'; Role = 'SQL'},
  )
}

Sysadvent -ConfigurationData $ConfigurationData

DependsOn

Resources in DSC are not ordered by default and there is no guarantee of ordering. The current WMF 4 implementation and the previews of WMF 5 all seem to serially process resources, but there is NO guarantee that will stay that way. If you need things to happen in a certain order, you need to use DependsOn to tell a resource what needs to happen first before that one can execute.

Node Names

In PUSH mode, the node name is either the server name, FQDN, or IP address (any valid way you can address that node via PowerShell remoting).

In PULL mode, the node name is not the server name. Servers are assigned a GUID and they use that to identify which configuration to retrieve from a pull server. Where this GUID comes from is up to you - you can generate them on the fly, pull one from AD, or use one from another system. Since the GUID is the identifier, you can use one GUID to represent an individual server or a class of servers.

WMF 5 - In Production

If you are running Windows Server 2012 R2, you can stay on the bleeding edge AND get production support. The PowerShell team recently announced that if you are using WMF 5, you can get production support for what they call “stable” designs - those features that either existed in previous versions of the Management Framework or have reached a level that the team is ready to provide support. Other features, which are more in flux, are labeled experimental and don’t carry the same support level. With this change, you can safely deploy WMF 5 and begin to test new features and get the bug fixes faster than waiting for the full release. WMF previews are released roughly quarterly.

With WMF 5, you can dig into new and advanced features like Debug mode, partial configurations, and separate pull servers for different resource types.

Building an Ecosystem

No tooling is complete without a community around it and Desired State Configuration is no different.

PowerShellGet and OneGet

OneGet and PowerShellGet are coming onto the scene with WMF 5 (although after they release they should be available somewhat downlevel too). OneGet is a package manager manager and provides an abstraction layer on top of things like nuget, chocolatey, and PowerShellGet, and eventually tools like npm, RubyGems, and more. PowerShellGet provides a way to publish and consume external modules, including those that contain DSC resources.

Finding new resources becomes as easy as:

Find-Module -Includes DscResource

Third Parties

Chef

Back in July 2014, Chef made a preview of our DSC integration available (video, cookbook) and in September shipped our first production-supported integration (the dsc_script resource) and have more coming. DSC offers Chef increased coverage on the Windows platform.

ScriptRock

The guys at ScriptRock (full disclosure - they are friends of mine) have done a pretty interesting thing by taking a configuration visualization and testing tool and offering an export of the configuration as a DSC script. Very cool.

Puppet

There is a Puppet module on the Forge showing some DSC integration. I’m not too familiar with the state of that project, but it’s great to see it!

Aditi

Brewmaster from Aditi is a deployment tool and can leverage DSC to get a server in shape to host a particular application, allowing you to distribute a DSC configuration with an application.

PowerShell.Org

PowerShell.Org hosts a DSC Hub containing forums, blog posts, podcasts, videos and a free e-book on DSC.

So, What Are You Waiting For?

Start digging in! There’s a ton of content out there. Shout at me on Twitter (@stevenmurawski) or via my blog if you have any questions.

[24ways] Cohesive UX

feeds@allinthehead.com (Cameron Moll) — Wed, 24 Dec 2014 12:00:00 +0000

Cameron Moll brings the tenth 24 ways to a close with a look at the increasing need for common experiences across devices. Despite our differences, there are more things we share than divide us. Merry Christmas!

Brought to you by Beanstalk. Celebrate shipping code again! With Beanstalk, your entire team can ship code as simple as git push.

[perl] Out of Order Perl

Augustina Ragwitz — Wed, 24 Dec 2014 05:00:00 +0000

This article will cover asynchronous programming with the AnyEvent library and will show use cases for managing multiple asynchronous requests in a single application. In addition, I hope to introduce good techniques for using metrics to drive technology decisions!

Is an Asynchronous Solution right for me?

New software techniques and practices are always rearing their heads in the industry. While asynchronous functionality is not new to Perl, it is not widely used where it could be, and maybe over-used where it shouldn't be. When considering whether or not to adopt some new methodology on existing software, it's important to make sure you have clearly identified what problem you are trying to solve. It's fine if you want to play around with something new, but remember that every solution has a cost. The cost of asynchronous functionality is that it can be difficult to read and debug (callback soup anyone?) and you have to consider compatibility of your current web framework as well as your current code base.

Step 1: Identify the Problem

“70% of users are unable to download the TPS Report from the website because the website times out.”

“Web site latency is reported for 75% of users.”

Step 2: Identify the cause the clarify the problem (with numbers!)

To identify the cause, you can check logs, collect metrics (a lot of folks use statsd), or attempt to reproduce the problem in a staging environment.

“The TPS report database query takes 30 seconds to run”

“Requests to HTTP services are averaging 1 second because each request is made individually and is only made once the previous request is complete.”

Step 3: Propose a solution (based on data and facts)

Now that you've collected data and clarified your problem in terms of metrics, you can say "Asynchronous HTTP requests would reduce overall report loading time to only the amount of time it takes the longest request to return."

Implement a Solution based on your results... and use metrics to determine success There is a fully functioning sample application! The sample application from which the code snippets in this article come is located on github.

The sample application is a simple Mojolicious::Lite application the generates a “TPS” report. This report makes calls to 2 external services to collect data and then uses that data to make a complex call to the database. I'm just using simple timing metrics to show the results for each of the reports. You can run it yourself to see the results as it records them in the database. First Improvement: Make it Async!

As we ascertained from our problem statement exercise just now, kicking off HTTP requests at the same time would drastically improve our report's performance.

Here's how you currently use LWP to make HTTP requests:

`1: 2: 3: 4: 5: 6: 7: 8:`	`my $ua = LWP::UserAgent->new(); my $req = HTTP::Request->new( GET => $url ); # make the request, and wait until we get the results back my $res = $ua->request($req); my $content = $res->content; return $content;`

Here's how it looks in our sample application:

`1: 2: 3: 4: 5: 6: 7: 8:`	`while (my ($service_name, $url) = (each %{$self->urls})) { my $req = HTTP::Request->new( GET => $url ); my $res = $self->_user_agent->request($req); my $content = $res->content; $http_data->{$service_name} = $content; } return $http_data;`

Notice that these requests happen for EACH URL in our list for EACH row in the database results one after the other, waiting for the previous request to return before starting the next. If we have 10 URLs and each one takes 3 milliseconds, our total request takes 10 x 3 milliseconds or 30 milliseconds total. If one of them takes any more time than that, it will hold up processing of the others and makes our request take even longer.

There are a lot of asynchronous libraries in the CPAN for you to consider. I'm using AnyEvent because we're going to be talking to RabbitMQ later and AnyEvent::RabbitMQ is one of the better libraries for doing that. When considering what asynchronous library to use, you always need to think about your requirements and to determine if it's likely you'll be adding other asynchronous functionality. Swapping one out for another, however, isn't impossible as they all pretty much follow the same rules (they just call things by different names). What you'll learn here about how to use AnyEvent will also apply in concept to any other library you might want to use instead.

Here's what the asynchronous solution looks like in our sample application using AnyEvent::HTTP.

`1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24: 25: 26: 27:`	`my $cv = AnyEvent->condvar; my $result; $cv->begin(sub { shift->send($result) }); while (my ($service_name, $url) = (each %{$self->urls})) { $cv->begin; my $request; $request = http_request( GET => $url, timeout => 2, # seconds sub { my ($content, $headers) = @_; $result->{$service_name} = $content; undef $request; $cv->end; } ); } $cv->end; return $cv->recv;`

Let's add a few more comments to try and make it a little clearer what's going on:

`1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24: 25: 26: 27: 28: 29: 30: 31: 32: 33: 34: 35: 36: 37: 38: 39: 40: 41: 42: 43: 44: 45: 46: 47: 48: 49: 50: 51: 52: 53: 54: 55:`	# A condition variable. Basically a mechanism to tell if we're done # processing all the requests or not. my $cv = AnyEvent->condvar; my $result; # tell the condition variable that it's not "ready" until it sees the "end" # call at the end of the code signifying we're done setting up all the # requests. This is good practice to avoid accidentally completing # while we're still setting up all the requests. # # Also pass the code block that will be executed at this point when the # condition variable is "ready". $cv->begin(sub { shift->send($result) }); # for each of our urls, start a request in parallel while (my ($service_name, $url) = (each %{$self->urls})) { # add another thing to the list of things that must be completed # before the condition variable is "ready". i.e. let the condition variable # know that we must wait until the HTTP request returns and the callback # calls a corresponding "end" on the condition variable before we're done. $cv->begin; # schedule a HTTP request to made asynchronously. Once it's done # and we've got the content, call the callback. my $request; $request = http_request( GET => $url, timeout => 2, # seconds sub { my ($content, $headers) = @_; # save the content for this request (i.e. what we downloaded) in our # result $result->{$service_name} = $content; # clean ourselves up undef $request; # and let the condition variable know that this scheduled thing # is done and once the last one is done it can fire. $cv->end; } ); } # signify that we're done with stage of setting up all the requests. The # condition variable become "ready" until all the requests that called "begin" # above have also completed and called a corresponding number of "end" calls $cv->end; # wait (i.e. block) until the condition variable is ready, i.e. until a # corresponding number of "end" calls have been called for each "begin" call return $cv->recv;

We'll get back to this code, explaining in more depth each section of code bit by bit once we've learned a little more about AnyEvent.

First, let's look at the advantage of using this significantly more complicated code. With an asynchronous solution, if each request takes 3 milliseconds, then we get our results back in 3 milliseconds. If one of those requests takes 1 second, then we get all of our results back in 1 second. The time it takes to get our results back takes only as long as the longest request!

What does it mean to be Asynchronous?

Asynchronous code is event-driven

An event queue manages code execution. If you've ever had to do any work with user interfaces, you should be familiar with the notion of an event-listener. An event in a user interface would be a button click. You would register a function to execute when the the button is pushed. In our example, the function (callback) we've registered with the event we're interested in is pushed to an event queue that fires after all the other code executes. The event loop is a queue of callback functions. When an asynchronous function executes, the callback function is pushed into the event queue.

Results are returned in the background while your application does other things

The rest of your application proceeds ahead using a placeholder for the result until it reaches a point where it can't go any further without the actual result (for example writing that result out to a data store).

The basic formula for Async: Event Loop + Listener + Callbacks

A listener registers your interest in a particular event. A callback is a function that runs when your request is in a ready state. No matter what Async library you use, all of them follow these rules.

Now we'll talk about more specifically what this looks like in the AnyEvent libraries.

AnyEvent 101

`condvar`

AnyEvent is just an abstraction layer on top of event loops. The condvar() method initializes the condition variable which represents a value that may or may not be available. The condition variable doesn't have a value until it is “ready”, meaning the request has completed. You use the condvar() method to interface with the event loop.

`send`

The send() method sets the conditional variable to ready. This will indicate that one event is complete. In the case of multiple requests, you can bind a callback to send() so that you can store the values retrieved each time the condvar is in a ready state.

`recv`

This returns a value when the condvar is in a ready state. Note that this is blocking because this is called at the point where your program cannot continue until it has the value it needs. I'll show why this is important later when we introduce RabbitMQ and have to manage multiple asynchronous requests.

`begin` and `end`

These methods are syntactic sugar for using a counter within an event loop. Begin increments the counter and end decrements it. The event loops runs until the counter is at 0. You can use these methods when you're managing multiple requests within an event loop and you want to make sure the event loop doesn't terminate until all of the requests you want have finished.

If you were to implement this yourself, you could create a loop where you create a condvar for each individual request. This syntactic sugar lets you declare just one <Ccondvar> and then specify when you want to set its ready state so recv can return the result.

AnyEvent::HTTP Code Walkthrough

Remember that code sample from earlier? Well now that we have a little better vocabulary, we can walk through it and understand what's happening in a little more detail, and hopefully understand how we came to write the code like that.

The first step is to initialize the condvar. Remember at this point, it has no actual value!

`1:`	`my $cv = AnyEvent->condvar;`

Next, you need to determine what you want to return at the ready state and assign it. This is where async programming can get a little tricky (hence the title of this article)! You need to think backwards when you start thinking async. Think from the result and work your way back. Decide at what point your application absolutely needs this data and put your recv call there.

`1: 2:`	`...all the other code... return $cv->recv;`

Now all we need to work out what code goes between these two statements. We need to figure out when to call the send event so we can capture our result. We're going to call it within a callback bound to a begin call. Each time we want to fetch data from a URL, we'll increment our event loop counter so that we don't exit out of the event loop before we've gotten a response for each of our URL requests. So, this means, in English: Once all the URLs are fetched, set the condvar to ready and return the result.

`1: 2: 3: 4:`	`my $cv = AnyEvent->condvar; $cv->begin(sub { shift->send($result) }); # <------------- we add this ... return $cv->recv;`

Because we've declared a begin, we need to declare an end outside of the loop to ensure that send gets called. If our loop has no data (in the while block) we would loop forever without this call to end here. This ensures the send we registered with our begin up above gets called.

`1: 2: 3: 4: 5:`	`my $cv = AnyEvent->condvar; $cv->begin(sub { shift->send($result) }); ... $cv->end; # <------------- we add this return $cv->recv;`

Now that we've defined our start and end points, we can focus on the actual logic around retrieving the URLs. We're going to use the AnyEvent::HTTP library, in particular the http_request method. Calling http_request means that you want the request to be made whenever it's possible (versus a procedural call which would mean stop everything and do it right now). This registers an IO watcher and tells the event loop that we are interested in this IO event.

When the request finishes, we want to invoke the callback that we've bound to the function call.

This method doesn't actually do the request, it returns right away and tells the event loop to do the request whenever it can. When this chunk of code runs, the event loop is not running because this chunk of code has control.

`1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20:`	`my $cv = AnyEvent->condvar; my $result; $cv->begin(sub { shift->send($result) }); while (my ($service_name, $url) = (each %{$self->urls})) { ... my $request; <---- $request = http_request( <---- GET => $url, <---- we add timeout => 2, # seconds <---- all of sub { <---- this my ($content, $headers) = @_; <---- code $result->{$service_name} = $content; <---- ... } <---- ); <---- } $cv->end; return $cv->recv;`

We need to unregister our interest in this event once we're done with it.

`1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23:`	`my $cv = AnyEvent->condvar; my $result; $cv->begin(sub { shift->send($result) }); while (my ($service_name, $url) = (each %{$self->urls})) { ... my $request; $request = http_request( GET => $url, timeout => 2, # seconds sub { my ($content, $headers) = @_; $result->{$service_name} = $content; undef $request; # <------------- we add this ... } ); ... } $cv->end; return $cv->recv;`

The inner begin and end calls define the individual requests that we want to track in the event loop. For each begin, the AnyEvent event loop will increment a counter. For each end the AnyEvent event loop decrements the counter. Once the counter reaches 0, AnyEvent can set the condvar to ready.

`1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24: 25: 26: 27:`	my $cv = AnyEvent->condvar; my $result; $cv->begin(sub { shift->send($result) }); while (my ($service_name, $url) = (each %{$self->urls})) { $cv->begin; # <------------- we add this my $request; $request = http_request( GET => $url, timeout => 2, # seconds sub { my ($content, $headers) = @_; $result->{$service_name} = $content; undef $request; $cv->end; # <------------- and we add this } ); } $cv->end; return $cv->recv;

And that's it, we've got a working example!

When should I use Asynchronous Programming?

An asynchronous solution works best for problems that meet all of the following criteria:

You need to do many things that could take awhile
You don't care about the order you do those things in

Given that an asynchronous solution can be hard to read and potentially hard to maintain, you should use metrics to determine if this added complexity really buys you anything.

Next Improvement: Let's create a work queue!

Work queues are good for throttling the work your application has to do, for scheduling when this work needs to happen, and is a good solution for anything that could potentially take a long time thus causing a timeout experience for the end user. To create a work queue, you need a message queue where one part of your application can post messages to (work to be done) and where another part of your application or a separate worker process can access the messages and perform the work.

You can throttle the database load by determining how many messages your worker handles at one time. This could be as simple as a while loop that pauses in between processing. You could also spin up more workers if your message queue gets too long. If you need to do massive updates to lots of records such that you don't want to do them at a peak time when the database indexes are constantly being updated due to a large number of writes, you can schedule your worker processes to run at an off-peak time. This can be as simple as a perl script you write and then use a cron job to schedule.

In our example, let's say our users all hit the database the same time every week for these reports. These reports hit the database pretty hard which causes timeouts for other users and reports to fail to complete. Again, before we jump into adding yet more complexity to our technology stack, we need to consider if this solution is the best one for our problem.

Other solutions could be to move our single database instance into a cluster, or to pre-compute the data into predictable date chunks. If we have a table that has a lengthy history we could look at windowing or partitioning of the table to limit the total results the query has to run against. Let's say in this situation, we don't have the resources to make changes to the database. Apparently that's another department and there's a lengthy red tape process we need to follow in order to get anything done in that regard. While that points to greater systemic problems at our example company, this does often reflect reality and we still need to come up with a solution.

Enter RabbitMQ. RabbitMQ runs as a separate process which the various Perl processes communicate with via AMQP (Advanced Message Queuing Protocol) and is implemented in Erlang. RabbitMQ essentially receives and forwards messages. You could also use Redis and the Redis::Requeue library for the same thing, but RMQ offers us an asynchronous option.

What does Asynchronous Programming have to do with RabbitMQ?

Sockets are traditionally a blocking connection. The actual connection attempt blocks until the connection is available. So to connect to RabbitMQ, we would need to wait for the HTTP connection and we would need to be able to respond to error messages. AnyEvent is an asynchronous library and AnyEvent::RabbitMQ is the canonical Perl library for interacting with RabbitMQ.

The reason using an asynchronous connection with RabbitMQ is important is because RabbitMQ sends a heartbeat to determine if the connection is still active. Most web framework applications by their nature are blocking, code is only triggered when a specific request (usually via a URL route) is made.

Once the publisher has established a connection, RabbitMQ can create an exchange. This exchange handles all incoming updates and redistributes them to the available queues. Once the exchange is created, RabbitMQ can send messages.

If the web application does not respond to the RMQ heartbeat, RMQ will hang up on you thereby closing the connection. So unless you want to hang up and reconnect every time you want to send a message to RabbitMQ, you'll need to double check that your web framework has a way to respond to this heartbeat.

In the example code, I've chosen to use Mojolicious because Mojo::IO Loop handles multiple reactor backends.

Step 1: Replace the original “work” section with publishing a message to RabbitMQ

Some things to think about:

do you need to keep a record of the queue data?
do you need to recreate the queue if the messages are lost?

First we need to set up our RabbitMQ connection. This is in the main.pl file of the worker queue version of the sample code.

`1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11:`	`my $cv = AnyEvent->condvar; my $ar = AnyEvent::RabbitMQ->new->load_xml_spec()->connect( host => 'localhost', port => 5672, user => 'guest', pass => 'guest', vhost => '/', on_success => sub { ... }, ... );`

This essentially connects to the message queue and then calls the on_success callback. Inside this on_success callback we then open the channel:

`1: 2: 3: 4: 5: 6: 7:`	`on_success => sub { my $ar = shift; $ar->open_channel( on_success => sub { ... } ... ); },`

And inside the on_success callback of the open_channel we have the bit of code that actually puts stuff on the message queue:

`1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24:`	on_success => sub { my $channel = shift; # use a named queue "reports" $channel->declare_queue( queue => 'reports', auto_delete => 0, ); # publish (send) to the message queue my %publish_args = ( header => { content_type => 'application/json', }, body => encode_json($params), routing_key => 'reports', ); $channel->publish(%publish_args); # and update the condition variable to say we're done with the # queue sending part $cv->send("Added report request to queue"); }, ...

Step 2: Move the work to a worker process

Now we've got something requesting work, we need a worker on the other end of the message queue actually doing the work.

You can schedule your worker as a cron job which works with RabbitMQ because there isn't any overstepping if the previous job hasn't finished yet. You can also use AnyEvent::timer on a loop if you aren't a fan of cron jobs. There a quite a few options for running your worker process on a schedule, the sample application just uses a simple Perl script with the idea that it would be a cron job.

The worker will need to take a message, process it, then grab the next one. You can have as many of these running as you want depending on your work load. To determine your work load, log metrics and keep an eye on them. Don't make assumptions that aren't based on actual data!

We'll use the consume() method to get the next message in the message queue. Once we get the message, we get the delivery tag for communication with the queue later. The payload is the queue item content. In our example we have this encoded in JSON so we need to decode it to do anything with it. This code can be found in the report_worker.pl file.

`1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24:`	# connect to the message queue as in the previous example $ar->open_channel( on_success => sub { my $channel = shift; ... # consume (read) messages when they arrive $channel->consume( on_consume => sub { my ($message) = @_; my $frame = $message->{deliver}->method_frame; my $delivery_tag = $frame->delivery_tag; my $params = decode_json($message->{body}->payload); # do work specified in %params... # confirm that we've done the work and the message # queue can drop the message $channel->ack(delivery_tag => $delivery_tag); } ); }, );

The Gotcha

Here's what the work looks like in the do work section mentioned above. We create a ReportBuilder instance and tell it to build the report.

`1: 2: 3: 4:`	`my $report_builder = Poo::ReportBuilder->new(); my $report = $report_builder->build_report($params); my $report_json = encode_json($report_data); my $report = Poo::Report->new(\%params)->save;`

Now here's the gotcha, and this is an easy mistake to make when you're first dealing with asynchronous programming: Remember that our build_report function itself uses AnyEvent::HTTP to build the reports - to run more than one web request at a time - and we didn't write it in a way that plays well with the AnyEvent code we just wrote to deal with the message queue.

If you were to run this code as is, you would get the following error:

  AnyEvent::CondVar: recursive blocking wait attempted at ../lib/Poo/ReportBuilder.pm line 238

For reference, I've implemented this in the async version of the application in Github with a comment explaining to not do that. Use that code to play around with adding another asynchronous call (say by adding the RabbitMQ code) and seeing what happens.

Remember, we have to call recv at the point in our code when we absolutely need the result to continue: Because our program cannot continue until this method returns something, recv() is considered a blocking function, but we don't want to block in the middle of generating a report - we need to be going back to the event loop in order to respond to those heartbeats!

recv always needs to be at the top level. We need to use callbacks at the highest level, not the condvar. Anything that depends on any return value shouldn't depend on a variable, it should bind to a callback.

This is also why asynchronous programming can be very confusing and also very hard to maintain and debug. There are some techniques to improve this workflow, including Promises or Futures, but that's outside of the scope of this article. I do encourage you to research them though as they are reasonable solutions for dealing with callback soup.

Our original build_report method had this code:

`1: 2: 3: 4: 5: 6:`	`sub build_report { ... my $http_data = $self->_get_data_from_urls( \%urls_list ); # do stuff with $http_data... }`

We need to change this to bind to a callback so a caller can manage their own condvar and their own recv call.

`1: 2: 3: 4: 5: 6: 7: 8: 9: 10:`	`sub build_report { ... $self->_get_data_from_urls( \%urls_list, sub { # do stuff with $http_data... ... } ); }`

We'll alter the _get_data_from_urls() method by removing the recv method and by using the cb method instead. Whenever it completes and something is sent to the condvar, the cb method of the condvar calls the specified function as soon as it completes. The caller is responsible for providing a callback.

`1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24: 25: 26: 27: 28: 29: 30: 31: 32: 33: 34: 35:`	sub _get_data_from_urls { my $urls = shift; my $callback = shift; my $cv = AnyEvent->condvar; my $result; $cv->begin(sub { shift->send($result) }); while (my ($customerid, $urls) = (each %{$urls})) { for my $service_name (keys %{$urls}) { $cv->begin; my $request; $request = http_request( GET => $urls->{$service_name}, timeout => 2, # seconds sub { my ($body, $hdr) = @_; $result->{$customerid}->{$service_name} = $body; $cv->end; } ); } } $cv->end; # this replaces "my $http_result = $cv->recv;" # <-- # now the condition variable calls the callback when it # <-- new # reaches the ready state # <-- code $cv->cb($callback); # <-- }

Sample Code Available

Again, I want to reiterate, I have sample code available for you to download and play with. Please experiment with this, try setting different values and using your debugger to step through the code to see when things get called and what values they produce. Asynchronous programming isn't something you really understand until you have to use it, and it's worth understanding.

[java] A Musical Finale

Alexander Turner — Wed, 24 Dec 2014 00:30:00 +0000

What could be more fitting than Christmas music for Christmas Eve?

In this post I want to discuss the joy of making music with Java and why/how I have come to use Python...

But first, let us celebrate the season!

We are all human and irrespective of our beliefs, it seems we all enjoy music of some form. For me some of the most beautiful music of all was written by Johan Sebastian Bach. Between 1708 and 1717 he wrote a set of pieces which are collectively called Orgelbüchlein (Little Organ Book). For this post and to celebrate the Java Advent Calendar I tasked Sonic Field to play this piece of music modelling the sounds of a 18th century pipe organ. If you did not know, yes some German organs of about that time really were able to produce huge sounds with reed pipes (for example, Passacaglia And Fugue the Trost Organ). The piece here is a 'Choral Prelude' which is based on what we would in English commonly call a Carol to be sung by an ensemble.

BWV 610 Jesu, meine Freude [Jesus, my joy]
This performance dedicated to the Java Advent Calendar
and created exclusively on the JVM using pure
mathematics.

How was this piece created?
Step one is to transcribe the score into midi. Fortunately, someone else already did this for me using automated score reading software. Not so fortunately, this software makes all sorts of mistakes which have to be fixed. The biggest issue with automatically generated midi files is that they end up with overlapped notes on the same channel; that is strictly impossible in midi and ends up with an ambiguous interpretation of what the sound should be. Midi considers audio as note on, note off. So Note On, Note On, Note Off, Note Off is ambiguous; does it mean:

One note overlapping the next or:
-----------------
---------------

One note entirely contained in a longer note?
----------------------------
----

Fortunately, tricks can be used to try and figure this out based on note length etc. The Java decoder always treats notes as fully contained. The Python method looks for very short notes which are contained in long ones and guesses the real intention was two long notes which ended up overlapped slightly. Here is the python (the Java is here on github).


def repareOverlapMidi(midi,blip=5):
    print "Interpretation Pass"
    mute=True
    while mute:
        endAt=len(midi)-1
        mute=False
        index=0
        midiOut=[]
        this=[]
        next=[]
        print "Demerge pass:",endAt
        midi=sorted(midi, key=lambda tup: tup[0])
        midi=sorted(midi, key=lambda tup: tup[3])
        while index<endAt:
            this=midi[index]
            next=midi[index+1]
            ttickOn,ttickOff,tnote,tkey,tvelocity=this
            ntickOn,ntickOff,nnote,nkey,nvelocity=next
    
            # Merge interpretation
            finished=False
            dif=(ttickOff-ttickOn)
            if dif<blip and tkey==nkey and ttickOff>=ntickOn and ttickOff<=ntickOff:
                print "Separating: ",this,next," Diff: ",(ttickOff-ntickOn)
                midiOut.append([ttickOn ,ntickOn ,tnote,tkey,tvelocity])
                midiOut.append([ttickOff,ntickOff,nnote,nkey,nvelocity])
                index+=1
                mute=True     
            elif  dif<blip:
                print "Removing blip: ",(ttickOff-ttickOn)
                index+=1
                mute=True     
                continue
            else:
                midiOut.append(this)       
            # iterate the loop
            index+=1
        if index==endAt:
            midiOut.append(next)
        if not mute:
            return midiOut
        midi=midiOut

[This AGPL code is on Github]

Then comes some real fun. If you know the original piece, you might have noticed that the introduction is not original. I added that in the midi editing software Aria Maestosa. It does not need to be done this way; we do not even need to use midi files. A lot of the music I have created in Sonic Field is just coded directly in Python. However, from midi is how it was done here.

Once we have a clean set of notes they need to be converted into sounds. That is done with 'voicing'. I will talk a little about that to set the scene then we can get back into more Java code oriented discussion. After all, this is the Java advent calendar!

Voicing is exactly the sort of activity which brings Python to the fore. Java is a wordy language which has a large degree of strictness. It favours well constructed, stable structures. Python relies on its clean syntax rules and layout and the principle of least astonishment. For me, this Pythonic approach really helps with the very human process of making a sound:


def chA():
    global midi,index
    print "##### Channel A #####"
    index+=1
    midi=shorterThanMidi(midis[index],beat,512)
    midi=dampVelocity(midi,80,0.75)
    doMidi(voice=leadDiapason,vCorrect=1.0)
    postProcess()
    
    midi=longAsMidi(midis[index],beat,512)
    midi=legatoMidi(midi,beat,128)
    midi=dampVelocity(midi,68,0.75)
    midi=dampVelocity(midi,80,0.75)
    doMidi(voice=orchestralOboe,vCorrect=0.35,flatEnv=True,pan=0.2)
    postProcessTremolate(rate=3.5)
    doMidi(voice=orchestralOboe,vCorrect=0.35,flatEnv=True,pan=0.8)
    postProcessTremolate(rate=4.5)

Above is a 'voice'. Contrary to what one might think, a synthesised sound does not often consist of just one sound source. It consists of many. A piece of music might have many 'voices' and each voice will be a composite of several sounds. To create just the one voice above I have split the notes into long notes and short notes. Then the actual notes are created by a call to doMidi. This takes advantage of Python's 'named arguments with default values' feature. Here is the signature for doMidi:


def doMidi(voice,vCorrect,pitchShift=1.0,qFactor=1.0,subBass=False,flatEnv=False,pure=False,pan=-1,rawBass=False,pitchAdd=0.0,decay=False,bend=True):

The most complex (unsurprisingly) voice to create is that
of a human singing. I have been working on this for
a long time and there is a long way to go; however, its
is a spectrogram of a piece of music which does
a passable job of sounding like someone singing.

The first argument is actually a reference to a function which will create the basic tone. The rest of the arguments describe how that tone will be manipulated in the note formation. Whilst an approach like this can be mimicked using a builder pattern in Java; this latter language does not lend it self to the 'playing around' nature of Python (at least for me).

For example, I could just run the script and add flatEvn=True to the arguments and run it again and compare the two sounds. It is an intuitive way of working.

Anyhow, once each voice has been composited from many tones and tweaked into the shape and texture we want, they turn up as a huge list of lists of notes which are all mixed together and written out to disk as a flat file format which is basically just a dump of the underlying double data. At this point it sounds terrible! Making the notes is often only half the story.

Voice Synthesis by Sonic Field

played specifically for this post.

You see, real sounds happen in a space. Our Choral is expected to be performed in a church. Notes played without a space around them sound completely artificial and lack any interest. To solve this we use impulse response reverberation. The mathematics behind this is rather complex and so I will not go into it in detail. However in the next section I will start to look at this as a perfect example of why Java is not only necessary but ideal as the back end to Python/Jython.

You seem to like Python Alex - Why Bother With Java?

My post might seem a bit like a Python sales job so far. What has been happening is simply a justification of using Python when Java is so good as a language (especially when written in a great IDE like Eclipse for Java). Let us look at something Python would be very bad indeed at. Here is the code for performing the Fast Fourier Transform, which is a the heart of putting sounds into a space.


package com.nerdscentral.audio.pitch.algorithm;

public class CacheableFFT
{

    private final int      n, m;

    // Lookup tables. Only need to recompute when size of FFT changes.
    private final double[] cos;
    private final double[] sin;
    private final boolean  forward;

    public boolean isForward()
    {
        return forward;
    }

    public int size()
    {
        return n;
    }

    public CacheableFFT(int n1, boolean isForward)
    {
        this.forward = isForward;
        this.n = n1;
        this.m = (int) (Math.log(n1) / Math.log(2));

        // Make sure n is a power of 2
        if (n1 != (1 << m)) throw new RuntimeException(Messages.getString("CacheableFFT.0")); //$NON-NLS-1$

        cos = new double[n1 / 2];
        sin = new double[n1 / 2];
        double dir = isForward ? -2 * Math.PI : 2 * Math.PI;

        for (int i = 0; i < n1 / 2; i++)
        {
            cos[i] = Math.cos(dir * i / n1);
            sin[i] = Math.sin(dir * i / n1);
        }

    }

    public void fft(double[] x, double[] y)
    {
        int i, j, k, n1, n2, a;
        double c, s, t1, t2;

        // Bit-reverse
        j = 0;
        n2 = n / 2;
        for (i = 1; i < n - 1; i++)
        {
            n1 = n2;
            while (j >= n1)
            {
                j = j - n1;
                n1 = n1 / 2;
            }
            j = j + n1;

            if (i < j)
            {
                t1 = x[i];
                x[i] = x[j];
                x[j] = t1;
                t1 = y[i];
                y[i] = y[j];
                y[j] = t1;
            }
        }

        // FFT
        n1 = 0;
        n2 = 1;

        for (i = 0; i < m; i++)
        {
            n1 = n2;
            n2 = n2 + n2;
            a = 0;

            for (j = 0; j < n1; j++)
            {
                c = cos[a];
                s = sin[a];
                a += 1 << (m - i - 1);

                for (k = j; k < n; k = k + n2)
                {
                    t1 = c * x[k + n1] - s * y[k + n1];
                    t2 = s * x[k + n1] + c * y[k + n1];
                    x[k + n1] = x[k] - t1;
                    y[k + n1] = y[k] - t2;
                    x[k] = x[k] + t1;
                    y[k] = y[k] + t2;
                }
            }
        }
    }
}

[This AGPL code is on Github]

It would be complete lunacy to implement this methematics in JPython (dynamic late binding would give unusably bad performance). Java does a great job of running it quickly and efficiently. In Java this runs just about as fast as it could in any language plus the clean, simple object structure of Java means that using the 'caching' system as straight forward. The caching comes from the fact that the cos and sin multipliers of the FFT can be re-used when the transform is the same length. Now, in the creation of reverberation effects (those effects which put sound into a space) FFT lengths are the same over and over again due to windowing. So the speed and object oriented power of Java have both fed into creating a clean, high performance implementation.

But we can go further and make the FFT parallelised:


def reverbInner(signal,convol,grainLength):
    def rii():
        mag=sf.Magnitude(+signal)
        if mag>0:
            signal_=sf.Concatenate(signal,sf.Silence(grainLength))
            signal_=sf.FrequencyDomain(signal_)
            signal_=sf.CrossMultiply(convol,signal_)
            signal_=sf.TimeDomain(signal_)
            newMag=sf.Magnitude(+signal_)
            if newMag>0:
                signal_=sf.NumericVolume(signal_,mag/newMag)        
                # tail out clicks due to amplitude at end of signal 
                return sf.Realise(signal_)
            else:
                return sf.Silence(sf.Length(signal_))
        else:
            -convol
            return signal
    return sf_do(rii)
            
def reverberate(signal,convol):
    def revi():
        grainLength = sf.Length(+convol)
        convol_=sf.FrequencyDomain(sf.Concatenate(convol,sf.Silence(grainLength)))
        signal_=sf.Concatenate(signal,sf.Silence(grainLength))
        out=[]
        for grain in sf.Granulate(signal_,grainLength):
            (signal_i,at)=grain
            out.append((reverbInner(signal_i,+convol_,grainLength),at))
        -convol_
        return sf.Clean(sf.FixSize(sf.MixAt(out)))
    return sf_do(revi)

Here we have the Python which performs the FFT to produce impulse response reverberation (convolution reverb is another name for this approach). The second function breaks the sound into grains. Each grain is then processes individually and they all have the same length. This performs that windowing effect I talked about earlier (I use a triangular window which is not ideal but works well enough due to the long window size). If the grains are long enough, the impact of lots of little FFT calculation basically the same as the effect of one huge one. However, FFT is a nLog(n) process, so lots of little calculations is a lot faster than one big one. In effect, windowing make FFT become a linear scaling calculation.

Note that the granulation process is performed in a future. We define a closure called revi and pass it to sf_do() which is executed it at some point in the future base on demand and the number of threads available. Next we can look at the code which performs the FFT on each grain - rii. That again is performed in a future. In other words, the individual windowed FFT calculations are all performed in futures. The expression of a parallel windowed FFT engine in C or FORTRAN ends up very complex and rather intractable. I have not personally come across one which is integrated into the generalised, thread pooled, future based schedular. Nevertheless, the combination of Jython and Java makes such a thing very easy to create.

How are the two meshed?

Now that I hope I have put a good argument for hybrid programming between a great dynamic language (in this case Python) and a powerful mid level static language (in this case Java) it is time to look at how the two are fused together. There are many ways of doing this but Sonic Field picks a very distinct approach. It does not offer a general interface between the two where lots of intermediate code is generated and each method in Java is exposed separately into Python; rather it uses a uniform single interface with virtual dispatch.

Sonic Field defines a very (aggressively) simple calling convention from Python into Java which initially might look like a major pain in the behind but works out to create a very flexible and powerful approach.

Sonic Field defines 'operators' which all implement the following interface:


/* For Copyright and License see LICENSE.txt and COPYING.txt in the root directory */
package com.nerdscentral.sython;

import java.io.Serializable;

/**
 * @author AlexTu
 * 
 */
public interface SFPL_Operator extends Serializable
{

    /**
     * <b>Gives the key word which the parser will use for this operator</b>
     * 
     * @return the key word
     */
    public String Word();

    /**
     * <b>Operate</b> What ever this operator does when SFPL is running is done by this method. The execution loop all this
     * method with the current execution context and the passed forward operand.
     * 
     * @param input
     *            the operand passed into this operator
     * @param context
     *            the current execution context
     * @return the operand passed forward from this operator
     * @throws SFPL_RuntimeException
     */
    public Object Interpret(Object input, SFPL_Context context) throws SFPL_RuntimeException;
}

[This AGPL code is on Github]

The word() method returns the name of the operator as it will be expressed in Python. The Interpret() method processes arguments passed to it from Python. As Sonic Field comes up it creates a Jython interpreter and then adds the operators to it. The mechanism for doing this is a little involved so rather than go into detail here, I will simply give links to the code on github:

The result is that every operator is exposed in Python as sf.xxx where xxx is the return from the word() method. With clever operator overloading and other syntactical tricks in Python I am sure that the approach could be refined. Right now, there are a lot of sf.xxx calls in Sonic Field Python ( I call it Synthon ) but I have not gotten around to improving on this simple and effective approach.

You might have noticed that everything passed into Java from Python is just 'object'. This seems a bit crude at first take. However, as we touched on in the section of futures in the previous post, it offers many advantages because the translation from Jython to Java is orchestrated via the Caster object and a layer of Python which transparently perform many useful translations. For example, the code automatically translates multiple arguments in Jython to a list of objects in Java:


    def run(self,word,input,args):
        if len(args)!=0:
            args=list(args)
            args.insert(0,input)
            input=args
        trace=''.join(traceback.format_stack())
        SFSignal.setPythonStack(trace)
        ret=self.processors.get(word).Interpret(input,self.context)
        return ret

Here we can see how the arguments are processed into a list (which is Jython is implemented as an ArrayList) if there are more than one but are passed as a single object is there is only one. We can also see how the Python stack trace is passed into a thread local in the Java SFSignal object. Should an SFSignal not be freed or be double collected, this Python stack is displayed to help debug the program.

Is this interface approach a generally good idea for Jython/Java Communication?

Definitely not! It works here because of the nature of the Sonic Field audio progressing architecture. We have processors which can be chained. Each processor has a simple input and output. The semantic content passed between Python and Java is quite limited. In more general purpose programming, this simple architecture, rather than being flexible and powerful, would be highly restrictive. In this case, the normal Jython interface with Java would be much more effective. Again, we can see a great example of this simplicity in the previous post when talking about threading (where Python access Java Future objects). Another example is the direct interaction of Python with SFData objects in this post on modelling oscillators in Python.


from com.nerdscentral.audio import SFData
...
            data=SFData.build(len)
            for x in range(0,length):
                s,t,xs=osc.next()
                data.setSample(x,s)

Which violated the programming model of Sonic Field by creating audio samples directly from Jython, but at the same time illustrates the power of Jython! It also created one of the most unusual soundscapes I have so far achieved with the technology:

Engines of war, sound modelling

from oscillators in Python.

Wrapping It Up

Well, that is all folks. I could ramble on for ever, but I think I have answered most if not all of the questions I set out in the first post. The key ones that really interest me are about creativity and hybrid programming. Naturally, I am obsessed with performance as I am by profession an optimisation consultant, but moving away from my day job, can Jython and Java be a creating environment and do they offer more creativity than pure Java?

Transition State Analysis using
hybrid programming

Too many years ago I worked on a similar hybrid approach in scientific computing. The GRACE software which I helped develop as part of the team at Bath was able to break new ground because it was easier to explore ideas in the hybrid approach than writing raw FORTRAN constantly. I cannot present in deterministic, reductionist language a consistent argument for why this applied then to science or now to music; nevertheless, experience from myself and others has show this to be a strong argument.

Whether you agree or disagree; irrespective of if you like the music or detest it; I wish you a very merry Christmas indeed.

This post is part of the Java Advent Calendar and is licensed under the Creative Commons 3.0 Attribution license. If you like it, please spread the word by sharing, tweeting, FB, G+ and so on!

[sysadvent] Day 24 - 12 days of SecDevOps

Christopher Webber — Wed, 24 Dec 2014 00:00:00 +0000

Written by: Jen Andre (@fun_cuddles)
Edited by: Ben Cotton (@funnelfiasco)

Ah, the holidays. The time of year when we want to be throwing back the eggnogs, chilling in front our fake fireplaces, maybe catching a funny Christmas day movie… but oh no we can’t, because guess what, a certain entertainment company was held hostage by a security breach the likes of which corporate America has never seen before… and no more movie for you.

It’s an interesting time to be a security defender. The recent Sony breach has just put a period on the worst-of-the-worst scenarios that us tinfoil-hat, paranoid security people have been ranting about all along: one bad breach could be business shattering.

But let’s step back, and look at the theme of this blog: the 12 days of SecDevOps. Besides being a ridiculous title that I’m 90% sure my ops director chose specifically as a troll for me (thanks, Pete), it underlines an important concept. Whether `security` is in your job title or not, operations is increasingly becoming the front-line for implementing security defenses.

Given that reality, and the fact that security breaches are NOT going away, and that most of us don’t have yacht-sized security budgets, I thought it would be interesting to come up with 12 practical, high-impact things that small organizations could be doing to shore up their security posture.

Day 1: Fear and Loathing and Risk Assessment and Hipsters

Risk assessment. It’s not just some big words auditors love to use. It’s simply weighing the probability of bad things happening against the cost to mitigate the risk of that bad thing happening. And using that to make good security decisions as you make day-to-day architecture and ops choices:

risk = (threat) x (probability) x (business impact)*

*whoever told you there would be no math lied to you

You may not be aware of it, but as an ops person you are likely doing risk assessment already, except more likely around things like uptime and reliability. Consider this scenario:

John, the web guy, proposes replacing PostgreSQL with SomeNewHipsterDB.
You ask yourself, ‘huh, what’s the chances that I’m going to get paged at 3am because writes stop happening and my web site starts screaming in pain?’ You are probably not having warm-fuzzy feelings about this plan.
Your development and ops team evaluates the benefits to the engineering team and business for switching to SomeNewHipsterDB and weighs it against the probability that you are going to get woken up all of the time, and the impact it will have on your sunny disposition and decide that yeah… maybe not gonna do it.
Or, you do, except you mitigate this risk by saying ‘John, you will be forever paged for all SomeNewHipsterDB issues. Done.’

Cool. Now do this for security. Every time you are making architecture choices, or changing configuration of your infrastructure, or considering some new third-party service SaaS you’ll be sending data to, you should be asking yourself: what’s the impact if that service or system gets hacked? How will you mitigate the risks?

This doesn’t have to be a formal or fancy report. It can be a running text file or spreadsheet with all of the possible points of failure. Get everyone involved with thinking of ways pieces of the infrastructure or organization can be hacked, and ways you are protected against those worst-case scenarios. It can be like ‘ANYONE WHO OWNS OUR CHEF SERVER COULD DESTROY EVERYTHING [but we have uber-monitoring and Jane over there reviews audit logs daily]’. Start with the scenario: what if… ? and have conversations with engineers and business owners defend why what we’re doing is good enough. Make security a fundamentally collaborative process.

Day 2: Shared Secrets: Figure it Out Now

There’s 3 things in life that are inevitable: death, taxes… and the fact that a sales guy left to his own devices will always put all of his passwords in a plain text file (or if fancy, an Excel spreadsheet).

The lesson is this: password management isn’t something that just the technical team decides and manages for itself. We should be advocating organization-wide education on managing credentials, because guess what? Access to Salesforce, Gmail, and all of these SaaS services with sensitive business data are being used by people who are not engineers.

Solution? As part of every employee’s onboarding process, install password management on an employee’s workstation, and show them how to use it (e.g. 1Password or LastPass, or whatever your tool of choice is). Start doing this from the outset, as it’s best to figure this out on Day 1 rather than 200 employees in.

Day 3: Shared Secrets for Infrastructure, Too

When it comes to infrastructure secrets, there are extra concerns because in most cases, systems needs to be able to access these secrets in a non-interactive, automated way (e.g. I need to be able to spin up an app server that knows how to authenticate to my database).

If all of your infra passwords start unencrypted somewhere in a git repo, You Are Going To Have A Bad Time. Noah has a good article on various options for managing shared secrets in your infrastructure.

Day 4: Config Management On All Of The Things (So You Aren’t Sweating from Shell Shocks)

This should be obvious to everyone who drinks from the DevOps Koolaid, but CM has done beautiful things for patch management. It may be tempting to deploy a one-off box used for dev manually without config management installed, but guess what? In the case of Browser Stack, that turned out to be a massive achilles heel.

Making the process easy for devs to get access to the infrastructure they need (while giving you the ability to manage systems) is key. Do this right away.

Day 5: Secure your Development Environments (Because No One Else Will)

If left to their own devices, development environments tend to veer to chaotic. This isn’t just because developers are lazy (and as a developer, I mean this in the nicest possible way) but because of the nature of the prototyping and testing process.

From a security perspective, this all means bad juju (see Browser Stack example above). I can assure you that if you start building your prototype or dev infrastructure exposed to the public internet, deploying it without even the basic config management, it will stay that way forever.

So: if you are using AWS, start with an Amazon VPC with strict perimeter security, and require VPN access for any development infrastructure. Get some config management on everything, even if it’s just for system patches.

Put some bounds around the chaos early on, and this will make it easy to mature the security controls as the product and organization mature.

Day 6: 2-Factor all of the things (well, the important things)

Require 2-factor wherever you can. Google Apps has made enforcing this super easy, and technologies like DuoSecurity and YubiKey make adding 2-factor to your critical infrastructure (e.g., your VPN accounts) far, far less annoying than it used to be.

Day 7: Encrypt your Emails (and other communications)

Encrypt your emails. It’s annoying to set up, but guess what? Hackers just love to post juicy stuff on pastebin. Again, from Day 1, help every single employee configure PGP or SMIME encryption as part of the onboarding process. Once installed, it’s relatively painless to use (as long as you don’t mind archaic mail clients from 1999).

This is especially important to drill into executives because they tend to have more sensitive emails (e.g. their private boardroom chatter), and are particularly susceptible to phishing style-attacks. With the recent Sony email leaks, you now have some leverage. You can throw the ‘Angelina Jolie’ emails in front of them and ask: how much do you think business and reputations would suffer were their entire email archives publically disclosed via a breach?

For many of us, chat is as crucial as email in terms of the type of reputation-critical information we put there. It may not be reasonable to switch to a self-hosted chat solution, but in that case, ensure you are picking a service that helps YOU mitigate your risk. E.g., do you need all of the history? Do you need private history for user chats?

Day 8: Security Monitoring: Start Small, Plan Big

Put the infrastructure in place to collect as much security data as possible, then start slowly making potential security issues visible by adding reports and alerts that deal with threat scenarios you are most worried about.

Start small. Remember that risk assessment list you made? Identify what you are most afraid of (um, that PHP CMS that has hundreds of vulnerabilities reported per year? Your VPN server?) and tackle monitoring for those items first.

Instrumenting your infrastructure from day 1 for security monitoring (even if it’s just collecting all of the system and application logs) puts you in a good position later on to start sophisticated reporting and intrusion detection on that data.

Day 9: Code/Design Reviews

Although there have been a lot of advancement in static and dynamic source code analysis tools (which you can integrate right into your CI process), a good-old fashion code review by a human being goes a long way. If you’re using GitHub, just make it part of the development workflow and testing pipeline. Whenever changes are made to authentication or authorization, have someone look for automated tests that deal with those cases.

Day 10: Test Your Users

Phish yourself regularly. It’s really easy to do, and can be illuminating to the rest of the business which may not be as technical as the operations/engineering side, and not really understand really the impact of opening an attachment in an email or not checking URLs where they are logging into a website. You can use some open source tools, but are also many services now that you can pay to do this for you.

Day 11: Make an Incident Response Plan Now

So, you see something odd in your logs. Like, Bob your DBA ran a Postgres backup on production DB, tar’d it up, and sent it to an FTP server in Singapore. Bob lives in Reston VA, and this is definitely not normal. You start seeing evidence of other weird stuff ‘bob’ is doing that he shouldn’t be.

What now? Do you email Bob and say ‘something weird is happening?’ Do you call the Director of Ops? Do you put a message in a lonely chat room?

Figure out a plan for escalating possible critical security issues. Doesn’t have to be fancy or use specialize ITIL incident response workflow tools. Make a group in PagerDuty. Have an out-of-band channel for communicating details, in case your normal network goes the way of Sony and is totally compromised or just plain is not working. Maybe it’s as simple as an email list that doesn’t use the corporate email accounts, or a conference bridge everyone can hop on.

Day 12: Don’t be the Security ‘A**hole’

You. Yes, you. Don’t be the security a**hole that gets in everyone’s way and loses sight of the real reason for everyone’s existence: to run a business. You can be the security champion without being the blocker. In fact, that’s the only way to be effective. If a user is coming to you and saying ‘this is really really annoying, I don’t want to do it’ - listen to them. Too many security personnel disregard the usability issue of security controls for the sake of security theater, which leads to (unsurprisingly) abandonment, cynicism, and apathy when it comes to real security concerns.

DevOps is really a philosophy: it’s not a job title, or a set of tools, it’s the concept of using modern tools and processes to facilitate collaboration with the engineers who deliver the code and those who must maintain it. Um, that was a lot of words, but the key word is collaboration. It’s no longer acceptable to throw ‘security over the wall’ and expect your users and ops people to just do what you say.

The best security cultures are not prescriptive, they are collaborative. They understand that business needs to get done. They are intellectually honest and admit ‘yeah, we could get hacked’ - but what can we do about this in a way that doesn’t bring everything to a halt? Zane Lackey has a great talk on building a modern security engineering organization that expounds many of these ideas, and more.

[ux] The Elf that Came in From the Cold

Tue, 23 Dec 2014 21:00:00 +0000

UCD City was asleep. The gin joints were shut. Hoods and flatfoots and gumshoes alike were all in their holes. Snow fell like a silent movie, blanketing the usual street noise of passing cars and fighting alley cats. Christmas neon flashed on every second apartment. Behind the lopsided shutters of my one-man agency I poured myself another lonely egg nog.

The doorknob of my front door jiggled. I thought nothing of it—probably kids on holidays up past their bedtime. Suddenly the door burst open, and an elf tumbled in, clutching a mobile phone. His eyes screamed silently as he reached toward me.

“Merry …”

He hit the floor. Only then did I notice the kitchen knife protruding from his kidneys.

I heard footsteps downstairs. Leaping over the elf’s body and flying down the staircase railings, I spied a shadow fleeing the building. I made it to the street, breathless, in time to watch twin tail lights screech around the corner. Another win for the bad guys. The neighbourhood fell silent and lonely again.

Footprints were melting into the muddy slush: size 12 at least, probably boots. I glanced up at the window to my room where a dead elf lay.

I was definitely going to be on the naughty list again this year.

* * *

Most of the dark red wasn't an intentional part of the elf's costume—it was now pooling on my floorboards. I was no stranger to crime, but this was heinous: a dead pixie on my doorstep on Christmas Eve? Hardly a Merry Christmas.

What had he been trying to say? “Merry!” What did that mean? Merry Christmas? How was that a dying man's secret this time of year? And why to me? I'd consulted with Santa's Workshop much earlier in my career—half a lifetime ago. But I’d had nothing to do with the operation since.

There was a buzz from the smartphone in his little hand: a SnapChat message. I swiped the screen before thinking what I was doing, and deserved what I saw.

It was Saint Nick himself, bound, gagged, and seated on a stool. He looked haggard and beaten, and two armed men stood over him. Good grief—who would dare mess with Santa? Was all of Christmas in jeopardy? I cursed as SnapChat erased the photo. Evidence and answers were slipping through my fingers. Damned ephemeral messaging!

I rifled through the elf's pockets, and turned up a Sharpie marker, some half-finished sketches on a bunch of post-it notes, the elf’s company RFID card, and a sealed envelope.

I wrapped the envelope in a plastic bag, and pinned it to the outside window ledge. Then I took a closer look at the rest of the items.

Printed on the RFID card was the elf’s name and position:

Zachary Kringle
Chief of Operations
Workshop Division
Santa’s Workshop Inc.

Not a junior then. I sifted through the post-it notes. The sketches were pretty basic—an early concept for some sort of board game similar to Snakes & Ladders, with additional cards that formed part of the mix. The sketches were annotated—process, iterate, collaborative—obviously some new gift idea he’d been working on. Too bad he’d croaked. It looked like a good game.

I retrieved the sealed envelope from the bag on the windowsill. The gum was now frozen stiff, making it easy to open the envelope without damaging it. There was a photo inside. I recognised the face instantly.

Mary Christmas.

The elf's words made more sense now: “Mary”, not “Merry”. Panic hit me like ice water. Could the Mary I knew really be behind such a brutal crime?

Mary was as curvy as always. Not a young woman, but a plenty real one, like someone had poured her into her clothes and forgot to say when. It had been taken recently, and with a zoom lens: she was getting into a black car while her minders stood around. Behind her, the corner of a warehouse partially hid a dockside crane.

I had my lead.

There were two things I needed to do right away. The first was a simple task. I sat on my desk looking at the elf and called Uncle Xavier, the only straight cop I knew in homicide. Xavier worked old-school, and although we didn’t always see eye to eye, we both knew the way the game was played. I spilled my story but conveniently omitted the photo—I would confront Mary myself when the time was right.

The last few hours of downing egg nog had made my brain as lazy as an old dog. So I shrugged on my hat and trenchcoat, packed my trusty 357 snubnose and hit the icy pavement to clear my head.

* * *

A dead elf, a captive Santa, Mary Christmas and a mystery warehouse: not much of a brief. I remembered a project I'd worked on with an engineer at Path-E-Tech Management. As the engineer said himself:

“We interviewed hundreds of users, and turned all their suggestions into features. As it turns out, every user we talked to was an idiot. And their dumb suggestions ruined our product. In hindsight we should have talked to people outside the building.”

Not the best attitude, but it does say something about user research outside the obvious. And that’s exactly what I planned to do.

A pile of cardboard and old blankets moved as I walked past. "Help an old graphic artist fallen on bad times?" a voice croaked from within.

"You see anyone come past here in the last half hour?" I asked.

A hooligan in a turtleneck poked his head out. "Just a short guy—real shorty he was, and a gorilla. Jumped into a car and took off that way.”

I flipped him a few coins. He grinned in appreciation.

"Thanks man, Merry Christmas", he said.

"Yeah. Merry Christmas."

No elf I'd ever met had size twelve boots and looked like a gorilla. Someone outside the Workshop had stabbed him in the back—someone who didn't want Christmas to run the usual clockwork. But who? And why was someone sending a SnapChat of a nicked Saint Nick to the elf? Did they know he was dead? What would happen if Santa wasn't released in time? These were the questions that needed answering.

I grunted. Two billion unhappy kids is what would happen. The stakes couldn’t be higher.

I talked to anyone who would listen: lovers buying last-minute gifts, retailers and late-night carol singers. And when my first coffee-stained notebook was full, I turned south—I needed to triangulate my findings with a different technique altogether.

* * *

I stopped at a local toy store and made my way towards the back. Two figures behind frosted glass moved like a puppet show, and I stepped behind a giant cuddly Pink Panther to eavesdrop. I waited patiently, catching only snippets of information, but enough to make out something about global shipments. I was on the right path.

The silhouettes left via a back door, and I waited until I heard the door slam. I stepped from my hiding place and into the back room. I fumbled with the lock on the drawer of the desk where they’d been seated. The lock was no match for my expertise with a paper clip, and once it was open I found the jackpot: the store manager’s journal.

My greedy eyes ran over the entries from the past few weeks. I felt dirty—it wasn't my usual method of conducting a diary study, but it sure delivered the goods. I tried to ignore the personal and stick to business: names, meeting times, summary notes … I struck gold with a purchase order form for a toy shipment. It listed a warehouse address down on the waterfront. I took a quick snap using my camera phone.

There was a creak behind me and a torch blinded me.

"Grab some air, pal" a gruff voice barked from the shadows. I raised my hands.

I turned to look at him. I didn't like his face. When he said his name was Skinny I didn't like his name either. But the secret to being a great detective is to listen more and talk less—I kept my trap shut.

Two goons loomed behind him: dull eyes, flat heads, black hats, and no brains at all. I looked uneasily at their hands. They were big—as big as plates of pork ribs, and twice as greasy. I knew exactly what was coming next.

With a nod from Mr Skinny, the other two moved in and started to work me over. I braced myself for the blows, and winced as they rained down on me. Then my world went dark.

* * *

The thing about getting licked by goons is that if you're lucky enough to wake up, you'll always be missing grey cells. I was a veteran now. I hoped what I’d lost in intelligence was more than made up for by experience. At least that’s what I told myself when returning the favour to some gunsel. When you’re on a case, you need to defend your thinking, and can’t pull punches.

I dragged my sorry ass to a joint where I knew I'd find a few fellow gumshoes. Donna was there, absent-mindedly sorting cards into groups. Patrick was there too, cradling a rye as he always did. He had a reputation for drinking anyone under the table, especially after a “UX fail”. He'd made a game of it, they said. But this time I didn’t feel like playing.

I accepted a glass of hooch but waved away their questions. All I wanted was quiet companionship. They got the gist and went back to cards and drinking.

My body was bruised, but my thinking was as sharp as ever. If Skinny and his goons wanted me out of the way they could have left me alone to just drink myself to death. Why didn’t they finish me? They obviously didn’t know why I was there.

After resuming my balance I returned to my office and stared at the post-its stuck between strips of peeling pink wallpaper.

Outside the snow had turned to rain. Someone somewhere was blowing a horn as mournful as a turkey in December. It went nicely with my mood.

The post-its were grouped with all the nuggets of info I’d collected so far. I had an affinity for mapping out thoughts like this since working with the feds. I added my new findings: the waterfront warehouse address, and the dates from the journal.

But there wasn't yet enough to work with—I couldn't draw any threads, and I reluctantly realised what I had to do. I preferred to work alone but there was no way this could be solved without getting another set of eyes on the case. And there was only one person who could help, even if she was too close to the furnace.

I picked up the phone and dialled her number for the first time in years.

* * *

Mary Christmas crossed her legs. She knew they were good.

“I’m just not that kind of girl," she snarled. “I played your game once, but now I do things differently. Sorry—you’re on your own.”

I hinted that I knew she was in up to her neck with the impolite crowd.

"I don't like to see cheap hoods messing with a sweet kid like you, Princess."

She went bright red, and I should have seen the slap coming. I nursed my jaw and just let her fume.

"Careful, Mac", she said. "Don't be a dick."

I let the obvious comment slide.

"That elf died on my doormat, and those goons don't mess about either." I reminded her. "I'm just saying ... don't go there."

She knocked back her shot and glared at me. "No" she said, "don't you go there. If I were you, I'd take the easy option and go back to your two-bit sleazy dive. Forget any of this ever happened!"

They were the same words as the Dear John letter she'd written me all those years ago. I was obviously still learning.

We looked at each other sadly over the table. "Lucky I'm not you then," I said, and this time I was the one to walk out.

* * *

Something woke me to the world of cigarette smoke and last night's leftover fruit mince pies. It was the office phone.

"Mac here," I mumbled.

"Sit this dance out, Mac," snapped a voice. Not Mr Skinny—he was older, more sure of himself. A Danish accent?

I waited for him to continue, but the line went dead.

Perhaps it was someone working with Skinny? Maybe the higher-ups were starting to show concern.

The thought gave me a mild sense of satisfaction. But the other side definitely had all the cards—and all the eyes. Time to get up and work. I pulled the blinds shut and prepared myself a Mexican Hot Chocolate. Taking a long swig, I turned back to the notes on my wall.

It was going to be a long night.

* * *

"You're not to work this case any more, Mac. You're fired!"

Uncle Xavier was none too happy that I’d been doing my own investigation, especially as I wasn’t off the suspects list.

"Maybe you've fired me, but I haven't fired me!"

The fuzz were on my case, the goons were hitting on me, and Mary was playing hard to get. I had to crack this before someone caught up, or Christmas would be cancelled and I'd be making licence plates.

I hadn't started this thing, but it was up to me to finish it.

Luckily, I had a plan.

* * *

Anyone that knows me will say I'm a sucker for punishment. I keep going back, failing again and again. Most people think I'm nuts, but you actually learn a lot from failing. More than winning the first time, at least. I never had the brains to win first time, but if I made my mind up I could stick to something like superglue. And I always came through in the end.

My limpet-style antics were about to pay off again. “Follow the data,” they say. My coffee-stained notebook contained multiple statements from Christmas shoppers stating that new toy store stock always seemed to be available in store on Wednesdays. Tonight was Tuesday.

The old waterfront warehouse that Santa’s Workshop had used before going digital back in the ‘80s was long abandoned. Were they planning to open operations again? That'd explain the elf's involvement. And if Skinny and his goons managed to squeeze Santa for his trade secrets, they’d have a monopoly on Christmas.

I was battling with barbed wire in the fence when a waft of perfume cut through the stink of grease and bilgewater and rotting fish. I sighed and sat pretty. I knew that smell. Mary strutted around the corner like she owned the place. Which I guess she did—the three minders she had with her underlined that fact.

“You’re getting rusty” she said to me, looking at the wire.

“You know me. I like to get stuck into things.”

“Tangled up in a mess, you mean.”

“It’d help if you gave me a hand rather than just standing there.”

The worry in my own eyes was reflected in hers.

“You’re gonna need this.” I handed her the elf’s RFID card.

Peering through a window, we saw a warehouse filled with crates of toys, puddings, holly, tinsel, and Christmas decorations. Either Christmas had been postponed, or these were next year’s goods.

One entire wall was covered with notes and sketches: annotated blueprints, moodboards, experience maps. This was a major project. I recognised some of the sketches from those I’d taken out of the elf’s pockets.

I stopped short when I saw the broad back and shoulders of a red-suited old man, sitting in the corner, bound and helpless. He turned to face us. His face said a lot. Those eyes would’ve even made a bloodhound sad.

* * *

Mary swiped the elf’s ID card, and the massive doors of the warehouse rattled open, revealing four figures. I’d have bet any money the biggest wore size 12 boots. He dwarfed the three others, especially the skinny runt at the front. I nodded in recognition at the two goons and they nodded back. They both packed Chicago heaters. Things were getting crowded.

Mr Skinny reached for the bulge inside his coat. He revealed the top corner of a Samsung Galaxy S6.

“Don’t make me use this.” he drawled.

What was he going to do—SnapChat me to death?

I made a beeline for Santa, and managed to untie him before the goons stitched a neat burst of holes just in front of our toes. Santa didn’t flinch. He shook the ropes free and flew at the gorilla, catching him off guard while he reloaded.

The two large men rolled across the floor while I fired a Hail Mary at Skinny and his goons to distract them. For an old guy, Santa could give plenty.

Their fisticuffs interspersed with gunfire in the cavernous warehouse was deafening. Bullets flew like wasps. Mary's men were solid but I was running low on ammo. I ducked behind a shipping container of Buzz Lightyear figurines, and crouched beside Mary. She pointed across the floor at a row of crates—one crate in particular looked like a single shove would topple it.

“If you’re thinking what I’m thinking, you’re crazy.” I whispered.

“This is nothing compared to ballet school!” she chuckled. She made a dash for it and covered the gap in less than a second—quick enough to reach the opposite side before the goons noticed. I wasn’t going to be left huddling there like a sissy, but they were watching me now, and I’d have to hightail it. I dug deep and sprinted towards Mary, throwing in a Commando roll for old time’s sake. I landed safely beside her, but my hat and coat now had some extra ventilation.

“Right as rain.” I grinned, feeling more alive than ever.

We leant into the crate and felt it start to set off a domino effect, the smaller crates crashing into ever larger ones. It was a giant, slow-motion tsunami. I couldn’t hear it over the chatter of the Tommy guns, but I was surprised the goons couldn’t feel each crash. Too late for them now anyway. Fifty tonnes of toy blocks makes a God almighty noise when it lands. It also turns goons into pancakes.

The dust settled as Xavier's men arrived, sirens blaring, late to the party as always. At least we'd saved Christmas. Santa's global delivery project would meet its annual deadline once more.

“You’re one helluva mental model, Mary.”

"You old dog," she grinned.

I’d certainly learnt some new tricks. Not sure I’d be up for putting them to use any time soon, though. This caper had almost done me, and I longed for the slow life.

A sprig of mistletoe and holly from the firefight dangled from a girder above us.

"Tell me you had nothing to do with this," I pleaded with Mary. "Even if you lie, I'll believe you."

She smiled and we kissed, but I knew it was for the last time.

The End.

[24ways] Taglines and Truisms

feeds@allinthehead.com (Andrew Clarke) — Tue, 23 Dec 2014 12:00:00 +0000

Andrew Clarke poses the question, that if we’re all telling prospective clients that we’re crafting and designing delightful, beautiful and remarkable digital experiences, what marks any of us out?

[perl] CLDR TL;DR

Nova Patch — Tue, 23 Dec 2014 05:00:00 +0000

2014 has been an exciting year for CLDR development on the CPAN. But first, what is the CLDR? The Unicode Common Locale Data Repository is a standardized repository of locale data along with a specification for its use and implementation. The simplest use case is easy access to translations for use in user interfaces, including month and day names, country and language names, and units of measure such as hours, bytes, meters, and even furlongs. More complex uses include localized ranges of dates using the local calendaring and numbering systems.

The CLDR specification, however, is increasingly complex and the amount of data is increasingly large. This makes sense because natural languages are complex and each release supports additional minority locales. Fortunately, the CPAN has had more CLDR-based development this year than ever before. This means you don’t have to worry about reading complex specifications or manually parsing large XML data structures.

What are locales?

There are two parts to a locale: an identifier and data. The identifier is used to specify user preferences, generally based on languages and regions. The simplest locale is a language code alone, such as es for Spanish and zh for Chinese. Including the user’s country in the locale can provide additional valuable information. For example there are many differences in displaying dates and even numbers between European Spanish (es-ES) and Mexican Spanish (es-MX). Much additional information can be explicitly included in the locale, but most of the time it’s implicitly derived from the language and region. For example, many locales default to the Gregorian calendar while some to the Buddhist calendar or others; zh-CN defaults to Simplified Han script while zh-TW defaults to Traditional Han. However, if you want to get really explicit, you could say tlh-Hira-AQ-u-ca-julian-nu-roman for Klingon in the Hiragana script as used in Antarctica with the Julian calendar and Roman numerals.

Whenever possible, include the user’s language and country when constructing a locale identifier in order to provide the most localized experience.

Now let’s take a tour of some simple solutions to common localization problems using CPAN modules.

CLDR::Number

CLDR::Number is a new module that become stable early this year and provides localized formatting of numbers, prices, and even percents and ranges of numbers. Full disclosure: I wrote this module and it powers Shutterstock in 20 languages, 150+ countries, and many currencies.

Here’s an example of formatting numbers:

`1: 2: 3: 4: 5: 6:`	`use CLDR::Number; my $cldr = CLDR::Number->new(locale => $locale); my $decf = $cldr->decimal_formatter; say "$locale: ", $decf->format(123456.7);`

Now let’s see the results in European Spanish, Mexican Spanish, and Bengali:

es-ES: 123 456,7
es-MX: 123,456.7
bn-IN: ১,২৩,৪৫৬.৭

This demonstrates that both the language and the country can significantly change the results of basic number formatting. Now let’s see this applied to prices in different currencies.

`1: 2: 3:`	`my $curf = $cldr->currency_formatter(currency_code => $currency); say "$locale / $currency: ", $curf->format(9.99);`

Here are the results in American English and Canadian English for both US Dollars and Canadian Dollars.

en-US / USD: $9.99
en-CA / USD: US$9.99
en-CA / CAD: $9.99
en-US / CAD: CA$9.99

This demonstrates that localized formatting is important even when your only supported language is English. When it comes to currency formatting, the language, country, and currency each can significantly change the results.

Locale::CLDR

Locale::CLDR is another new module released earlier this year by John Imrie, with the goal of providing access to all of the CLDR via locale objects—an impressive task!

Different locales use different punctuation and this is commonly ignored even in applications with translations in many languages. Fortunately, Locale::CLDR makes this aspect of localization easy.

Here is a simple solution to formatting a list of strings for the user:

`1: 2: 3: 4: 5: 6:`	`use Locale::CLDR; my $cldr = Locale::CLDR->new($locale); my @gifts = qw( foo bar baz ); say "$locale: ", $cldr->list(map { $cldr->quote($_) } @gifts);`

The quote method is used to quote each element and the list method is used to format the entire list. Let’s take a look at the results in Portuguese, French, and Urdu.

pt: “foo”, “bar” e “baz”
fr: «foo», «bar» et «baz»
ur: ”foo“، ”bar“، اور ”baz“

Note that for support of all locales, you currently have to use the Locale::CLDR v0.25.x release on CPAN instead of v0.26.x because the latter is in the process of being broken into locale bundles and that work is ongoing.

New year, new development

We’ve had a great year for Perl localization and I hope 2015 will be even better. Once the most important CLDR::Number::TODO tasks are completed, DateTime::Locale will receive some much needed love. The top gift on my wishlist is a Perl wrapper for ICU4C (International Components for Unicode), which is a mature project providing full CLDR support. I’m confident that if I continue to fill my uncle’s boots with coleslaw on Yaksmas Eve, the Gilded Yak may finally deliver.

[java] The Java Ecosystem - My top 5 highlights of 2014

Martijn Verburg — Tue, 23 Dec 2014 00:15:00 +0000

1. February the 1st - RedMonk Analyst firm declares that Java is more popular & diverse than ever!

The Java Ecosystem started off with a hiss and a roar in 2014 with the annual meeting of the Free Java room at FOSDEM. As well as the many fine deep technical talks on OpenJDK and related topics there was also a surprise presentation on the industry from Steve O'Grady (RedMonk Analyst). Steve gave a data lead insight into where Java ranked in terms of both popularity and scope at the start of 2014. The analysis on where Java is as a language is repeated on RedMonk's Blog. The fact it remains a top two language didn't surprise anyone, but it was the other angle that really surprised even those of us heavily involved in the ecosystem. Steve's talk clearly showed that Java is aggressively diverse, appearing in industries such as social media, messaging, gaming, mobile, virtualisation, build systems and many more, not just Enterprise apps that people most commonly think about. Steve also showed that Java is being used heavily in new projects (across all of those industry sectors) which certainly killed the myth of Java being a legacy enterprise platform.

2. March the 18th - Java 8 arrives

The arrival of Java 8 ushered in a new Functional/OO hybrid direction for the language giving it a new lease of life. The adoption rates have been incredible (See Typesafe's full report on this) it was clearly the release that Java developers were waiting for.

Some extra thoughts around the highlights of this release:

Lambdas (JSR 335) - There has been so much written about this topic already with a ton of fantastic books and tutorials to boot. For me the clear benefit to most Java developers was that they're finally able to express the correct intent of behaviour with collections without all of the unnecessary boiler plate that imperative/OO constructs forced upon them. It boils down to the old adage of That there are only two problems in computer science, cache invalidation, naming things, and off-by-one errors. The new streams API for collection in conjunction with Lambdas certainly helps with the last two!
Project Nashorn (JSR 223, JEP 174) - The JavaScript runtime which allows developers to embed JavaScript code within their Java applications. Although I personally won't be using this anytime soon, it was yet another boost to the JVM in terms of first class support for dynamically typed languages. I'm looking forward to this trend continuing!
Date and Time API (JSR 310, JEP 150) - This is sort of bread and butter API that a blue collar language like Java just needs to get right, and this time (take 3) they did! It's been great to finally be able to work with timezones correctly and it also set a new precedence of Immutable First as a conscious design decision for new APIs in Java.

3. ~July - ARM 64 port (AArch64)

RedHat have lead the effort to get the ARMv8 64-bit architecture supported in Java. This is clearly an important step to keep Java truly "Run anywhere" and alongside SAP's PowerPC/AIX port represents two major ports that are primarily maintained by non-Oracle participants in OpenJDK. If you'd like to help get involved see the project page for more details.

Java still has a way to go before becoming a major player in the embedded space, but the signs in 2014 were encouraging with Java SE Embedded featuring regularly on the Raspberry Pi and Java ME Embedded getting a much needed feature parity boost with Java SE APIs.

4. Sept/Oct - A Resurgence in the JCP & it's 15th Anniversary

The Java Community Process (JCP) is the standards body that defines what goes into Java SE, Java EE and the Java ME. It re-invented itself as a much more open community based organisation in 2013 and continued that good work in 2014, reversing the dropping membership trend. Most importantly - it now truly represents the incredible diversity that the Java ecosystem has. You can see the make up of the existing Executive Committee - you can see that institutions like Java User Groups sitting alongside industry and end user heavyweights such as IBM, Twitter and Goldman Sachs.

Community Collaboration at an all time high & Microsoft joins OpenJDK.

The number of new joiners to OpenJDK (see Mani's excellent post on this) was higher than ever. OpenJDK now represents a huge melting pot of major tech companies such as Red Hat, IBM, Oracle, Twitter and of course the shock entry this year of Microsoft.

The Adopt a JSR and Adopt OpenJDK programmes continue to bring more day to day developers involved in guiding the future of various APIs with regular workshops now being organised globally around the world to test new APis and ideas out early and feed that back in OpenJDK and the Java EE specifications in particular.

Community conferences & the number of Java User Groups continue rise in numbers, JavaOne in particular having it's strongest year in recent memory. It was also heartening to see a large number of community efforts helping kids learn to code with after school and weekend programmes such as Devoxx for Kids.

What for 2015?

I'll expect 2015 to be a little bit quieter in terms of changes for the core language or exciting new features to Java EE or Java ME as their next major releases aren't due to 2016. On the community etc front I expect to see Java developers having to firmly embrace web/UI technologies such as AngularJS, More systems/Devops toolchains such as Docker, AWS, Puppet etc and of course migrate to Java 8 and all of the functional goodness it now brings! The community I'm sure will continue to thrive and the looming spectre of IoT will start to come into the mainstream as well. Java developers will likely have to wait until Java 9 to get a truly first class platform for embedded, but early adopters will want to begin taking a look at early builds throughout 2015. Java/JVM applications now tend to be complex, with many moving parts and distributed deployments. It can often take poor frustrated developers weeks to fix issues in production. To combat this there are a new wave of interesting analysis tools dealing with Java/JVM based applications and deployments. Oracle's Mission Control is a powerful tool that can give lots of interesting insights into the JVM and other tools like Xrebel from ZeroTurnaround, jClarity's Censum and Illuminate take the next step of applying machine learned analysis to the raw numbers. One final important note. Project Jigsaw is the modularisation story for Java 9 that will massively impact tool vendors and day to day developers alike. The community at large needs your help to help test out early builds of Java 9 and to help OpenJDK developers and tool vendors ensure that IDEs, build tools and applications are ready for this important change. You can join us in the Adoption Group at OpenJDK: http://adoptopenjdk.java.net I hope everyone has a great holiday break - I look forward to seeing the Twitter feeds and the GitHub commits flying around in 2015 :-).
Cheers,
Martijn (CEO - jClarity, Java Champion & Diabolical Developer)

This post is part of the Java Advent Calendar and is licensed under the Creative Commons 3.0 Attribution license. If you like it, please spread the word by sharing, tweeting, FB, G+ and so on!

[sysadvent] Day 23 - The Importance of Pluralism, or The Danger of the Letter "S"

Christopher Webber — Tue, 23 Dec 2014 00:00:00 +0000

Written by: Mike Fiedler (@mikefiedler)
Edited by: Hugh Brown (@saintaardvark)

Prologue: A Concept

One aspect of Chef that’s confusing to people comes up when searching for nodes that have some attribute: just what is the difference between a nodes reported ‘role’ attribute, and its ‘roles’ attribute? It seems like it could almost be taken for a typo – but underlying it are some very deep statements about pluralism, pluralization, and the differences between them.

One definition of the term ‘pluralism’ is “a condition or system in which two or more states, groups, principles, sources of authority, etc., coexist.” And while pluralism is common in descriptions of politics, religion and culture, it also has a place in computing: to describe situations in which many systems are in more than one desired state.

Once a desired state is determined, it’s enforced. But then time passes – days, minutes, seconds or even nanoseconds – and every moment has the potential to change the server’s actual state. Files are edited, hardware degrades, new data is pulled from external sources; anyone who has run a production service can attest to this.

Act I: Terms

Businesses commonly offer products. These products may be composed of multiple systems, where each system could be a collection of services, which run on any number of servers, which run on some amount of hosts. Each host, in turn, provides another set of services to the server that makes up part of the system, which then makes up part of the product, which the business sells.

An example to illustrate: MyFace offers a social web site (the product), which may need a web portal, a user authentication system, index and search systems, long-term photo storage systems, and many more. The web portal system may need servers like Apache or Nginx, running on any number of instances. A given server-instance will need to use any number of host services, such as /o, cpu, memory and more.

So what we loosely have is: products => systems => services => servers => hosts => services. (Turtles, turtles, turtles.)

In Days of Yore, when a Company ran a ‘Web Site’, they may have had a single System, maybe some web content Service, made up of a web Server, a database Server (maybe even on the same host) - both consuming host services (CPU, Memory, Disk, Network) - to provide the Service the Company then sells, hopefully at a profit (right!?).

Back then, if you wanted to enact a change on the web and database at the same time (maybe release a new feature), it was relatively simple, as you could control both things in one place, at roughly the same time.

Intermission

In English, to pluralize something, we generally add a suffix of “s” to the word. For instance, to convey more than one instance, “instance” becomes “instances”, “server” becomes “servers”, “system” becomes “systems”, “turtle” becomes “turtles”.

We commonly use pluralization to describe the concept of a collection of similar items, like “apples”, “oranges”, “users”, “web pages”, “databases”, “servers”, “hosts”, “turtles”. I think you see the pattern.

This extends even in to programming languages and idiomatic use in development frameworks. For example, all tables in a Rails application will typically pluralize a table name for objects named Apple to apples.

This emphasizes that the table in question does not store a singular Apple, rather many Apple instances will be located in a table named apples.

This is not pluralism, this is pluralization - don’t get them confused. Let’s move on to the next act.

Act II: Progress

We’ve evolved quite a bit since the Days of Yore. Now, a given business product can span hundreds or even thousands of systems of servers running on hosts all over the world.

As systems grow, it becomes more difficult to enact a desired change at a deterministic point in time across a fleet of servers and hosts.

In the realm of systems deployment, many solutions perform what has become known as “test-and-repair” operations - meaning that when provided a “map” desired state (which typically manifests in human-written and readable code), that when executed, will “test the current state of a given host, and perform and ”repair" operations to bring the host to the desired state - whether it be installing packages, writing files

Each system calls this map something different - cfengine:policies, bcfg2:specifications, puppet:modules, chef:recipes, ansible:playbooks, and so on. While they don’t always map 1:1, they all have some sort of concept for ‘Things that are similar, but not the same.’ They will have unique IP addresses, hostnames, while sharing enough of a set of common features to become termed something like “web heads” or the like.

Act III: Change

In the previous sections, I laid the groundwork to understand one of the more subtle features in Chef. This feature may be available in other services, but I’ll describe the one I know.

Using Chef, there is a common deployment model where Chef Clients check in with a Chef Server to ask “What is the desired state I should have?” The Chef terminology is ‘a node asks the server for its run list’.

A run list can contain a list of recipes and/or roles. A recipe tells Chef how to accomplish a particular set of tasks, like installing a package or editing a file. A role is typically a collection of recipes, and maybe some role-specific metadata (‘attributes’ in Chef lingo).

The node may be in any state at this point. Chef will test for each desired state, and take action to enforce it: install this package, write that file, etc. The end result should either be “this node now conforms to the desired state” or “this node was unable to comply”.

When the node completes successfully, it will report back to Chef Server that “I am node ‘XYZZY’, and my roles are ‘base’ and ‘webhead’, my recipes are ‘base::packages’, ‘nginx’, ‘webapp’” along with a lot of node-specific metdata (IP addresses, CPU, Memory, Disk, and much more).

This information is then indexed and available for others to search for. A common use case we have is where a load balancing node will perform a search for all nodes holding the webhead role, and add these to the balancing list.

Pièce de résistance, or Searching for Servers

In a world where we continue to scale and deploy systems rapidly and repeatedly, we often choose to reduce the need for strong consistency amongst a cluster of hosts. This means we cannot expect to change all hosts at the precise same moment. Rather we opt for eventual consistency: either all my nodes will eventually be correct, or failures will occur and I’ll be notified that something is wrong.

This changes how we think about deployments, and more importantly, how do we use our tools to find other nodes.

Using Chef’s search feature, a search like this:

webheads = search(:node, 'role:webheads')

will use the node index (a collection of node data) to look for nodes with the webheads role in the node’s run list - this will also return nodes that have not yet completed an initial Chef run and reported the complete run list back to Chef Server.

This means that my load balancer could find a node that is still mid-provisioning, and potentially begin to send traffic to a node that’s not ready to receive yet, based on the role assignment alone.

A better search, in this case might be:

webheads = search(:node, 'roles:webheads')

One letter, and all the difference.

This search now looks for an “expanded list” that the node has reported back. Any node with the role webheads that has completed a Chef run would be included. If the mandate is that only webhead nodes get the webhead role assigned to them, then I can safely use this search to include nodes that have completed their provisioning cycle.

Another way to use this search to our benefit is to search one axis and compare with another to find nodes that never completed provisioning:

badnodes = search(:node, 'role:webheads AND NOT roles:webheads')
# Or, with knife command line:
$ knife search node 'role:webheads AND NOT roles:webheads'

This will grab any nodes with an assignment but not a completion – very helpful when launching large amounts of nodes.

Note: This is not restricted to roles; this also applies to recipe/recipes. I’ve used roles here, as we use them heavily in our organization, but the same search patterns apply for using recipes directly in a run list.

Curtain

This little tidbit of role vs roles has proven time and again to be a confusing point when someone tries to pick up more of Chef’s searching abilities. But having both adjectives describe a state of the node is helpful in making a determination of what state the node is in, and whether it should be included in some other node’s list (such as in the loadbalancer/webhead example from before).

Now, you may argue against the use of roles entirely, or the use of Chef Server and search, and use something else for service discovery. This is a valid argument - but be careful you’re not tethering a racehorse to a city carriage. If you don’t fully understand its abilities, someday it might run away on you.

Epilogue

A surgeon spends a lot of time how to use a sharpened bit of metal to fix the human body. While there are many instruments he or she will go on to master, the scalpel remains the fundamental tool, available when all else is gone.

While we don’t have the same risks involved as a surgeon, the tools we use can be more complex, and provide us with a large amount of power at our fingertips.

It behooves us to learn how they work, and when and how to use its features to provide better systems and services for our businesses.

Chef’s ability to discern between what a node has been told about itself, and what it reports about itself, can make all the difference when using Chef to accomplish complex deployment scenarios and maintain flexible infrastructure as code. This not only lets you accomplish fundamentals of service discovery and less hard-coded configurations, but lets you avoid the uncertainty of bringing in yet another outside tool.

On that note, Happy Holiday(s)!

[ux] Designing For the Slow Web

Mon, 22 Dec 2014 21:00:00 +0000

Let’s face it. Our obsession with the internet has gotten out of hand.

We’re spending more time online than ever before. Every spare moment is spent in front of a screen—whether it’s checking our email in bed, Facebook on the train or Twitter in the toilet (come on, admit it). And the explosion in mobile this decade means that we are always online, to some degree.

On top of that, we are constantly bombarded with notifications—likes, retweets, favourites, friend requests, alerts. And 'shiny-new-object syndrome’ is always drawing us towards the newest gadget or app.

On average:

An adult will spend over 5 hours per day online
Workers send and receive 40 email per day, and an estimated 20 notifications
People visit over 10 websites per day
A smartphone user will picks up their phone more than 1,500 times in any week

The stream of information flows like a sip from a firehouse and we can’t stop it. Even if we wanted to. And it’s getting us nowhere.

Jack Cheng calls this the era of the ‘Fast Web’:

What is the Fast Web? It’s the out of control web. The "oh my god there’s so much stuff, I can’t possibly keep up" web. It’s the "spend two dozen times a day checking" web. The "in one end and out the other" web. The web designed to appeal to the basest of our intellectual palettes, the salt, sugar and fat of online content web. It’s the scale hard and fast web. The "create a destination for billions of people" web. The you have two hundred twenty six new updates web. Keep up or be lost. Click me. Like me. Tweet me. Share me. The Fast Web demands that you do things and do them now. The Fast Web is a cruel wonderland of shiny shiny things.

And you know what? Things are going to get worse before they get better. The advances in new technologies around home automation will only add to the noise. Soon your alarm clock will be talking to your coffee machine, and you’ll get a notification about it. And wearable technology means we’ll never be completely off the grid.

It’s not just us, it’s our children too. Leading paediatricians suggest that kids should spend no more than 2 hours per day in front of a screen. Yet, they too are obsessed. The task of keeping a child away from an ipad is sisyphean.

'Internet addiction disorder’ is now recognised as a legitimate clinical disorder, with its own wikipedia page. And don’t get me started with 'Phantom vibration syndrome’.

It can’t go on like this. The fast web is killing us.

The first step to recovery is admitting we have a problem. But what’s the solution?

The slow web: the antidote to our fast-web woes.

Imagine a web that wasn’t so rushed. A web that empowered us to consume it in a more leisurely fashion. A web that felt more like a farmers market than a hyper market. Perhaps even a web that encouraged you to disconnect from time to time.

You’ve heard of slow food, but have you heard about the slow web movement?

Again, I’ll allow Jack to explain:

The Slow Web Movement is a lot like the Slow Food Movement, in that they’re both blanket terms that mean a lot of different things. Slow Food began in part as a reaction to the opening of a McDonald’s in Piazza di Spagna in Rome, so from its very origin, it was defined by what it’s not. It’s not Fast Food, and we all know what Fast Food is… right?

We spend way too much time eating fast-food online. Facebook is the cheeseburger of the web. Twitter is the chicken nugget. We need to cut down on quarter-pounders, and we need to spend less time the fast-web too.

I think 2015 will be the year of the slow web. And together we can help build it.

Designing for a slower web

As designers, we’ll never win over our users if we’re constantly hassling them with more notifications and email updates.

The slow web is the opposite. It’s about getting out of our user’s way. It’s about reducing the noise, not adding to it. It’s about providing real quality, not quantity. It’s about turning down the volume, and letting our users enjoy the quiet.

So how might we design for the slow web? By showing you some examples below, I hope provide some ideas into how we can do it.

Do less, but better

As designers and product people, we’re under constant pressure to give more. More content, more features. More updates.

We need to get off this hamster wheel and aim to give our users more by giving them less. Focusing on producing more quality, not more quantity.

Probably one of the early winners of the slow web movement is Medium. It’s the writing platform that that has made long-form reading popular again. Medium has developed a community of quality writing and imagery from its contributors. It feels more like blogging for the slow web.

Designing for a slower web is also about making things simpler and intuitive without draining our cognitive resources. Let’s think about reducing features, and delivering fewer, but better, updates so that we can get out of our users’ way sooner.

Hold back on notifications

Here are two notifications I received today. The first one was from Fire Ready app, alerting me of a fire near my home. An important message that I’m interested in knowing about and that I opted-in to. The second was an irrelevant message from Shazaam that I never signed up for.

Guess which one pissed me off the most?

In a quest to get users to to form long-lasting habits with our products, we’re bombarding our user base with shit and dribble. The Android design guide puts it more politely:

Unwanted or unimportant notifications can annoy the user or make them resent how much attention the app wants from them, so use notifications judiciously.

Next year, let’s be more strategic about using notifications. Let’s make sure that the notifications we send are:

Crucial - information that the user must see.
Expected - make sure that your users have clearly opted in to your notifications, and are expecting your message.
Timely - Don’t send notifications in the middle of the night. Send notifications at a time that is most useful for your user.

Or, perhaps, we shouldn’t send any at all.

The web on my watch

One of the biggest problems with the fast web is the enormous sense of immediacy. The Dopamine hit. The sense that something is out of date when it’s five minutes old. And the need to ‘stay up to date’ is a major cause of anxiety. At least for me.

A slower web should reduce immediacy. It should allow people to take use it on their terms, and at a time that suits them.

One of my favourite web applications is Pocket. Much like Instapaper before it, Pocket allows me to save articles and blog posts to for later reading. I can read saved articles whenever I choose, even when I’m offline. This save-for-later feature gives me a greater control over the web I consume, and also where and when I consume it.

Likewise, we should be creating experiences that fit a little better into the rhythm of our users’ lives. Don’t just think about how and why—understand the cadence of your experience, and give your users control over when, and how often, they interact with you.

Send them outside

As slow web designers, perhaps we should be encouraging our users to spend less time using our apps. Less time in front of screens.

There are some great examples of technology that do this well.

Firstly, there’s Strava, a run-tracking app that’s all about getting you outside to run or ride.

Or Moment. Moment is an iOS app that automatically tracks how much you use your iPhone and iPad each day - and actively encourages you to turn off.

And then there’s Toymail. Toymail make wi-fi connected toys that encourages your kids to express themselves and communicate with family digitally, but without the screen exposure. Using an Android or iOS app, you can send messages to your children, via the Mailmen toys - like snort here.

Kids can also reply right back, again via the toy. So it’s a two-way conversion that doesn’t happen over a screen.

Are there ways that you can encourage your users to stop, unplug, and smell the roses?

Respect thy user

Ultimately, designing for the slow web is about showing respect for our users.

Your users are probably very busy people, with constant distractions in their lives. So make sure you're always respecting their time. Never be a distraction to your user’s already-busy day. Never be pushy.

A slower web should reduce the noise and informational clutter, not add to it. So next time you’re adding a feature, or sending out a notification, ask yourself, “Is this interaction or this touch-point truly going to make my user’s day better?”

If not, why do it?

[24ways] Integrating Contrast Checks in Your Web Workflow

feeds@allinthehead.com (Geri Coady) — Mon, 22 Dec 2014 12:00:00 +0000

Geri Coady bends low the Christmas tree branches with bright and contrasting baubles. This isn’t some gaudy seasonal distraction, though. It’s responsible accessibility advice you can work with throughout the year.

[perl] A Holiday PAPR-ation

John SJ Anderson — Mon, 22 Dec 2014 05:00:00 +0000

Jesse, Junior IT Elf Second Class grumbled to himself, "Who even thought this was a good idea anyway." He had grabbed a ticket out of the North Pole's ticketing system, thinking that adding a simple new endpoint to an existing RESTful API would be a fairly trivial task. He had hoped to get out of the office early today, and instead he was staring at a creeping horror and the possibility of working all weekend.

Clearly, the code had originally started out as a very simple API of only a few endpoints. The original coder, either being in a hurry or just not knowing any better, had decided to implement it as a series of if - elsif - else blocks based on the first element in the URL string.

Now, 10 years later, there were dozens of elsif blocks, each with its own specialized handling of the subsequent parts of the URL. There was no overall organization to the code -- it seemed as if every time a new endpoint was required, the elf doing the work had just slapped in an elsif block whereever they felt like. The end result now had a structure something like this (only much worse -- the example is sanitized for a family audience):

`1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24: 25: 26: 27: 28: 29: 30: 31: 32: 33: 34: 35: 36: 37: 38: 39:`	use CGI; my $cgi = CGI->new(); my( $first , @rest ) = $cgi->path_info(); # /present/present_id/action if( $first eq 'present' ) { my( $id , $action ) = @rest; validate_present_id( $id ) or return_error("bad id"); if( $action ) { return_error("bad method") unless $ENV{REQUEST_METHOD} eq 'POST'; $action =~ /^(add\|update\|delete)$/ or return_error("bad action"); _handle_present_update( $id , $action ); } else { return_error("bad method") unless $ENV{REQUEST_METHOD} eq 'GET'; _handle_present_get( $id ) } } # /good_child/child_id/reason elsif( $first eq 'good_child' ) { my( $id , $reason ) = @rest; validate_child_id( $id ) or return_error("bad id"); validate_reason( $reason) or return_error("bad reason"); return_error("bad method") unless $ENV{REQUEST_METHOD} eq 'POST'; _handle_child_update( 'good_child' , $id , $reason ); } # /bad_child/child_id/reason elsif( $first eq 'bad_child' ) { my( $id , $reason ) = @rest; validate_child_id( $id ) or return_error("bad id"); validate_reason( $reason) or return_error("bad reason"); return_error("bad method") unless $ENV{REQUEST_METHOD} eq 'POST'; _handle_child_update( 'bad_child' , $id , $reason ); } else { return_error("no match!") }

As Jesse sat and stared at screen after screen of cascading conditionals, grumbling and muttering to himself, his boss Margaret walked by and asked, "What's going on, Jesse?"

Jesse started to rant about the API code, but Margaret cut him off quickly. "Listen, didn't you see the last memo that came out from the Big Guy? We're supposed to be replacing all those old CGIs with PSGIs when ever we have to make any changes to them. This is your lucky day, Junior! You get to clean up this mess. But we've got apps deployed in the field that rely on this API, so make sure you don't change any of the endpoints while you're porting it over, or there will be heck to pay."

"Fine", Jesse sighed. "I guess I can go on MetaCPAN and research the best way to..."

Margaret cut him off again. "No need, kid. Just use PAPR -- Plack::App::Path::Router!"

Jesse sighed again as he pulled up the documentation on MetaCPAN. He learned that Plack::App::Path::Router provides a convenient way to take a Path::Router specification and wrap it up as a PSGI. Jesse was quickly able to turn the existing conditional cascade into a much more declarative set of add_route statements. He also quickly realized that he could replace all the custom validation routines in the old code with Path::Router's parameter extraction and Moose-style validations.

Within a few hours, Jesse had converted all the old style code to a form that looked like this:

`1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24: 25: 26: 27: 28: 29: 30: 31: 32: 33: 34: 35: 36: 37: 38: 39: 40: 41: 42: 43:`	use Plack::App::Path::Router; use Path::Router; my $router = Path::Router->new(); # note - formerly handled by a single conditional stanza; now split into two $router->add_route( '/present/:present_id' => ( validations => { present_id => 'Int' } , target => sub { # $request is a L<Plack::Request> object my( $request , $id ) = @_; return_error("bad method") unless $request->method() eq 'GET'; _handle_present_get( $id ) } )); $router->add_route( '/present/:present_id/:action' => ( validations => { present_id => 'Int' , action => qr/^(add\|update\|delete)$/ , # validations can also be plain regexps }, target => sub { my( $request , $id , $action ) = @_; return_error("bad method") unless $request->method() eq 'POST'; _handle_present_update( $id , $action ); } )); # covers both /good_child/ and /bad_child/ endpoints $router->add_route( '/:child_type/:child_id/:reason' ) => ( validations => { child_type => qr/^(good\|bad)_child/ , child_id => 'Int' , reason => 'Str' , } , target => sub { my( $request , $type , $id , $reason ) = @_; return_error("bad method") unless $request->method() eq 'POST'; _handle_child_update( $type , $id , $reason ); } )); # now create the Plack app my $app = Plack::App::Path::Router->new( router => $router );

He commited his changes, and then added in one more change that added the new endpoint that had originally started him down this path. Work all done, he pushed his branch, and moved the ticket into the code review stage. While it had taken him a little longer than he expected, it still looked like he'd be done early.

As he made one last scan over his email, Jesse realized that Margaret had already responded to his code review request ... and worse, she'd rejected his changes! She was asking for something that was more data-driven and didn't have so much boilerplate code. Her suggestion was to look at Plack::App::Path::Router::Custom -- which Jesse dutifully pulled up on MetaCPAN. After a few minutes of reading, he saw exactly what she was talking about. He rapidly converted the previous version into this completely data-driven form:

`1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24: 25: 26: 27: 28: 29: 30: 31: 32: 33: 34: 35: 36: 37: 38: 39: 40: 41: 42: 43: 44: 45: 46: 47: 48: 49: 50: 51: 52: 53: 54: 55: 56: 57:`	use Path::Router; use Plack::App::Path::Router; use Plack::Request; use SantasWorkshop::API; # all the '_handle' methods were migrated to here # each entry is 'route definition' => 'method' => 'hashref of options' my @routes = ( [ '/present/:present_id' => 'handle_present_get' => { methods => { 'GET' => 1 } , validations => { present_id => 'Int' } , }] , [ '/present/:present_id/:action' => 'handle_present_update' => { methods => { 'POST' => 1 } , validations => { present_id => 'Int' , action => qr/^(add\|update\|delete)$/ , # validations can also be plain regexpsld }, }] , # covers both /good_child/ and /bad_child/ endpoints [ '/:child_type/:child_id/:reason' => 'handle_child_update' => { methods => { 'POST' => 1 } , validations => { child_type => qr/^(good\|bad)_child/ , child_id => 'Int' , reason => 'Str' , } , }] , ) # dynamically build the routing table based on the @routes data structure my $router = Path::Router->new(); foreach (@routes) { my ( $endpoint, $action, $options ) = @$_; $router->add_route( $endpoint => ( target => sub { my ( $request, @rest ) = @_; if ( exists $options->{methods} ) { my $request_method = $request->method(); return_error("bad method") unless $options->{methods}{$request_method}; } return SantasWorkshop::API->new()->$action(@rest); } , validations => $options->{validations} , )); } # once the routing table is all build, make the app using that router. my $app = Plack::App::Path::Router::Custom->new( router => $router, new_request => sub { Plack::Request->new(shift) } , );

Jesse pushed the final revision up for code review and got notification of Margaret's merging of the code into the release branch only a few minutes later -- along with an IM saying, "Nice work, kid -- how about you take off early today."

[java] A persistent KeyValue Server in 40 lines and a sad fact

Rüdiger Möller — Mon, 22 Dec 2014 00:32:00 +0000

Advent time again .. picking up Peters well written overview on the uses of Unsafe, i'll have a ~~short~~ fly-by on how low level techniques in Java can save development effort by enabling a higher level of abstraction or allow for Java performance levels probably unknown to many.

My major point is to show that conversion of Objects to bytes and vice versa is an important fundamental, affecting virtually any modern java application.

Hardware enjoys to process streams of bytes, not object graphs connected by pointers as "All memory is tape" (M.Thompson if I remember correctly ..).

Many basic technologies are therefore hard to use with vanilla Java heap objects:

Memory Mapped Files - a great and simple technology to persist application data safe, fast & easy.
Network communication is based on sending packets of bytes
Interprocess communication (shared memory)
Large main memory of today's servers (64GB to 256GB). (GC issues)
CPU caches work best on data stored as a continuous stream of bytes in memory

so use of the Unsafe class in most cases boil down in helping to transform a java object graph into a continuous memory region and vice versa either using

[performance enhanced] object serialization or
wrapper classes to ease access to data stored in a continuous memory region.

(source of examples used in this post can be found here, messaging latency test here)

Serialization based Off-Heap

Consider a retail WebApplication where there might be millions of registered users. We are actually not interested in representing data in a relational database as all needed is a quick retrieve of user related data once he logs in. Additionally one would like to traverse the social graph quickly.

Let's take a simple user class holding some attributes and a list of 'friends' making up a social graph.

easiest way to store this on heap, is a simple huge HashMap.

Alternatively one can use off heap maps to store large amounts of data. An off heap map stores its keys and values inside the native heap, so garbage collection does not need to track this memory. In addition, native heap can be told to automagically get synchronized to disk (memory mapped files). This even works in case your application crashes, as the OS manages write back of changed memory regions.

There are some open source off heap map implementations out there with various feature sets (e.g. ChronicleMap), for this example I'll use a plain and simple implementation featuring fast iteration (optional full scan search) and ease of use.

Serialization is used to store objects, deserialization is used in order to pull them to the java heap again. Pleasantly I have written the (afaik) fastest fully JDK compliant object serialization on the planet, so I'll make use of that.

Done:

persistence by memory mapping a file (map will reload upon creation).
Java Heap still empty to serve real application processing with Full GC < 100ms.
Significantly less overall memory consumption. A user record serialized is ~60 bytes, so in theory 300 million records fit into 180GB of server memory. No need to raise the big data flag and run 4096 hadoop nodes on AWS ;).

Comparing a regular in-memory java HashMap and a fast-serialization based persistent off heap map holding 15 millions user records, will show following results (on a 3Ghz older XEON 2x6):

	consumed Java Heap (MB)	Full GC (s)	Native Heap (MB)	get/put ops per s	required VM size (MB)
HashMap	6.865,00	26,039	0	3.800.000,00	12.000,00
OffheapMap (Serialization based)	63,00	0,026	3.050	750.000,00	500,00

[test source / blog project] Note: You'll need at least 16GB of RAM to execute them.

As one can see, even with fast serialization there is a heavy penalty (~factor 5) in access performance, anyway: compared to other persistence alternatives, its still superior (1-3 microseconds per "get" operation, "put()" very similar).

Use of JDK serialization would perform at least 5 to 10 times slower (direct comparison below) and therefore render this approach useless.

Trading performance gains against higher level of abstraction: "Serverize me"

A single server won't be able to serve (hundreds of) thousands of users, so we somehow need to share data amongst processes, even better: across machines.

Using a fast implementation, its possible to generously use (fast-) serialization for over-the-network messaging. Again: if this would run like 5 to 10 times slower, it just wouldn't be viable. Alternative approaches require an order of magnitude more work to achieve similar results.

By wrapping the persistent off heap hash map by an Actor implementation (async ftw!), some lines of code make up a persistent KeyValue server with a TCP-based and a HTTP interface (uses kontraktor actors). Of course the Actor can still be used in-process if one decides so later on.

Now that's a micro service. Given it lacks any attempt of optimization and is single threaded, its reasonably fast [same XEON machine as above]:

280_000 successful remote lookups per second
800_000 in case of fail lookups (key not found)
serialization based TCP interface (1 liner)
a stringy webservice for the REST-of-us (1 liner).

[source: KVServer, KVClient] Note: You'll need at least 16GB of RAM to execute the test.

A real world implementation might want to double performance by directly putting received serialized object byte[] into the map instead of encoding it twice (encode/decode once for transmission over wire, then decode/encode for offheaping map).

"RestActorServer.Publish(..);" is a one liner to also expose the KVActor as a webservice in addition to raw tcp:

C like performance using flyweight wrappers / structs

With serialization, regular Java Objects are transformed to a byte sequence. One can do the opposite: Create wrapper classes which read data from fixed or computed positions of an underlying byte array or native memory address. (E.g. see this blog post).

By moving the base pointer its possible to access different records by just moving the the wrapper's offset. Copying such a "packed object" boils down to a memory copy. In addition, its pretty easy to write allocation free code this way. One downside is, that reading/writing single fields has a performance penalty compared to regular Java Objects. This can be made up for by using the Unsafe class.

"flyweight" wrapper classes can be implemented manually as shown in the blog post cited, however as code grows this starts getting unmaintainable.
Fast-serializaton provides a byproduct "struct emulation" supporting creation of flyweight wrapper classes from regular Java classes at runtime. Low level byte fiddling in application code can be avoided for the most part this way.

How a regular Java class can be mapped to flat memory (fst-structs):

Of course there are simpler tools out there to help reduce manual programming of encoding (e.g. Slab) which might be more appropriate for many cases and use less "magic".

What kind of performance can be expected using the different approaches (sad fact incoming) ?

Lets take the following struct-class consisting of a price update and an embedded struct denoting a tradable instrument (e.g. stock) and encode it using various methods:

a 'struct' in code

Pure encoding performance:

Structs	fast-Ser (no shared refs)	fast-Ser	JDK Ser (no shared)	JDK Ser
26.315.000,00	7.757.000,00	5.102.000,00	649.000,00	644.000,00

Real world test with messaging throughput:

In order to get a basic estimation of differences in a real application, i do an experiment how different encodings perform when used to send and receive messages at a high rate via reliable UDP messaging:

The Test:
A sender encodes messages as fast as possible and publishes them using reliable multicast, a subscriber receives and decodes them.

Structs	fast-Ser (no shared refs)	fast-Ser	JDK Ser (no shared)	JDK Ser
6.644.107,00	4.385.118,00	3.615.584,00	81.582,00	79.073,00

(Tests done on I7/Win8, XEON/Linux scores slightly higher, msg size ~70 bytes for structs, ~60 bytes serialization).

Slowest compared to fastest: factor of 82. The test highlights an issue not covered by micro-benchmarking: Encoding and Decoding should perform similar, as factual throughput is determined by Min(Encoding performance, Decoding performance). For unknown reasons JDK serialization manages to encode the message tested like 500_000 times per second, decoding performance is only 80_000 per second so in the test the receiver gets dropped quickly:

"
...
***** Stats for receive rate: 80351 per second *********
***** Stats for receive rate: 78769 per second *********
SUB-ud4q has been dropped by PUB-9afs on service 1
fatal, could not keep up. exiting
"
(Creating backpressure here probably isn't the right way to address the issue ;-) )

Conclusion:

a fast serialization allows for a level of abstraction in distributed applications impossible if serialization implementation is either
- too slow
- incomplete. E.g. cannot handle any serializable object graph
- requires manual coding/adaptions. (would put many restrictions on actor message types, Futures, Spore's, Maintenance nightmare)
Low Level utilities like Unsafe enable different representations of data resulting in extraordinary throughput or guaranteed latency boundaries (allocation free main path) for particular workloads. These are impossible to achieve by a large margin with JDK's public tool set.
In distributed systems, communication performance is of fundamental importance. Removing Unsafe is not the biggest fish to fry looking at the numbers above .. JSON or XML won't fix this ;-).
While the HotSpot VM has reached an extraordinary level of performance and reliability, CPU is wasted in some parts of the JDK like there's no tomorrow. Given we are living in the age of distributed applications and data, moving stuff over the wire should be easy to achieve (not manually coded) and as fast as possible.

Addendum: bounded latency

A quick Ping Pong RTT latency benchmark showing that java can compete with C solutions easily, as long the main path is allocation free and techniques like described above are employed:

[credits: charts+measurement done with HdrHistogram]

This is an "experiment" rather than a benchmark (so do not read: 'Proven: Java faster than C'), it shows low-level-Java can compete with C in at least this low-level domain.
Of course its not exactly idiomatic Java code, however its still easier to handle, port and maintain compared to a JNI or pure C(++) solution. Low latency C(++) code won't be that idiomatic either ;-)

About me: I am a solution architect freelancing at an exchange company in the area of realtime GUIs, middleware, and low latency CEP (Complex Event Processing).
I am blogging at http://java-is-the-new-c.blogspot.de/,
hacking at https://github.com/RuedigerMoeller.

This post is part of the Java Advent Calendar and is licensed under the Creative Commons 3.0 Attribution license. If you like it, please spread the word by sharing, tweeting, FB, G+ and so on!

[sysadvent] Day 22 - Largely Unappreciated Applicability

Christopher Webber — Mon, 22 Dec 2014 00:00:00 +0000

Written by: John Vincent (@lusis)
Edited by: Joseph Kern (@josephkern)

I have had the privilege of writing a post for SysAdvent for the past several years. In general these posts have been focused on broader cultural issues. This year I wanted to do something more technical and this topic gave me a chance to do that. It’s also just REALLY cool so there’s that.

Nginx

I’m sure most people are familiar with Nginx but I’m going to provide a short history anyway. Nginx is a webserver created by Igor Sysoev around 2002 to address the C10K problem. The C10K problem isn’t really a “problem” anymore in the sense that is was originally. It’s morphed into the C10M problem. With the rise of the sensors, we may be dealing with a Cgazillion problem before we know it.

Nginx addressed this largely with an event loop (I know some folks think the event loop was invented in 2009). The dominant webserver at the time (and still), Apache, used a model of spawning a new process or a new thread for each connection. Nginx did it differently in that it spawned a master process (handles configuration, launching worker threads and the like) and a pool of workers each with their own event loop. Workers share no state between each other and select from a shared socket to process requests. This particular model works and scales very well. There’s more on the history of Nginx in its section in the second edition of AOSA. It’s a good read and while not 100% current, the basics are unchanged.

Lua

Lua is a programming language invented in 1993. The title of this article is a shout out to how underappreciated Lua is not only as a language but in its myriad uses. Most people who have heard of Lua know it as the language used for World of Warcraft plugins.

Lua is an interesting language. Anyone with experience in Ruby will likely find themselves picking it up very quickly. It has a very small core language, first-class functions and coroutines. It is dynamically typed and has one native data structure - the table. When you work in Lua, you will learn to love and appreciate the power of tables. They feel a lot like a ruby hash and are the foundation of most advanced Lua.

It has no classes, but they can be implemented after a fashion using tables. Since Lua has first-class functions, you can create a “class” by lumping data and function into a table. There’s no inheritence but instead you have prototypes (There’s a bit of sugar to help you out when working with these ‘objects’ - e.g calling foo:somefunc() to imply self as the first argument as opposed to foo.somefunc(self)).

For a good read on the language history - Wikipedia - Lua website

For some basics on the language itself, the wikipedia article has code sample and also the official documentation. There is also a section on Lua in the newest edition of the Seven Languages series - Seven More Languages in Seven Weeks

I’ve also written a couple of modules as well (primary for use with OpenResty):

If you want to see an example of how the “classes” work with Lua, take a look at the github example and compare the usage described in the README with the module itself.

Combining the two

As I mentioned, Lua is an easily embeddable language. I’ve been unable to find a date on when Lua support was added to Nginx but it was a very early version (~ 0.5).

One of the pain points of Nginx is that it doesn’t support dynamically loaded modules. All extended functionality outside the core must be compiled in. Lua support in Nginx made it so that you could add some advanced functionality to Nginx via Lua that would normally require a C module and a recompile.

Much of the Nginx api itself is exposed to Lua directly and Lua can be used at multiple places in the Nginx workflow. You can:

modify request or response headers via Lua
create rewrite rules via Lua.
create content itself via Lua.
dynamically set redirection and proxy backends
postprocess log events in a non-blocking fashion (i.e. send directly to Logstash from Nginx)

All of these are documented on the Nginx website.

For example, if I wanted to have the response body be entirely created by Lua, I could do the following in Nginx:

location /foo {
  content_by_lua '
  ngx.header.content_type = 'text/plain'
  local username = "bob"
  ngx.say("hello ", username)
  ngx.exit(ngx.HTTP_OK)
  '
}

This example would return hello bob as plain text to your browser when you requested /foo from Nginx.

Obviously escaping could get to be a headache here so most of the *_by_lua directives (which is for inlined Lua code in the Nginx config files) can be replaced with a *_by_lua_file where the Lua code is stored in an external file.

Some other neat tricks you have available are using the cosocket api where you can actually open arbitrary non-blocking network connections via Lua from inside an Nginx worker.

As you can see, this is pretty powerful. Additionally the Lua functionality is provided in Nginx via a project called LuaJIT which offers amazing speed and predicatable usage. By default, Lua code is cached in Nginx but this can be disabled at run-time to help speed up the development process.

Enter the OpenResty

If it wasn’t clear yet, the combination of Nginx and Lua basically gives you an application server right in the Nginx core. Others have created Lua modules specifically for use within Nginx and a few years ago an enterprising soul started bundling them up into something called OpenResty.

OpenResty combines checkpointed versions of Nginx, modified versions of the Lua module (largely maintained by the OpenResty folks anyway), curated versions of LuaJIT and a boatload of Nginx-specific Lua modules into a single distribution. OpenResty builds of Nginx can be used anywhere out-of-the-box that you would use a non-Lua version of Nginx. Currently OpenResty is sponsored by CloudFlare where the primary author, Yichun Zhang (who prefers to go by “agentzh” everywhere) is employed.

OpenResty is a pretty straightforward “configure/make/make install” beast. There is a slightly dated omnibus project on Github from my friend Brian Akins that we’ve contributed to in the past (and will be contributing our current changes back to in the future). Much of my appreciation and knowledge of Lua and OpenResty comes directly from Brian and his omnibus packages are how I got started.

But nobody builds system packages anymore

Obviously system packages are the domain of greyhaired BOFHs who think servers are for serving. Since we’re all refined and there are buzzword quotas to be maintained, you should probably just use Docker (but you have to say it like Benny says “Spaceship”).

Seriously though, Docker as a packaging format is pretty neato and for what I wanted to do, Docker was the best route. To that end I give you an OpenResty tutorial in a box (well, a container).

The purpose of this repo is to help you get your feet wet with some examples of using Lua with Nginx via the latest OpenResty build. It ships with a Makefile to wrap up all the Docker invocations and hopefully make things dead simple. It works its way up from the basics I’ve described all the way to communicating between workers via a shared dictionary, making remote API calls to Github, two Slack chat websocket “clients” and the skeleton of a dynamic load balancer in Nginx backed by etcd:

In addition, because I know how difficult it can be to develop and troubleshoot against code running inside Nginx I’ve created a web based repl for testing out and experimenting with the Nginx Lua API:

To use the basic examples in the container, you can simply clone the repo and run make all. This will build the container and then start OpenResty listening on port 3131 (and etcd on 5001 for one of the demos). The directory var_nginx will be mounted inside the container as /var/nginx and contains all the neccessary config files and Lua code for you to poke/prod/experiment with. Logs will be written to var_nginx/logs so you can tail them if you’d like. As you can see it also uses Bootstrap for the UI so we’ve pretty much rounded out the “what the hell have you built” graph.

Please note that while the repo presents some neat tricks, the code inside is not optimized by any stretch. The etcd code especially may have some blocking implications but I’ve not yet confirmed that. The purpose is to teach and inspire more than “take it and run it in prod”.

Advanced Examples

If you’d like to work with the Slack examples, you’ll need to generate a slack “bot” integration token for use. The Makefile includes support for running an etcd container appropriate for use with the tutorial container. If you aren’t a Slack user then here’s a screenshot so you can see what it WOULD look like:

Wrap up

Maybe this post has inspired you to at least take a look at OpenResty. Lua is a really neat language and very easy to pick up and add to your toolbelt. We use OpenResty builds of Nginx in many places internally from proxy servers to even powering our own internal SSO system based on Github Oauth and group memeberships. While most people simply use Nginx as a proxy and static content service, we treat it like an application server and leverage the flexibility of not requiring another microservice to handle certain tasks (in addition to using it as a proxy and static content service).

The combination of Nginx and Lua won’t replace all your use cases but by learning the system better, you can better leverage the use of Nginx across the board.

[ux] 3 Ways to Select the Perfect Method for your Research Goal

Sun, 21 Dec 2014 21:00:00 +0000

It’s exciting to see more professionals wanting to research customers to make their services and products fit better. But there’s a lot of confusion about how to choose the most appropriate research methods to suit your goals.

So what’s the best way to discover customer motivations? How do you understand how customers feel and think? How can you probe the emotional aspects of a user’s task? These are all different questions with different goals.

Here’s how you can decide what research to do in each case.

1. Avoid fixating on a particular method

Customer journey maps are hot right now. Journey maps usually result in appealing and informative diagrams, but if someone in your organization wants to do this kind of research, make sure he or she understands what kind of goal it serves.

A customer journey map is about a particular persona type having a particular experience. Particular persona type means that you match a researched-and-well-defined audience segment to a behavioral area. For example, you might choose the, “I need a pick-me-up, so I do a little retail-therapy shopping” audience segment for the “shop for fashion” behavioral area.

Another example is the, “nearly professional hobbyist photographer artist” within the “decide whether to buy new gear” behavioral area. Particular experience means a defined scope that your organization would like to improve, but which is translated to the customer’s perspective. For example, it could be the “select the right product for my situation” scope, or the “track my shipment” or “unboxing” scopes. It could even be the, “decide I need a new _____” scope.

2. Decide which research methods meet your knowledge-gathering goals

If your colleague wants to learn about the experiences of online shoppers on your ecommerce site, you need to be able to recognize what goal this represents. It’s all in the wording.

When you hear or say words that identify someone as a person using your service or product, for example “shopper”, then you know that your goal is to understand how well your solution (or idea) will support that person. You can call these goals solution-focused.

On the other hand, when you use a word that does not define the way a person interacts with your organization, for example “person” or “human,” then your goal is to understand that person’s mind and heart. Call these goals person-focused. If you can be deliberate about the wording of your goal, you can be clear about which research method to use to reach that goal.

You are also probably aware of the two goals for using the knowledge that research will bring:

evaluate how well something you make works, or
generate new improvements or new ideas.

These are called "evaluative" and "generative" research respectively. When you make a matrix from these two research usage goals and the two internal research goals above, then you can see which methods best support the union of your particular goal at the moment.

For example, the customer journey map can help you define new ideas to support the customer so it is considered generative. And because you are using the word “customer,” it is solution-focused. Sometimes professionals use the customer journey map in an evaluative manner, but that is less common.

Clarity about which methods support which of your goals makes it easier for you to know what to do. For example, Steve Portigal’s book Interviewing Users describes the solution-focused user interview. Aga Bojko’s book Eye Tracking covers this type of evaluative, solution-focused research. Indi Young’s book Practical Empathy defines how to develop empathy for generative, person-focused work.

3. Avoid pre-categorizing your results before you get them

Many UX professionals accidentally fall into this trap. They may categorize results into three stages: before purchase, during purchase and after purchase. The danger here is that thought processes and emotions do not necessarily span all three of these pre-defined categories.

This example would be better approached with a single scope from the customer’s point of view, such as “select the right product for my situation.” During a research session with a customer, inevitably the story you hear will hop back-and-forth and intertwine all of your pre-defined stages. Instead of trying to artificially separate them out, concentrate on different themes that the story brings out. For example, you could look for emotions like doubt, distrust, excitement, impatience, or behaviors such as asking friends, looking for advice, deciding based on gut, knowing from past experience, etc. These elements will end up defining your categories if you see them appear as patterns across customers.

***

So give your research goal the perfect gift this Christmas. Analyse what it is you are doing and who you are doing it for. Don’t simply choose the most popular method-du-jour, or your research goal may end up in a corner, the shiny toy that isn’t useful and is never played with.

Indi's new book Practical Empathy defines how to develop empathy for generative, person-focused work. It's on pre-order at 30% off until 08-January 2015!

[24ways] Naming Things

feeds@allinthehead.com (Paul Lloyd) — Sun, 21 Dec 2014 12:00:00 +0000

Paul Lloyd perches his partridge in the CSS pear tree to discuss naming methodologies, ontologies and semantics. What’s in a name? That which we call a cherub by any other name would smell as sweet.

[perl] Beyond Grep

Mark Fowler — Sun, 21 Dec 2014 05:00:00 +0000

Ack is a Perl based command line utility designed to replace "99% uses of grep" which is a fancy way of saying it's a much smarter version of grep designed to be used at the command line interactively rather than in scripts.

The Advantages of Ack

Whenever I get started on a big new project I need to familiarize myself with the codebase. Typical questions I tend to ask are:

Where are the Perl source files? The JavaScript? The HTML, etc?
What files are using this subroutine? Where is this method defined?

I used to use grep -r to help me with this. But I always found it was searching in the wrong places (Inside my version control files...in big log files and images rather than inside the Perl modules...) And even when it did find the thing I was looking for in files I'd end up staring at the output trying to work out exactly where the bit of the string that matched was.

Then I discovered ack. And my life became a lot easier.

At its most basic ack can be thought of a version of grep that does the right things by default. These include:

Highlighting the matched text
Automatically ignoring common version control, scratch, and temporary files.
Searching recusively in the current directory by default when a filename isn't passed and it's not being used in a pipe.
Showling clearly both filenames and line numbers of each match (and optionally column numbers)
Easily allowing restrictions to seaching particlar types of files
Using Perl's powerful regular expression engine

Or to put it another way, this grep:

`1: 2: 3: 4: 5: 6: 7:`	`grep --exclude-dir .git \ --include .pl \ --include .pm \ --include ,pod \ --include .t \ --include *.psgi \ -r -n -E '(foo\|bar)' .`

Could be better written as this ack:

`1:`	`ack --perl '(foo\|bar)'`

(And ack will also do highlighting of match and group matches within a file together in a readable form!)

Ack file types

One of the great advantages in using ack is that ability to only search files that of a particular type. In addition to the large range of built in types you can easily add a specification for your own types to ack or modify existing files by using extra command line arguments.

For example, several places I have worked have had their own template system that can contain embedded Perl code, but because these files don't use the standard Perl extensions of .pl,.pm,.pod, .t, or .psgi, nor start with a perl shebang line, ack invoked with --perl won't search them. I'd like ack to search within the .perlt files also. This can be done by adding another option to ack:

`1:`	`ack --type-add=perl:ext:perlt --perl foo`

Of course typing this every time I ran ack would be extremely tiresome. For this reason I copy this arguments into my .ackrc where ack reads them in each time ack is invoked so I always have my perlt search available.

`1: 2:`	`echo '--type-add=perl:ext:perlt' >> ~/.ackrc ack --perl foo`

We're not limited to file extensions here; For example, I can tell ack that any file that starts with { on the first line should be considered JSON:

`1:`	`ack '--type-add=json:firstlinematch:/^\s*\{/' --json foo`

Ack power use

The above should probably be enough to convince you to switch to using ack on a day to day basis for interactive file grepping. But what about some of the really crazy powerful things you can do with ack?

Find all files of a particular type

ack is pretty smart about working out what files it should ignore, including things like version control files, editor backup files, and even things like minified versions of JavaScript.

This makes it a good choice for simply getting a list of particular files of a given type in a project:

`1: 2: 3: 4: 5:`	`# find all un-minified JavaScript files in your file tree ack -f --js bootstrap/js/bootstrap.js js/main.js js/jquery.js`

(This uses -f to just print the filename that would be searched rather than actually searching the file and --js to specify that we want JavaScript extensions)

Once you've got a list of files like this you can start doing interesting things with them by using utilities like xargs:

`1: 2:`	`# delete all php code! ack -f --php --print0 \| xargs -0 rm -f`

This works the same as the previous example but uses the --print0 argument to ack and -0 argument to xargs (so that the two communicate the filenames across the pipe delimited by a null character rather than whitespace preventing problems with filenames with whitespace in their names.)

Replacing -f with -l means we will also search the files with our passed regex, and only list those files that match (but not show how we match like in traditional output.) Combining this with xargs again we can do even more powerful things:

`1: 2: 3:`	`# commit all the code that have "use strict" bash$ ack -l --perl --print0 'use strict' \| xargs -0 git add bash$ git commit -m 'add strictures'`

Using ack to highlight things in a file

ack can be used just to highlight a string in a file. For example, everywhere that $jolly is used:

`1:`	`ack --passthru -Q '$jolly' lib/Santa.pm`

The --passthru option tells ack to display all lines, matching or not. The -Q tells ack to treat the string we're highlighting as a literal pattern rather than a regexp.

We can even use this technique in a tail. For example to highlight the string "Error" in our logfile:

`1:`	`tail -n0 -f /tmp/log \| ack --passthru Error`

Custom Output

Ack allows you to modify the output from the default matching line with match highlighted output to something custom. If we want to get a list of all the subroutines in our file tree we can easily do that: We search for all the sub something but only display the something not the leading sub. By using --output we can pass a Perl string that ack will eval after each match and use the result for the output:

With a little imagination this can be used to very powerful things. For example, the size of the URLs mentioned in a file:

`1: 2: 3: 4: 5:`	`ack --output='$&: @{[ eval "use LWP::Simple; 1" && length LWP::Simple::get($&) ]} bytes' \ 'https?://\S+' list.txt http://google.com/: 19529 bytes http://metacpan.org/: 7560 bytes http://www.perladvent.org/: 5562 bytes`

However the smartest thing I've ever used the custom output for is to generate lines that look just like Perl error messages:

   ack --no-filename --no-group \
       --output 'at $filename line $line_no$/$line$/' \
       'sub new'

Why is this so helpful? Because I use the PerlErrorSublime.popclipext PopClip extension to allow me to highlight any perl-error like string in my terminal and open that line in Sublime Text:

Conclusion

Hopefully I've convinced you now not only that ack can be used as a basic replacement for grep, but also there's some very formidable things you can do with it if you take a little time to learn some of the more powerful options.

Tech Advent Calendars

[perl] So here it is Merry Christmas

Everybody's having fun

Look to the future now

It's only just begun

It's Christmas!

[java] Merry Christmas everyone!

[sysadvent] Day 25 - Windows has Configuration Management?!?

An Introduction to Desired State Configuration

A Platform Play

The Evolution of DSC

Standards Based

An Evolving API

Diving In

The Local Configuration Manager

A Quick Note - Push vs. Pull

DSC Resources

The Configuration DSL

Other Tidbits

ConfigurationData and AllNodes

DependsOn

Node Names

WMF 5 - In Production

Building an Ecosystem

PowerShellGet and OneGet

Third Parties

Chef

ScriptRock

Puppet

Aditi

PowerShell.Org

So, What Are You Waiting For?

[24ways] Cohesive UX

[perl] Out of Order Perl

Is an Asynchronous Solution right for me?

Step 1: Identify the Problem

Step 2: Identify the cause the clarify the problem (with numbers!)

Step 3: Propose a solution (based on data and facts)

What does it mean to be Asynchronous?

Asynchronous code is event-driven

Results are returned in the background while your application does other things

AnyEvent 101

condvar

send

recv

begin and end

AnyEvent::HTTP Code Walkthrough

When should I use Asynchronous Programming?

Next Improvement: Let's create a work queue!

What does Asynchronous Programming have to do with RabbitMQ?

Step 1: Replace the original “work” section with publishing a message to RabbitMQ

Step 2: Move the work to a worker process

The Gotcha

Sample Code Available

See Also

[java] A Musical Finale

What could be more fitting than Christmas music for Christmas Eve?

But first, let us celebrate the season!

You seem to like Python Alex - Why Bother With Java?

How are the two meshed?

Is this interface approach a generally good idea for Jython/Java Communication?

Wrapping It Up

[sysadvent] Day 24 - 12 days of SecDevOps

Day 1: Fear and Loathing and Risk Assessment and Hipsters

Day 2: Shared Secrets: Figure it Out Now

Day 3: Shared Secrets for Infrastructure, Too

Day 4: Config Management On All Of The Things (So You Aren’t Sweating from Shell Shocks)

Day 5: Secure your Development Environments (Because No One Else Will)

Day 6: 2-Factor all of the things (well, the important things)

Day 7: Encrypt your Emails (and other communications)

Day 8: Security Monitoring: Start Small, Plan Big

Day 9: Code/Design Reviews

Day 10: Test Your Users

Day 11: Make an Incident Response Plan Now

Day 12: Don’t be the Security ‘A**hole’

[ux] The Elf that Came in From the Cold

[24ways] Taglines and Truisms

[perl] CLDR TL;DR

What are locales?

CLDR::Number

`condvar`

`send`

`recv`

`begin` and `end`