The physical setup is much like as you might expect if you’ve ever setup a DSL modem or cable modem + a residential router. Gigaom has a nice rundown @ Google Fiber: Check out the in-home set up and equipment. Namely I wanted to focus on any stats available from where the physical fiber turned into an Ethernet port. This is known as the Fiber Jack, I have the model GFLT100 which is also known as an Optical Network Terminal or ONT.

A little searching reveals that it’s likely Google is using Gigabit-capable Passive Optical Networks (GPON). There’s a particularly interesting paper: FTTH Look Ahead - Technologies & Architectures.

Off the GFLT100 information page there’s a link to the Google Fiber Open Source Components, specifically the Open source code for the Google Fiber GFLT100 (which is what Matthew Stone notes in his article points to all the GPON source additions).

My goal is to collect some statistics from the device itself (and to understand what it’s running). The next natural thing to do is start investigating the device itself:

Unfortunately the chips themselves are covered (rather cleverly) by heat-sinks, so looking up the internals of this little device require a little more investigation…

(Thanks to Google for a pretty amazing service.)

• Videos
• Conversation archive / slide refs: Twitter feed

There were many great talks, I highly recommend catching some (or all) of the videos!

Mission Critical Innovation

Dr. Jeff Norris Gave a visually compelling story of innovation and what it means to him. This talk was more a TED style, but worth seeing particularly because of his visual story telling. It’s a bit difficult to explain, but very-much worth seeing as the augmented reality presentation style alone is technically compelling.

“Reflect with me: We launched the tallest building in Florida into space!”

Adding Static Type Checking to a Dynamic Language (Julia)

Leah Hanson gave a really interesting explanation of time and space complexity on a simple type inference in a loop as well as nice debugging tools available in the Julia language. This was a particularly enlightening take on dynamic language performance pitfalls that aren’t necessarily readily obvious in usage.

• http://julialang.org
• Notable deep-dive https://github.com/JuliaLang/julia/issues/5927

Simplifying Big Data with Apache Crunch

Micah Whitacre shared how using Apache Crunch can help with better defining MapReduce pipelines; with the goal to make them easier to write, more testable, and efficient. It’s worth taking a glance at his breakdown of an example workflow and mapping the components with Crunch.

Keeping Your Edge by Deploying Faster

Hudl shared their deployment changes that have resulted in faster deploys and greater application stability. A couple of great takeaways were: start doing some research when you’re feeling lost; this ultimately resulted in their use of many Netflix components (using them as a guide). Importance of key performance indicators (KPIs) to benchmark themselves with the changes they implemented (the largest of which was the number of deploys per day and the number of hot-fixes). Also the structure of teams that map to service oriented components, which allowed them to lean on using components both technically and culturally.

• Netflix/Hystrix - Control of failure and latency in distributed environments
  • hudl/Mjolnir
• Routing: Eureka metadata routes (spark plug, nginx; not public - yet)

Erlang, or How I Learned to Stop Worrying and Let Things Fail

Basho / Riak’s John Daily: “We talk about reusable code, but we almost never talk about rebootable code. … Failure management shouldn’t be an add-on.” Erlang is designed for failure, speed, and has very small processes. “Network transparency: the machine boundary is an ugly place… Async”

Erlang resources: https://gist.github.com/macintux/6349828

MythBashers: Adventures in Overlooked Technologies

Avdi Grimm, of Ruby Rogues and Ruby Tapas, gave an overview of diving into a Mythbusters style attack on the hypothesis: “is bash web scale?”. Obviously quite a bit of humor and somewhat of a wild-goose-chase in technology; but, the takeaway: “Outlier knowledge - comes from asking outlier questions”. Also the discovery of some new commands, for example: coproc, stdbuf, mitmproxy.

Robots, JavaScript, and Drones: Welcome to the Hardware Revolution

Julia Grace from Tindie gave a nice overview and opinions of the state of hobbyist hardware. Particularly noting: “2014 is the 50th anniversary of BASIC… There’s an analog to hardware right now”, i.e. hardware is accessible, usable, and manipulatable by the layman.

Rules as a Control Structure

Ryan Brush gave a great architecture vs. engineering analogy of blueprint:floor-plan::code:(projection) flow-chart. Noting that we can provide more “live” documentation that assists stakeholders without having to be experts in the underlying components.

Data science / Building a Production Machine Learning Infrastructure

Josh Wills shared some experiences of working with machine learning at Google and his opinions of what it takes to have an operationally useful environment. Good guiding principle: “any same thing, but faster is always good”.

Prerequisite for industrial machine learning:

• Logging
• Monitoring (to see the change in KPIs)
• Feature flags (to tune to control KPIs)

Industry examples:

• Airbnb
• Conjecture at Etsy

Themes:

• Simple key value pairs
• Operational lang vs analytical lang (e.g. SciPy vs. Java)

Time-series data with Apache Cassandra

Patrick McFadin gave an entertaining and technical overview of storing time-series data in Cassandra. Time series database requirements - Baron Schwartz (of High Perf. MySQL) - good basis for needs. Interesting tidbit: Cassandras use of TimeUUID (UUID sortable by time) - v1 UUID.

Chef / Packing It In: Images, Containers, and Config Management

Michael Goetz overviewed a workflow building VMs from scratch and using containers (with Docker) to speed up deployment and get a handle on deployed/configured environments. Obviously using Chef as the configuration management tool, but also noting the physical/time constants like always building from scratch can be very time consuming. This also introduces the concept of convergence (coming into alignment) vs congruence (rebuilding from scratch).

Hallway-track takeaways:

• Chef-zero should replace chef-solo (local mode server)
• Chef-metal will eventually have PXE boot

Mobile Ad Hoc Networks (MANET) Are Coming, But We Aren’t Ready

Justin Collins shared his views and opinions in the exploration of MANETs and what technologies we’d need to support the next “killer-app” in this space. This was an interesting introduction into technologies that are emerging and necessary, particular in mobile, computing; also notable: peer-to-peer and Delay-tolerant networking (DTN).

• https://github.com/presidentbeef/melon
• http://tools.ietf.org/html/rfc2501
• Color audience commentary: FireChat

SQLite as an Application File Format - Richard Hipp

• “Representation Is the Essence of Programming” -Frederick Brooks (Mythical Man Month)
• “SQLite is not designed to replace Oracle. It is designed to replace fopen().”
• “Data lasts longer than code.”

Side notes:

• SQLite Archiver
• Fossil SCM - very similar to Git in concept (distributed VCS), but keeps everything in one SQLite file. Also, only a single binary that supports built-in wiki, issue tracking, and basic GUI functions. Ultimately it allows for some some nice semantics like: SELECT DISTINCT name FROM users; (rather than shell ginsu).

Slackware ragchew - David Cantrell

(http://www.slackware.com, http://www.burdell.org)

• Still alive and an impressive South-East Linux sub-culture.
• Fascinating history and interaction during the Linux “boom” (many trade shows!) with Sun, FreeBSD, and ultimately Walnut Creek CDROM (later BDSi, then Wind River Systems, then iXsystems)
• Slackware was the first “live boot” (see also zipslack)
• Version control regrets – would have been great to see the evolution, but it’s lost.
• Suse - dev’s mail spool file made it onto a distro release (wild west of builds and Linux)
• As a community we could do better… packaging is a great example (every distro has their own)
• Hallway track: awesome history of tz database (note: this was needed for his own distro as glibc? stopped pulling in the latest for static builds, which required distros to manage the dependency) - “Time (and timezones) are both technical and political.”

PBI v10: Application Management Made Easy - Ken Moore

AppCafe

• Wiki as data-source for rating system
• runs pkgng through a series of shell scripts and parses the output for the GUI utilities (Qt)

PCBSD

• Modified GRUB to support full disk encryption with ZFS
• Life Preserver takes a snapshot prior to upgrade so that in the instance of a reboot and fail you can rollback.
• Errata: Ken Moore is Kris Moore’s brother, BSDNow.tv

ZFS 101 - Dru Lavigne

• Becoming a ZFS Ninja
• Open ZFS office hours
• Encryption: GELI, PEFS
• Other
  • ZFS (on Linux) - use your disks in best possible ways

Other

• GPG/CACert KeySigning
• Ham radio meetup
  • Freedv
  • Fldigi
    • Digital modes - sights & sounds
  • N8VNR
• http://agrimmsreality.blogspot.com
• Other talks
  • Making Pointers Point - Andrew Psaltis (Basic introduction to pointers in C and memory allocation)
  • Trolls arent the only threat under the bridge
  • DNSSEC
  • Hands on ARMs - Jason Plum
    • Open Blocks (4GB ethernet device)

Slides

(Note: the post as follows is identical to the slides)

The Web: HTTP/1.0

• Client <– –> Server
• GET, PUT, POST, DELETE, HEAD, …
• Headers

Auth experience

• HTTP Basic Auth - IETF RFC 1945
  • Simple!
  • Plaintext (SSL can “help”)
  • Relys on client/browser authentication popup

Example with Basic Auth:

curl -u "username:password" https://www.example.com/special

Auth experience

• Session Auth (cookies) IETF RFC 6265
  • “free” with most web frameworks
  • A session is usually tied to one user account and is normally authorized fully or not.

Example with cookies holding state through a session:

curl -c cookiejar.txt https://www.example.com/login
curl -b cookiejar.txt https://www.example.com/special

Auth experience

• OAuth IETF RFC 5849
  • “Valet key for the web” (limited access OAuth Token)
  • Scope access to resources

Example of a signed OAuth request:

<special signing sauce>
curl -v -H 'Authorization: OAuth
 oauth_consumer_key="zsQpwbL3AGRNV4272Xc8Msi3hxhQWGrS",
 oauth_signature_method="HMAC-SHA1",
 oauth_timestamp="1346887460",
 oauth_nonce="1548267549",
 oauth_version="1.0",
 oauth_token="wvokahqtGMLS5o4AvVvokGZaA9pZjBcW",
 oauth_signature="tvHRw2fLNxYE2FR62EfH6tAfBW4%3D"'
https://www.example.com/special

So what’s so special about OAuth?

So what’s so special about OAuth?

• Each client is assigned credentials granted by a resource owner to access a protected resource on a server
  • Note: Client <– (stuff) –> Server
  • Therefore the credentials granted to a specific client can be managed/revoked by the resource owner.
• An endpoint(s)/HTTP resource(s) can be scoped (to limit its functionality)
  • “I give you (Ms. client) access to my API, but read only.”
  • “Access all of my public data (Mr. client)”
  • “(Mrs. Client) Is requesting access you your bank account, allow?”

Halt!

What about that guy on the internet (Eran Hammer) that was like “OAuth 2.0 sucks” !!rage quit!! ?!!

Agree or disgree … most importantly: he’s an expert and is prosthelytizing great information - Listen to him! (and read carefully)

The takeaway: dont throw the baby out with the bathwater and his commentary is directed at the 2.0 draft

Let’s Dance

What OAuth looks like

What OAuth looks like: Protocol workflow

What OAuth looks like: Protocol workflow

What OAuth looks like: Protocol workflow

What OAuth looks like: Protocol workflow

Technical pieces

Terminology

• Consumer: client
• Service Provider: server
• User: resource owner
• Consumer Key and Secret: client credentials
• Request Token and Secret: temporary credentials
• Access Token and Secret: token credentials

URL pattern(s)

(Related to protocol workflow)

• https://provider.example.net/{initiate,request_token} (Temporary Credential Request)
• https://provider.example.net/authorize (Resource Owner Authorization URI)
• https://provider.example.net/{token,access_token} (Token Request UR)
• http://consumer.example.com/{oauth_redirect,ready,…}

(Ref: URL patterns for Twitter, AllPlayers.com)

Refs

• http://tools.ietf.org/html/rfc1945#section-10.16 / http://en.wikipedia.org/wiki/Basic_access_authentication
• http://tools.ietf.org/html/rfc6265 / http://en.wikipedia.org/wiki/HTTP_cookie
• http://tools.ietf.org/html/rfc5849 / http://en.wikipedia.org/wiki/OAuth
• OAuth Checklist
• Build out and scratch pad notes
• Public Key Cryptography: Diffie-Hellman Key Exchange

Enter CloudInit + Puppet Git Receiver + Puppi

Puppet Git Receiver

I believe the most “magical” aspect of any of the PaaS offerings is the ability to push code+infrastructure changes (IMO the best being with git) to a remote server. The Puppet git reciever has to be one of the cleanest and well thought out implementations of what is essentially a fancy post-recieve git hook.

What makes this so great, besides the really clean and well thought out code to deploy reasonably complex servers with Puppet, is the simple deb package install that basically configures a git repo, with hook scripts, and configured users.

CloudInit

CloudInit is a nice tool that makes this whole process quite seamless. Basically CloudInit allows you to specify a set of scripts (and Ubuntu specific commands) to run at the VM creation; effectively letting you bootstrap a machine exactly the way you want it. (This is a little additional help that traditionally lives as process step to login to a new machine and “get it going”)

At this point with CloudInit + Puppet git reciever - you can now have a “blank” server ready to recieve whatever infrastructure defined by Puppet. This leaves your application code, which is addressed by … (read on)

Puppi

I’m curious if this Puppet module should actually be called Puppet-Deploy, because that’s what it does! (along with some additional sysadmin helper tools too) By specifying a few simple instructions this module configures and stages everything for your app to run on our accompanying infrastructure.

class myapp {
  puppi::project::git { "myapp":
    source      => "https://github.com/../myapp.git",
    deploy_root => "/srv/myapp",
  }
}

Simply run: puppi deploy myapp on the target server (noteworthy: this can be scripted a variety of ways: with Puppet, via SSH, etc…).

But wait! There’s more!

Given that we’re already building out a server and deploying our app with super-simple-commands, why not make it easier? Yeah lets do that! Check out librarian-puppet a tool to manage Puppet module dependencies similar to how Bundler, Composer, and the like. This let’s us specify a Puppetfile in our infrastructure repo that defines all the “common” modules we will be using - making a very clear the line between libraries and the “working set” (and avoiding those pesky submodules). Bonus points to letting the post recieve hooks auto build the Puppetfile! This can lead to some pretty simple infrastructure+app definitions:

├── Puppetfile
├── README.md
├── manifests
│   └── site.yml
└── modules
    ├── imapnote
    │   └── manifests
    │       └── init.pp
    └── mysites
        └── manifests
            └── init.pp

(Brought to you by: Labor Day Weekend Research Group™)

Did some testing with drush tonight and have some tips:

http://drupal.org/project/drush project has nice pear install info: (note: may need to sudo)

pear channel-discover pear.drush.org
pear install drush/drush-5.0.0 # note: there may be some instability, but seems ok

(note: drush-5.0.0 has the --gitinfofile which injects version into .info)

Drush bash completion is probably a handy tool for sbox CLI: http://drupalcode.org/project/drush.git/blob/refs/heads/master:/drush.complete.sh namely: sourcing the bash completion script e.g.: . /usr/share/php/drush/drush.complete.sh (this provides tab completion for drush commands)

Drush5 provides an injection to the .info file; an example use of this would be: drush dl --package-handler=git_drupalorg --gitinfofile uuid (note: the --gitinfofile parameter)

(running .info inject by default) Using the drushrc.php (a file to specify specific specific drush options in a config file...) we can ensure that gitinfofile is enabled by default: $command_specific['pm-download'] = array('gitinfofile' => TRUE); (note: unfortunately drush_make doesn't respect the drush dl paradigm ... drush 5 this will be the preferred method) (note: to get drush to continue reading my rc file I had to do: drush -c ~/.drush/drushrc.php make example.make www) (tip: to see what is happening with the actual drush commands use debug/verbose: drush --debug --verbose -c ~/.drush/drushrc.php make example.make www)

We can also have specific drushrc.php settings per repo to be fancy: http://drupalcode.org/project/drush.git/blob/refs/heads/master:/examples/example.drushrc.php#l8 and/or http://drupalcode.org/project/drush.git/blob/refs/heads/master:/examples/example.drushrc.php#l332 (note: I believe this to be a nice way - and the drush recommended way - to organize our alias' and other drush specific settings.) (note: to accomplish this I believe minimal setup through Puppet could get things in place - this would be for deployed env. - everything else would have a Readme)

<(soon) deprecated>

For drush_make .info inject... (or running drush 4.5)

cd ~/.drush/drush_make/
# http://drupal.org/node/1302820
wget http://drupal.org/files/drush_make_inject_info.patch
patch -p1 < drush_make_inject_info.patch

</deprecated>

Intro

Lately at AllPlayers.com I have been spending a lot of time working on API’s for our site and spending some time making Drupal talk web. Luckily, this is something that is being thought about a lot in the community (see: WSCCI, services.module); but, there are some interesting growing pains:

The connection to Drupal: There are a variety of ways to connect to external sites and obviously “the way the web should work”, but I get the explicit feeling that many of types of connections are “toys”. As a site administrator we built out a DLAMP (Drupal, Linux, Apache, MySQL, PHP) install because we are serious about building a web-app! In our engineering efforts, we are probably only slightly interested in handing over our data to 3rd party services, so likely our next interest is in taking on the data for ourselves (which is no small matter).

Get the data

So, we’ve committed ourselves to “big data”; great - how do we do it?! Honestly - look towards to the industry leaders; short of a major engineering effort (which perhaps you should be looking at hadoop, hbase, and map-reduce), we are here for Drupal - and that’s gonna make this fast and powerful: feeds! Target an endpoint, upload a file, or configure the plugin an do what you want, then parse that data with a processor (e.g. XPath, QueryPath, CSV mapping, or build your own). It’s a little like magic! (if you don’t believe me, maybe a demo could convince you. Now let’s get a little more technical…)

Oh the pain…

You’ve got the data but, was that all there was to it? Mostly… but, because we are good researchers we note something about feeds requests: they’re pretty good, but there turns out to be a variety of different ways to do essentially the same thing:

• drupal_http_request
• World Bank API (depends on drupal_http_request)
• Acquia Connector which uses XML-RPC (depends on drupal_http_request)
• Pantheon System API

Some observations: the Drupal way seems to be drupal_http_request but, there is also a distinct lack of PEAR dependencies like HTTP_Request; a nice place to ruminate.

Expose your data

Data acquired! But we want to share (easy): fields and views. However, as a respecting netizen you want to share semantically! Drupal already comes feature packed for the web, but we also want to share through API’s. Luckily contrib space has this “solved” with services, where we can expose our data through a variety of servers and varieties of formats.

Thus we can do something like this: GET http://drupal6-services/services/plist/node/1.json and expect

{
  "changed": "1286592762",
  "language": "",
  "uid": "0",
  "type": "story",
  "status": "0",
  "nid": "1",
  "created": "1286592762"
  ...
}

amazingly powerful! (it is also worth noting that the services endpoint can pull from views as well!)

Oh the pain…

The story so far has gotten us everything we need very quickly, but as consumers of many public APIs we quickly note that what we are leaking is Drupal through exposing endpoints. This is actually fairly useful to developers of Drupal, but is probably pretty useless to someone consuming your product, therefore we need to take configure (or theme) the exposed services.

Additionally, given a glimpse of web services future of Drupal this seems particularly pertinent! WSCCI (Web Services Context Core Initiative) is the holding place for the high performance REST server.

Data and the front-end interaction

I probably don’t share very much, but I have a love affair with javascript; and given the amazing HTML5 features and great javascript libraries (jQuery is just the tip of the iceberg - or the first taste depending on how you look at it) - there is a bright future in the browser! So bright in fact: I am of the opinion that “web-pages” should be delivered as apps! There are a lot of great tools to facilitate the concept of a client side app, often examples of this are seen in the Chrome web store (some neat technologies include the cache-manifest).

This leaves me wondering: where exactly does the theme layer live? Particularly, as in a traditional sense, the theme layer seems to live “in the browser”, which means that any processing done server side to render CSS, JS, and/or HTML would be in the delivery of an application - the version of which the client is requesting to be delivered by the server.

sent via mobile

Caveat Emptor: Its probably very short sighted to assume that this is a one sided conclusion, but I think there are some great reasons to follow this pattern; there are many great reasons to have a fantastic UI, but when thinking about reaching the maximum audience it is comforting to know that following this pattern can help development remain agile in the face of new device releases (see: iPad). ]]> 2009-02-08T00:00:00+00:00 https://imetchrischris.com/what-we-have-to-look-forward-to-64-bits-and-b The emergence of the 64-bit architecture effectively increases the memory ceiling to 264 addresses, equivalent to approximately 17.2 billion gigabytes, 16.8 million terabytes, or 16 exabytes of RAM. To put this in perspective, in the days when 4 MB of main memory was commonplace, the maximum memory ceiling of 232 addresses was about 1,000 times larger than typical memory configurations. Today, when over 2 GB of main memory is common, the ceiling of 264 addresses is about ten trillion times larger, i.e., ten billion times more headroom than the 232 case.

Test run...
]]>