Justin Brodley Weblog http://www.justinbrodley.com SaaS, Web 2.0, Social Enterprise, Virtualization, Cloud Computing posterous.com Wed, 23 Jan 2013 12:00:00 -0800 Book Review: The Phoenix Project http://www.justinbrodley.com/book-review-the-phoenix-project http://www.justinbrodley.com/book-review-the-phoenix-project

Just finished reading Gene Kim's, Kevin Behr, and George Spafford's new book, The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.  Gene Kim is well known for the The Visible Ops Handbook: Implementing ITIL in 4 Practical and Auditable Steps, which came out during the early days of the ITIL movement, and I know it was my first real deep dive into change management. The book is written in a similar style to the 5 dysfunctions of a team which is a "leadership fable".  

The Pheonix Product follows a fictional auto parts suppy store called parts unlimited, it opens with Bill the main character being promoted by their CEO with little guidance to the VP of IT Operations after the CIO and CTO are fired and that there is a critical payroll issue that must be fixed ASAP.  Bill's prior experience at Parts Unlimited is running IT Operations for the "Midrange Server Group", but he has no oversite of the distributed or helpdesk operations of the parts unlimited which are now under his perview.

Several IT Charicatures are represented in the book, your typical Process Guru/ITIL person, your overconfident/arrogant IT Manager, Obstructive IT Security team, etc.  While character development isn't a strong suit in this book, I was easily able to see links to people that i've worked with in the past.   

The first half of the book consists of a ton of outages and problems caused by this completely mismanaged IT organization. Familiar statements like "IT is in the way, IT screws up all the time, etc" are all represented.  They finally start getting on the right track when a new board advisor starts coaching bill on identifying the types of work in the IT shop, and relating it to factory floor operations. This drives the team into implementing change management, Kanban methodologies for workflow, and eventually continuous deployment with even a few mentions of being "allspawed" which i didn't realize had become a verb. But Kudos to John Allspaw over at etsy, you've crossed over to the otherside. 

Overall the book is quite good especially for teams that haven't embraced ITIL, Lean Manufacturing and Devops into their culture and business processes. For me I really liked the setup of the book (the first two chapters I read after Velocity last year when Gene published them as an early preview) are excellent. I felt a lot of unnecessary time was setup in showing how piss poor the operations were and not enough time in the solutions. I would have hoped for more detail on Kanban processes, ITIL and Devops practices, but instead they were regulated to a single chapter and the complexities of setting up this infrastructure was a bit glossed over.  

There is one character in the book called Brent, that I actually found to be quite unannoying, he is the "know it all" IT guy characture, and of course all changes, and major sev 1 issues always require Brent to be involved to get the issue fixed. At numerous points the management team puts in processes to elevate him as their "most critical resource" and to limit the work coming into his area of expertise. While this is good, and you should follow similar processes to make sure the constrained resources in your group have a clear work allignment and goal.  He may not have been malicous in his retention of tribal knowledge that only he knew, but several times I would have leaned towards firing him. Ultimately I never felt that he was a team player, was protecting his base of knowlege as he liked being the Hero.  Maybe i'm alone in this, and with the limited character development i shouldn't get hung up on it, but its probably the one piece of the book that I felt was counter-culture to the devops movement. 

Overall, I liked the book, i'll be sending it to some friends as gifts in the future as there is a lot of good stuff in there. But i don't know if it will become a favorite of mine as the Visible ops handbook is. I'm excited to see the next collaboration from Gene Kim and his team at ITRevolution the devops cookbook. I think what I was hoping to get out of The Pheonix Project will end up in the cookbook, and than this may be the perfect pairing! Rating B+

Gene Kim and his coauthors are all excellent people to be following on twitter and their respective blogs.

IT Revolution Website http://www.itrevolution.com

Gene Kim's site: http://www.realgenekim.me/  or twitter @realgenekim

Kevin Behr's site: http://www.kevinbehr.com/

George Spafford twitter: @gspaff

 

 

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/974119/NBP-6713.jpg http://posterous.com/users/3tpAMG6ZMmjv Justin Brodley Justin Justin Brodley Wed, 02 Jan 2013 08:37:00 -0800 Netflix/Amazon Outage on December 24th http://www.justinbrodley.com/netflixamazon-outage-on-december-24th http://www.justinbrodley.com/netflixamazon-outage-on-december-24th

Note:  I know I haven't been consistent with the blogging this last year or so, its been crazy busy for me professionally and personally. Lots of exciting things going on, plus I relocated to the San Francisco Bay area for a fantastic job opportunity. Launched a small consulting business last year, to help out a few friends and my former employer after I left.  Overall though, i'm hoping to at least blog weekly on some topic either personal, tech related, or just of overall interest.  

Netflix Outage on Christmas Eve

I'm shocked at how much press attention the Netflix outage on Christmas Eve has gathered. Its not like this is the first time that either Amazon or Netflix has had an outage, nor will it be the last sadly.  I think the large part of the scorn is that it hit Netflix at an unfortunate time when a lot of their users actually wanted to use the service for Christmas specials, holiday traditions, etc. 

Most interesting of this is the blame that Amazon is taking for this outage and while they did cause the initial issue, the issue is Netflix's to resolve and prevent future outages.  (Amazon Blog Post on outage: http://aws.amazon.com/message/680587/)  The cloud is fantastic, lowered the barrier to entry for a ton of startups, and provided Amazon scale to companies who wanted to focus on the software not the infrastructure. This power though comes with greater complexity and business challenges that must be addressed. (Spidermans: "With great power comes great responsibility" comes to mind).

When you rely on equipment, services, providers that you ultimately have little control over you must plan for failure.  You must assume that any component that you rely on will be gone at any moment, or perform at a suboptimal level and there is nothing that you can do for these scenarios.  At my last employer we hit several of the "amazon oops" moments.  First we had our application deployed in only one region/Availibility zone.  This is the same thing has running your all of your servers and systems in one datacenter, supported by a single provider/telco/utility/etc.  Its a huge Single Point of Failure (SPOF).

Next we moved to single region/ multiple avaialibility zones, while this is a nice improvement it still bit us when an Amazon Technician made a human error that killed EBS and Storage in the East Region.  Suddenly we realized that while Amazon advertises that each Availability Zone is agnostic from each other on a hardware level, the control tier and shared services back end could and does get shared across multiple AZ's (Actually in reality some of the control tier is global in nature from some of the recent RCA's).  

It of course took code changes and infrastructure enhancements for us to tackle multiple AZ's in a single region. Some of the things you may need to do in your applicaiton are:

  1. Read/Write Master/Slave Replication strategy (NoSQL & Relational all have varying ways to accomplish this depending on your software) 
  2. Traffic load balancing between each AZ for incoming user traffic (if active/active)
  3. Application awareness of databases and databases states.  Some of this is handled by the drivers for the database (ie: Mongo), although implementations are a mixed bag and must be heavily tested

The above is an additional cost in complexity, testing, load testing, network design and thought to how the system is developed and deployed.  Once you realize that this still isn't good enough you start talking about multiple regions and/or cloud resilient (amazon & rackspace, etc).  This adds new complexities that you now must factor in:

  1. Global load balancing now becomes a must, or enhanced round robin DNS services
  2. Increased latency between sites, syncronous commits become damaging to performance rapidly
  3. Application complexity increases 3x with 2 Regions/Clouds, but expect 4-5x increase in complexity if you add more
  4. Active monitoring and diagnoses of issues must be detected by monitoring and nodes/systems isolated as the number of users impacted could be small or large, or worse impossible to detect

I give lots of Kudos to Netflix for the Chaos monkeys, not a lot of people have the stomach to have a "rogue" agent in their systems breaking stuff on perfect and testing their resiliency. But as more and more companies move to the cloud the practice must become more common, at least in the lab environments. (http://techblog.netflix.com/2011/07/netflix-simian-army.html). 

Global scale once the area of tech giants (Yahoo, Google, Microsoft, Amazon) are available to the masses. Of course finding the tech talent who has dealt with scale at this level is difficult and/or their pretty happy at the companies they work for.  The Devops community is a huge help in this area, with folks sharing their infrastructure, war stories, solutions for scale, and of course a relentless pursuit if metrics and automation that allows the complexity of this scale to become manageable. 

Check back for future posts on devops culture, hiring, global scale, etc!!  Plus you guys can keep me honest on posting at least weekly.

Interested in the Netflix/Amazon outages check out these blogs:

Amazon Post Mortem

Adrian Cockcroft's (Netflix) analysis of issue

 

 

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/974119/NBP-6713.jpg http://posterous.com/users/3tpAMG6ZMmjv Justin Brodley Justin Justin Brodley Wed, 01 Aug 2012 08:55:40 -0700 New blog post - Amazon EBS Provisioned IO http://www.justinbrodley.com/new-blog-post-amazon-ebs-provisioned-io http://www.justinbrodley.com/new-blog-post-amazon-ebs-provisioned-io
Check out my latest blog post over at brodleygroup.com.

http://www.brodleygroup.com/1/post/2012/08/amazon-launches-provisioned-iops.html

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/974119/NBP-6713.jpg http://posterous.com/users/3tpAMG6ZMmjv Justin Brodley Justin Justin Brodley Mon, 23 Jul 2012 12:25:02 -0700 Password Reset Mechanisms http://www.justinbrodley.com/password-reset-mechanisms http://www.justinbrodley.com/password-reset-mechanisms Great write up on password reset mechanisms. I learned quite a few things… Anyone who is designing these types of systems should take a few minutes to read this.

http://www.troyhunt.com/2012/05/everything-you-ever-wanted-to-know.html?HN2

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/974119/NBP-6713.jpg http://posterous.com/users/3tpAMG6ZMmjv Justin Brodley Justin Justin Brodley Thu, 05 Jul 2012 12:33:32 -0700 Checkout YBUY! http://www.justinbrodley.com/checkout-ybuy http://www.justinbrodley.com/checkout-ybuy
For a tech gadget junkie like me this is the perfect service. I was always jealous of the Bag, Borrow or steal system for Purses and other high end luxury items for woman.  Glad someone is finally bringing this to tech gadgets. 

If interested follow the link below:
Here's my link: http://www.ybuy.com/?ref=aZp8v

Thanks.

In case you're interested, YBUY is a membership club that lets you try the greatest products in the world, risk-free! Here's how YBUY works:

1. Get products on the 1st of the month. YBUY ships you a product so that it arrives by the first of every month.

2. Try products for 30 days. After the trial period, you can return the product at no cost, or you can buy it - your call.

3. Skip a month at any time. We won't charge you and you'll still be a member of this exclusive club.

If you want to show some love, you can "like" YBUY on Facebook - http://www.facebook.com/pages/ybuy/232115610153403

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/974119/NBP-6713.jpg http://posterous.com/users/3tpAMG6ZMmjv Justin Brodley Justin Justin Brodley Fri, 08 Jun 2012 09:24:17 -0700 Amazon Instance Types Chart http://www.justinbrodley.com/amazon-instance-types-chart http://www.justinbrodley.com/amazon-instance-types-chart So glad that someone published a simple table of all Amazon Instance types and configurations.  Makes it much simpler to see all of the data in one place, and i'm surprised Amazon hadn't already done this. 

Check it out, and you can commit changes to Github. I guess i can throw away my google spreadsheet i've been using. 

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/974119/NBP-6713.jpg http://posterous.com/users/3tpAMG6ZMmjv Justin Brodley Justin Justin Brodley Wed, 16 May 2012 09:07:00 -0700 Justin Brodley @ Google+ http://www.justinbrodley.com/justin-brodley-google http://www.justinbrodley.com/justin-brodley-google
Day 3 at Daptiv. Things are going well, starting to figure things out.
via plus.google.com

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/974119/NBP-6713.jpg http://posterous.com/users/3tpAMG6ZMmjv Justin Brodley Justin Justin Brodley Wed, 29 Feb 2012 10:02:00 -0800 Code deployment http://www.justinbrodley.com/code-deployment http://www.justinbrodley.com/code-deployment

As most Ops guys know Code deployment can be a challenge, especially when the code relase is dropping 100's or 1000's of changes to the product into production in one large code push.  While I'm a huge fan of Devops and continuous deployments, realistically a lot of organizations are still deploying code the old fashioned way.  Nate over at TechOpsGuys recently ranted about organizations that his company relies on pushing production code on friday and breaking his companies website (http://www.techopsguys.com/2012/02/03/dont-push-code-on-a-friday-damnit/).  This rant brings up a few things i've been thinking about for a long time and felt needs to be expressed more clearly in a blog post:

1. Friday Code Deployments or Large code pushes are going the way of the dinosaur, more frequent smaller code releases with feature flags, continual integration testing, etc are becoming the norm.

2. If your software product is relying on services via API's for your product to function or provide availability you need to start thinking differently.

Code Deployment in Devops

With the push for Devop's in SAAS and B2C Business you are seeing more and more companies going to continous integration and continous deployment. In fact some software companies won't even show a new developer where the bathroom is until they push code into production. (http://www.scottporad.com/2010/11/01/cheezburger-network-doesnt-show-its-new-employees-the-bathroom-until-theyve-checked-in-code/)

Having been involved in code deployment for over 5 years in a SAAS environment, I much prefer the smaller code deployment without taking major downtime of a site or service.  Rolling in features bit by bit and turning them on when their fully deployed or turning them on to a select set of beta users makes it so much easier to test features, functions, etc with real production load and know how these systems operate in production. Plus if you find an issue once you've deployed the code you can roll back to the previous code by redeploying from source or fix it and push the code out.

Relying on Third Party Services

Once you understand that code from a lot of startups is being deployed all the time (Etsy pushed 10,000 code changes in 2011 (http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/). You need to start thinking about how you integrate with these services differently. If you main home page pulls in a twitter status updates from your CEO you need to make sure the following happens:

1. When the API is functioning, display the data (Yeah i know Duh)

2. When the API is not functioning that the following happens:

A. Your website still loads and doesn't throw a nasty 500 error

B. Your website doesn't have an obnoxious hole with a 404 or 500 error in the sidebar or header of the site.

This means your developers must be planning for the inevitable situation that the API is going to break, the way you call the API will change, or plagues of locusts have infested their datacenters.  Your developers need to focus on thinking about all states of the service and the behavior they want their application to exhibit when the failure occurs. 

 

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/974119/NBP-6713.jpg http://posterous.com/users/3tpAMG6ZMmjv Justin Brodley Justin Justin Brodley Tue, 27 Dec 2011 11:01:35 -0800 True SaaS and why you should care http://www.justinbrodley.com/true-saas-and-why-you-should-care http://www.justinbrodley.com/true-saas-and-why-you-should-care
Great article on True SaaS (multitenant, single code line, hosted software) and why enterprises should care about it.  Its a common practice for software vendors with highly successful products to just assume that it can "be SaaS" if we just repackage the solution as hosted.  While this may look "SAAS" and your customers may even like the appearance of a higher level of data security, it has huge trade offs in terms of an enterprises need to manage the solution.  Customers of these type of solutions are tied to higher per user costs, they have to do their own upgrade coordination and scheduling, normally with high up front PS costs, and they suffer less than stellar quality and performance as they scale the solution.  If your in the market for a SaaS solution you should be asking about the delivery model, if your not your doing yourself a disservice.

http://www.enterpriseirregulars.com/44507/what%E2%80%99s-true-saas-and-why-the-hell-should-customers-care/ 

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/974119/NBP-6713.jpg http://posterous.com/users/3tpAMG6ZMmjv Justin Brodley Justin Justin Brodley Wed, 02 Nov 2011 14:17:15 -0700 Google Design Asthetic http://www.justinbrodley.com/google-design-asthetic http://www.justinbrodley.com/google-design-asthetic I used to think Microsoft had terrible design aesthetic for their applications, but I have to say the new Google designs are downright HORRIFIC.  They first attacked google reader this week with this grey on grey background, white space between everything, and an ok integration into Google Plus.  Today they allowed you to switch to the new Gmail interface... which is just as bad as the new google reader color pallet, yep grey on grey.  I seriously don't understand the design process at Google unless its completed by color blind engineers.

Come on Google!! Hire away some UI people from Apple, or hell even Microsoft and fix this!!

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/974119/NBP-6713.jpg http://posterous.com/users/3tpAMG6ZMmjv Justin Brodley Justin Justin Brodley Mon, 12 Sep 2011 15:02:44 -0700 Why!! http://www.justinbrodley.com/why http://www.justinbrodley.com/why Why has Posterous pivoted into a social network!! UGH... I don't need another Google Circles, Facebook or Twitter!! I just want to post stuff to you, point my domain at you, and have you distribute my content everywhere my friends live on the internet!! 

Not happy right now!! Will give it a few days and see if I can cope with this change.

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/974119/NBP-6713.jpg http://posterous.com/users/3tpAMG6ZMmjv Justin Brodley Justin Justin Brodley Sun, 14 Aug 2011 20:08:13 -0700 What? http://www.justinbrodley.com/what http://www.justinbrodley.com/what
Photo

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/974119/NBP-6713.jpg http://posterous.com/users/3tpAMG6ZMmjv Justin Brodley Justin Justin Brodley Thu, 14 Jul 2011 10:59:17 -0700 Why not Vmware? http://www.justinbrodley.com/why-not-vmware http://www.justinbrodley.com/why-not-vmware In the late 90's early 2000's there was a large push by major enterprises to modernize their ERP, CRM, HR and associated mission critical business applications.  Oracle began acquiring enterprise software vendors like Peoplesoft, BEA and Seibel.  If your interested in their acquisitions you can read about them here.  In talking to my colleagues in Oracle shops they started the Mantra "why not Oracle" when new business requirements came up.  A few years later with Virtualization gaining popularity I began hearing "why not VMware?" when talking about new server requirements and needs.  This drive for virtualization rapidly dropped costs, allowed companies to be more flexible and eventually resulted in the new "cloud" age that we are in today. 

I'm a huge fan of VMWare, i've promoted it to my colleagues, used it to save my employers money and ultimately built more agile IT teams to deal with changing business requirements.  While i'm a huge fan, some of my colleagues are not pointing out the high price of VMWare licensing, additional complexity, and the 15-20% performance hit between Virtualized servers and bare metal.  Having been an ESX user since 2.5 I've been a long time supporter, unfortunately with the announcement of Vsphere 5 I have to now reevaluate my entire thought process.

Vsphere 5 isn't the first time that VMware has alienated their customers, the first was "Enterprise-Plus" that was introduced with Vsphere 4.0. Enterprise customers had traditionally been buying Enterprise licenses for ESX after being told that it was the "premier license" that included all of the features of ESX.  Vsphere 4 introduced Enterprise-Plus now going against the messaging from VMWare sales.  Luckily while there were some compelling features in Enterprise Plus (vswitch, standardized host config, etc) and larger memory configurations 99% of VMWare customers didn't find these as critical features they must have. 

Now with the introduction of Vsphere 5 they have added to their socket licensing model vRAM allocations.  I understand the rational for this the major push in Virtualization shops is larger servers, with more cores per socket and lots of ram.  This allows you to reduce your VMWare server needs and increase your consolidation ratios.  With customers buying less ESX Servers this put VMwares cash cow at risk and they needed to take action.  My issue is with their vRam allocations per ESX version, for Enterprise-Plus their "premier" license you only get 48gb of Allocated memory in your license.  A server with 96gb of ram only costs around 16k from Dell right now, meaning that your VMWare licenses are now over 70% of your hardware cost! This is stupid, and is going to cause nothing but pain for customers and increase costs in a time when were being ever squeezed to reduce costs and provide more.  

So, VMware has taken a page from Oracles book, get your customers invested or buy your competitors, and than jack up your rates holding them hostage to you unless you want to go to great lengths to migrate off your existing solution! VMWare is the new Oracle, and shame on them for screwing their loyal customers and advocates. 

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/974119/NBP-6713.jpg http://posterous.com/users/3tpAMG6ZMmjv Justin Brodley Justin Justin Brodley Sat, 18 Jun 2011 11:57:26 -0700 California is exhausting!! http://www.justinbrodley.com/california-is-exhausting http://www.justinbrodley.com/california-is-exhausting
Media_httpimagesinsta_bpijb

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/974119/NBP-6713.jpg http://posterous.com/users/3tpAMG6ZMmjv Justin Brodley Justin Justin Brodley Sun, 15 May 2011 20:43:00 -0700 Moxie competes in enterprises with Yammer and Salesforce Chatter with a better design http://www.justinbrodley.com/moxie-competes-in-enterprises-with-yammer-and http://www.justinbrodley.com/moxie-competes-in-enterprises-with-yammer-and
via youtube.com

Great interview with Robert Scoble and the CEO of Moxie Software (my employer).

If your curious about what were doing at Moxie its a great video.

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/974119/NBP-6713.jpg http://posterous.com/users/3tpAMG6ZMmjv Justin Brodley Justin Justin Brodley Fri, 29 Apr 2011 19:53:32 -0700 Rainbow outside house http://www.justinbrodley.com/rainbow-outside-house http://www.justinbrodley.com/rainbow-outside-house
Media_httpimagesinsta_fxtnk

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/974119/NBP-6713.jpg http://posterous.com/users/3tpAMG6ZMmjv Justin Brodley Justin Justin Brodley Wed, 23 Mar 2011 10:02:00 -0700 Dilbert's take on work life balance http://www.justinbrodley.com/dilberts-take-on-work-life-balance http://www.justinbrodley.com/dilberts-take-on-work-life-balance
Media_httpdilbertcomd_ektgn
via dilbert.com

Sometimes I swear Scott Adams and I work for the same companies.

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/974119/NBP-6713.jpg http://posterous.com/users/3tpAMG6ZMmjv Justin Brodley Justin Justin Brodley Mon, 07 Mar 2011 12:44:51 -0800 Facebook getting old? http://www.justinbrodley.com/facebook-getting-old http://www.justinbrodley.com/facebook-getting-old Recently i've found that I no longer visit facebook as often as I used too. Now that i'm not compelled to tend to my farm, fish, city, starship, etc (I quit all the social games cold turkey back in July), I just don't find as much stuff that I find interesting. Facebook is my friends and former colleagues I wanted to  stay connected too, and a few of my Wife's friends husbands and friends that we've become friends with as couples.  Facebook was fun at first to catch up with what was going on with them, but it seems like they are even posting less and less.  I've been thinking about why this is happening and have a few theories:

1. Value/Reward - Posting updates and photos on Facebook is not as rewarding as it used to be. Sure you may get some likes, but you don't get any feedback and most of the time when you do get feedback it something negative and discouraging or a counter opinion putting you immediately on the defensive.
2. Trying to keep track of the ever increasing privacy issues on Facebook is exhausting and confusing, and people are tired of dealing with it. 
3. They get more value out of other websites that have like minded individuals (website for your knitting club, computer club, book club, etc) that have more value to you and put you in touch with like minded people who can add value. 
4. Honestly if were really friends we talk outside of facebook and I already know about what your posting... having a debate over a political issue is more rewarding in person and you don't hurt peoples feelings, etc. The need for human feedback is better met with face to face, phone to phone or some other interactive method like IM than in posts on facebook.

So what am I doing with my time now that i'm not using facebook as much.

1. Well a new baby and a 22 month old take a up a ton of my personal time (and do you really want to know that i just changed the 5th poopy diaper for Corbin for the day?? No not reallly.
2. Work. My company makes an awesome product called Spaces, that we use for corporate collaboration and social, I keep track of it throughout my day for information on prospective sales, HR updates, and projects my team is working on. 
3. Posterous - Why just post on facebook, when i can post on Twitter, facebook, disqus and a dozen other social avenues and feed most of the comments and discussion back to one central place. 
4. Twitter - If you want breaking news this is the best place, faster then any traditional media company. 
5. Google Reader - Keep up on the blogs and twitter feeds you can't watch all day and night, but are interested in following.

Is anyone else finding Facebook less and less valuable than it used to be?

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/974119/NBP-6713.jpg http://posterous.com/users/3tpAMG6ZMmjv Justin Brodley Justin Justin Brodley Wed, 09 Feb 2011 11:47:00 -0800 Colby James Brodley http://www.justinbrodley.com/colby-james-brodley http://www.justinbrodley.com/colby-james-brodley

After being stubborn for the last two weeks, Colby has decided to finally join us! 

Colby James Brodley
Born February 9th 2011
Weight: 7 Pounds, 4 Ounces
Length: 19.5 Inches

Mommy and Baby are doing fine, and dad is doing alright too! 

Dsc_0071 Dsc_0062 Dsc_0064 Dsc_0067 Dsc_0068 Dsc_0065
See the full gallery on Posterous

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/974119/NBP-6713.jpg http://posterous.com/users/3tpAMG6ZMmjv Justin Brodley Justin Justin Brodley Fri, 04 Feb 2011 22:30:00 -0800 Article: How does Etsy manage development and operations? « Code as Craft http://www.justinbrodley.com/how-does-etsy-manage-development-and-operatio http://www.justinbrodley.com/how-does-etsy-manage-development-and-operatio

Great Article, I have a ton of respect for John Allspaw and the Developers at Etsy and Flickr.  -- Justin

How does Etsy manage development and operations?

Posted by chaddickerson | Filed under operations, people, philosophy

I’ve been loving using Quora these past few months, and have been amazed at the level of behind-the-scenes detail people are providing about really complex and specific things (like how Facebook does automated testing).

Recently, someone asked, “How does Etsy manage development and operations?” with these comments:  Etsy seems to have scaled far and fast, whilst continuing to add new features; how is all this managed – is there a strictly-defined process within which engineers operate, or is it a case of hiring clever people and letting them get on with it (Facebook-style)?

First of all, I love the team and am proud of the work that they do.  It’s an amazing group and none of this would work or be as fun as it is without them.

So, here’s the answer I just posted:

In 2010, we did grow the engineering team pretty fast, going from 20 to about 70, and the rest of the company grew quickly, too.  As we grew, overall speed has been really important to us, and we’ve continually tuned our processes, tools, and culture to support that.  I wrote about some of these principles behind all of it in my blog over the summer: http://bit.ly/foHPc1

Right now, developers are divided up into a number of small teams, usually 3-7 engineers.  These teams are paired with a product manager and a designer, and there is some movement across teams as needed.  All designers at Etsy code and product managers code at various levels, too. Ops and dev work really closely together, and we have one development team that is very ops-like and straddles both domains. Everyone in the company uses IRC.  Lots of ideas are worked out on a wiki, and people around the company comment on those ideas and plans (we use Confluence).  Some projects form organically, and others are more top-down.

We generally plan in 60-day chunks and divide the deliverables up into 2-week periods (though we’re not officially using capital-A Agile).  The 60-day cycle has no special significance — we just felt like it was a reasonable timeframe for planning near-term deliverables.  The 60-day plans go through a review, we set goals, and we publish the plans on the wiki.  Our founder, CEO, and head of product (Rob Kalin) participates in these reviews and stays in close contact with the product and engineering teams throughout.  In general, the teams have a lot of autonomy in how they get their work done within a set of architectural principles we’ve established (a subject for another post) and our overall design approach.  Specs are typically very light, and the focus is on building working features.

We onboard engineers quickly and their first goal is simple: deploy on your first day.  The goal here is to constantly emphasize shipping, and get over any deployment fears early.  Engineers get productive very quickly.  The level of cooperation between developers and ops is also really high (see our engineering blog for more: http://etsy.me/hMtu1A)

We practice continuous deployment and make small changes frequently to the site.  We use what we call “config flags,” which are more or less an exact copy of what Flickr does (see the Flickr engineering blog: http://bit.ly/dZZzfY) and a lot of the code for features runs “dark” for days or weeks, and feature launches mean flipping a switch in the code.  We have a lot of Flickr DNA in the company (John Allspaw, our VP of Ops, ran ops at Flickr, and Kellan Elliott-McCrea was architect at Flickr).  In January (a month in which we did over a billion page views), code committed by 76 unique individuals was deployed to production by 63 different folks a total of 517 times.   Product managers make changes and do deploys (here’s Jenn Vargas, one of our newest product managers, tweeting about it) and we have trained aspiring developers on our support team to make small changes with our help and guidance, too.  Our deployment environment requires a lot of trust, transparency, communication, coordination, and discipline across the team.  We’ve invested a lot in our automated unit and functional testing (we have a team devoted just to this), tooling for deployment (see our blog post about Deployinator: http://etsy.me/c6RJD7), and metrics and monitoring (see “Tracking Every Release”: http://etsy.me/e1ULhO).   Key system-level and business level metrics (like checkout/listing/registration/sign-in rates)  are projected on screens in the office and we have a number of internal dashboards that the team uses (we mainly use Ganglia and Graphite).  We also have lots of switches and knobs to help us roll features out to percentages of users and ramp them up slowly, or quickly.  Features are used and tested by us here at Etsy for some period of time before they are rolled out publicly.

When we make mistakes, we conduct blameless post-mortems and assign remediation items to the appropriate team members.  Engineers frequently post in our community forums when we have any issues and we have a status blog that we maintain (http://www.etsystatus.com/).  I think that interacting with Etsy members gives everyone a deeper sense of responsibility for the code we’re writing.  We also write about the mistakes we make pretty openly (http://etsy.me/hgZ4qh).

Overall, engineers are treated as creative collaborators in the overall process with design and product, and products are worked out and iterated on with engineers instead of simply being handed to them for implementation.   Rob (our founder and head of product) likes working with engineers and the engineers spend a lot of time interacting with Rob.  Our ability to work this way has as much to do with the personalities of the people involved and the culture as the technologies involved. We’re always learning and adjusting and we’ll continue to evolve as time goes on.

 

-->

No responses yet. You could be the first!

Leave a Response

via codeascraft.etsy.com

 

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/974119/NBP-6713.jpg http://posterous.com/users/3tpAMG6ZMmjv Justin Brodley Justin Justin Brodley