Arnon's shared items in Google Reader

Jeff Xiong: Move To Multi-threaded Rails

2009-07-17T02:11:05Z

When doing performance in local box, we found that requests are waiting to be processed due to Rails is single-threaded by default. We changed the configuration to enable multi-threaded mode as described in this blog entry:

Thread safety for your Rails

(also refer to Rails 2.2 release notes )

The most important change was to eager-load /lib directory, and thus move libraries for test out of it.

config.threadsafe! also disables automatic loading by ActiveSupport::Dependencies. Alternatively, you can just add lib/ directory to eager load paths. The following inside production.rb will do that :
config.eager_load_paths << "#{RAILS_ROOT}/lib"

We had some rough tests to ensure the functionality is not broken. Multi-threaded mode also might require more memory than single-threaded mode. Still keep testing.

Lean Reading List

2009-07-17T08:10:00Z

I read Mary and Tom Poppendieck's first book on Lean Software Development, "Lean Software Development: An Agile Toolkit" in 2005. It went in one eye and out the other.

I was the Software Development track chair for Austin's InnoTech conference in November of that year. Mary was gracious enough to accept our invitation to come to Austin to keynote the track and to moderate a panel.

I listened to Mary's talk and got a few more clarifying tid bits from it, but mostly, I dismissed it at the time as some form agile sideshow that didn't quite measure up to the specifics that XP brought to the table. A number of us went out for burgers with Mary after the conference and continued the conversation, but I don't think anyone in the group was fundamentally moved by Lean.

I wasn't equipped with the experience to see Lean for what it was. Later, I realized the deep hubris that encumbered my thinking and the vanity that would lead me to expect that I could intuit a subject as vast as Lean from a single book, the way I could with XP and Scrum.

I suffered a serious set-back in 2007. My career's pinnacle dream project was flirting with disaster.

The company I worked for faced an intractable intellectual property constraint that limited our product's market to a small fraction of the whole. In November of 2006, I sold the company on an ambitious plan to solve the problem by building our own platform that we would have unlimited rights to sell. We started exploratory work and envisioning in December. The undertaking had significant executive sponsorship. When our board of directors voted to not fund the project, the CEO bought out the board and gave us the green light. We started official work in earnest in January.

In November of 2007, I sounded the failure warning alarm to my management, and I kept sounding it. In January of 2008, with the situation becoming increasingly intractable, I parted ways with the project and the company. Three months later, after having given the team a chance to pull itself together, the project was canceled and the team was dismissed from the company - from the most junior technologist, up to the executive overseeing the project.

I was frustrated by having felt handcuffed by Agile development orthodoxies that no longer fit the problems of the team, and yet were followed mechanically, and reinforced by management that was captivated by its first exposure to Agile.

I told some of the details of my experiences to a friend. We talked about Agile, in many variants, including Lean Software Development. My friend recommended that I read about the Theory of Constraints and recommended "The Goal: A Process of Ongoing Improvement" by Eliyahu Goldratt and Jeff Cox. In reading The Goal, I recognized the detailed mechanical process that I faced in the failed project. At that time, I hadn't connected what I had begin to learn in The Goal with what I know about Agile Development and the gaps in Agile that I had begun to see after seven years of immersion.

Following The Goal, I wanted to read more into some of Lean's roots. I had largely avoided the Toyota literature up till that point, and I wasn't convinced that I would get much from The Toyota Production System. I knew a number of people in my community who had read The Machine that Changed the World and The Toyota Production System, but I never really got the sense that the reading had connected them with the transformative learning that I had experienced in Agile development.

Before committing to throwing myself into the Toyota literature, I wanted a sample of what I might be getting into. I started with an article. I read "Decoding the DNA of the Toyota Production System" by Steven J. Spear and H. Kent Bowen published in the Harvard Business Review.

In this article, the authors talk about the things that companies seemed to miss when trying to duplicate Toyota's successes using Toyota's methods. I'm a sucker for stories of unasked questions. The rest of my reading about Toyota and Lean in general would be an exploration of the unasked question. The authors' message: Toyota's fundamental nature as a learning organization is often overlooked, with undue attention paid to Toyota's more obvious practices and mechanics.

And this is what drew me in to the Toyota literature. Here I saw the parallels to my failed organization, my own nature as a learner, a seeker, a pathfinder, and a teacher. I recognized the many of the intractable problems that I had observed in the behavior of my failed organization.

On my team, it had become an unspoken entitlement conferred that people were not required to take direction. This was largely exasperated by a management approach that was fixated on social experimentation and which was not capable of guiding the technical execution or product design imperatives of the project. It was an Agile methods laboratory that produced no real acceptable features for a year.

I learned in Spear's and Bowen's article that the organizational structure, mechanics, and protocols that I felt would benefit the team were the basis of Toyota's organization and culture as I was coming to understand them.

The Toyota Way by Jeffrey Liker was my first read into the Toyota literature. I chose this book to specifically continue learning about the Toyota DNA rather than dig into Toyota's specific process mechanics. Understanding that focusing on the process mechanics led to common problems in learning and adopting Toyota's methods, I wanted to hold off on the obvious aspects longer.

I read the Toyota Way with the observations of the Harvard Business Review article fresh in mind, as well as the mind-opening lessons about work management and problem solving from The Goal providing a backdrop.

Cautious of becoming yet another Toyota disciple, I took a turn away from Toyota-specific literature and back toward Lean in general and read "Lean Thinking: Banish Waste and Create Wealth in Your Corporation" by James P. Womack and Daniel T. Jones. This book tells the story of the companies and people taught by Womack and Jones as they traveled around the world after writing The Machine that Changed the World. It reinforced and what I had learned about learning culture and continuous improvement as well as organizational structure and process in the books that preceded it.

While this reading and studying was happening, I was also tweeting about my experiences and studies, and having numerous long conversations on the phone with some of my peers who had themselves started studying the same material. I also revisited my failed project with the previous product owner, who was still with the company, and still succeeding in his own work.

If I had read this material in a vacuum, without any of the constant interaction with my professional network, and the continual revisitation of previous failure, I believe it would have been a much less informative and transformative experience. My circle of friends and network of colleagues continues to inform my learning, and I expect that it will continue to do so.

I turned the reading back to software, choosing to read Mary and Tom Poppendieck's second book, "Implementing Lean Software Development: From Concept to Cash". Reading a Lean software book from my vantage at that time was a very different experience than when I had read the first Lean software book. The subject had now come to life. It was tangible, and it was deeper than what I had presumed previously. From here, my perspective of Lean Software Development takes meaning beyond my perspective of XP and Agile culture and mechanics.

I had organized the ALT.NET Open Space Conference the previous year, and hadn't want to simply fall into the trap of trying to duplicate an original moment. I wanted another theme for the second annual conference. Over the course of many of those conversations with friends and colleagues, we talked about the essential force of the ALT.NET movement as something akin to Lean's Continuous Improvement. The theme of the second conference became Continuous Improvement, with hopes that we would come to understand it more, and maybe understand better whether ALT.NET is indeed a Continuous Improvement culture.

I subsequently read another Toyota book, "Extreme Toyota: Radical Contradictions That Drive Success at the World's Best Manufacturer" by Emi Osono, Norihiko Shimizu, Hirotaka Takeuchi. This book takes on seeming contradictions in Toyota's culture and organization, such as the simultaneity of both a flat organization and a rigid hierarchical organization and the imperatives of a learning culture and continuous improvement that unify the two. The book also spoke about the climate of contradiction that Toyota uses to stimulate creativity and problem solving.

Tom and Mary accepted our invitation to come to the Continuous Improvement Conference in Austin and share their experience with learning organizations, software development, product development, scientific method, and leadership. I can't imagine that Mary and Tom got as much out of the experience as we did, but they had a lasting impact on our community.

Mary and Tom often say that Lean Software Development is informed more by the Toyota Product Development System than the Toyota Production System. Dave Laribee got to the Toyota Product Development book before I did and warned me that it was quite dry but worth reading, and it was. "The Toyota Product Development System: Integrating People, Process And Technology" by James M. Morgan and Jeffery K. Liker spoke a great deal more about requirements, design, planning for work, and creating workspaces, as well as the risk-mitigation processes that Toyota uses. This book also goes into greater detail about the Toyota's people and roles, and fostering "towering technical competence".

The TPDS book also talks about Lean leadership and the Chief Engineer role. It further reinforces the notion of managers and group leaders as having great technical competence, often knowing the work of their staff better than they do, and being paramountly responsible for teaching their staff though the scientific method and Toyota's A3 report technique.

Dave read David Anderson's book, "Agile Management for Software Engineering: Applying the Theory of Constraints for Business Results" while I was reading the TPDS book, and I followed the TPDS book with David Anderson's book. I wish I would have read this book years ago, but again my biases obstructed my perception of the value I would get from it. In this book, David Anderson ties the Theory of Constraints directly to software development and translates pull systems and throughput accounting to the work.

Dave turned me on to Cory Ladas' writing on the Lean Software Engineering blog. I read a few articles, and then went back to the start of the blog and read forward. Much of that writing has been compiled into his book "Scrumban - Essays on Kanban Systems for Lean Software Development". Cory’s writing goes quite deep into pull systems and Kanban for software development. This material and David Anderson’s book offer a profound exploration of applying the mechanical aspects of Lean.

I put into practice what I learned both from my own experience and from studying and study groups on a project with a distributed team in Austin starting in August 2008. My instinct at first was to overlay their existing organization and process with Scrum. Instead, I looked for signs of what was keeping the organization back, established some basic measurements, and did my best to teach what I knew about how to solve these problems. It wasn't trivial work, but it reaffirmed my experience and study of both the cultural and behavioral aspects of Lean, as well as the mechanics, such as Kanban. The experience also continued to reaffirm XP practices as the tactical foundation of both Lean and Agile strategies.

While in Austin, Mary and Tom mentioned the book "Managing to Learn: Using the A3 Management Process to Solve Problems, Gain Agreement, Mentor, and Lead" by John Shook, which tells the story of the techniques and processes used in learning culture and learning organization, and goes much deeper into the the teacher/student relationship and responsibilities that are the foundation of Lean.

I'm presently reading this book. It's an innovative book not only for its content but also for its form, and is well worth the read, and practice.

Context

I wrote this article because people in my network asked me for some recommendations on books on Lean. The more I thought about it, the more I thought that listing some titles and links might not be entirely responsible.

The books that I read and the order in which I read them is inseparable from the context in which I read them. This isn't a canonical list and it shouldn't be treated that way. There are many, many more resources available.

I've benefited tremendously from the choices I've made for study and for engaging community to enliven that study. I wholeheartedly recommend the books I've talked about here, and I would even recommend going through them in the order that I read them. In retrospect, the reading order worked well as an evolutionary thread and helped me reinforce a deeper understanding of subtler aspects of Lean while continuing to layer on ever broader knowledge.

Nonetheless, your mileage may vary, and if your experiences are quite different from those I've laid out here, then you might disregard my own learning adventure and concoct your own.

Either way, foster a community to learn with, and sit at the feet of as many masters who'll tolerate your presence. Anything you learn from a book is just material until you light it up with experience (or reflection) and turn it into knowledge. Learning in isolation rarely has the yield of learning enlivened by experience and community. That's not always the case, but if you have a tendency to hide in a cave, understand that much of Lean is a social practice.

I still haven't read The Machine that Changed the World or The Toyota production System. I may read them at some point, but my goal isn't to consume every Toyota book that I can find. My goal is to synthesize as much understanding as I can, and the past two years have been very rewarding in this regard.

I recently founded the Lean Software Austin group, and I'm looking forward to continuing the study and the work in Lean principles and software development as this community grows as a learning organization itself. The story doesn't end here, but the narrative reading list does (for now).

For your convenience, here is an actual list of the reading I referenced in this article:

Lean Software Development, "Lean Software Development: An Agile Toolkit by Mary and Tom Poppendieck
The Goal: A Process of Ongoing Improvement by Eliyahu Goldratt and Jeff Cox
Decoding the DNA of the Toyota Production System by Steven J. Spear and H. Kent Bowen (Harvard Business Review)
The Toyota Way by Jeffrey Liker
Lean Thinking: Banish Waste and Create Wealth in Your Corporation by James P. Womack and Daniel T. Jones
Implementing Lean Software Development: From Concept to Cash by Mary and Tom Poppendieck
Extreme Toyota: Radical Contradictions That Drive Success at the World's Best Manufacturer by Emi Osono, Norihiko Shimizu, Hirotaka Takeuchi
The Toyota Product Development System: Integrating People, Process And Technology by James M. Morgan and Jeffery K. Liker
Agile Management for Software Engineering: Applying the Theory of Constraints for Business Results by David Anderson
Lean Software Engineering by Cory Ladas
Scrumban - Essays on Kanban Systems for Lean Software Development by Cory Ladas
Managing to Learn: Using the A3 Management Process to Solve Problems, Gain Agreement, Mentor, and Lead by John Shook

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Lessons from The Golden Hammer and The Mule

2009-07-17T16:40:00Z

'I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.' (Abraham Maslow, The Psychology of Science, 1966). Most college freshmen learn about the hammer. And we certainly know all about the hammer in the IT industry. By 1998 this worst-practice had become so common in IT circles, in fact, that the book 'AntiPatterns: Refactoring Software, Architectures, and Projects in Crisis' immortalized this anti-pattern with the happy title of 'the golden hammer'.

Recently, Ross Mason, the CTO of open-source SOA tools creator MuleSource, wrote a great blog about the golden hammer problem in the world of the ESB. We feel his pain. As an emerging technology vendor ourselves, we are very eager to understand and share the best uses of enterprise mashup technology.

You have a fairly good use-case for enterprise mashups if you have a dynamic information environment that requires the real-time assembly of disparate data silos in a user-specific context for critical decision-making. But that's a bit of a mouthful. So what do enterprise mashups mean in business terms? I like the answer I got from a CTO at BlueCross BlueShield, the medical insurance giant: mashups '...create valuable and actionable information'. Recent examples we've seen include solutions for Situational Awareness, budget analysis, Recovery and Reinvestment Act reporting, clinical research, data packaging, and executive KPI dashboards.

Interestingly, we are also occasionally asked why can't an ESB can't do the job of an Enterprise Mashup Platform. ESBs and Enterprise Mashup Platform ('EMP', for short) have some similar qualities but they differ in one fundamental way: ESBs are decidely 'A-to-A' (application-to-application) and EMPs are 'A-to-U' (application-to-user).

ESBs emphasize features and qualities that support robust application-to-application integration and just don't have any true user-facing, collaborative capabilities. And I think Ross agrees: 'Mule and other ESBs offer real value in scenarios where there are at least a few integration points or at least 3 applications to integrate. They are also well suited to scenarios where loose coupling, scalability and robustness are required.'

In contrast, enterprise mashups are as much about viewing, rating, personalizing and sharing as they are about data assembly. Here at JackBe we think about mashlets, porlets, spreadsheets, and iPhone apps as much as we think about data-source virtualization, mashup transformations and standardized mashup interfaces. To paraphrase one of my favorite Gartner analysts, an enterprise mashup platform without meaningful capabilities to deliver the mashups to consuming users is simply not an enterprise mashup platform.

What this really means is that ESBs like MuleSource and EMPs like JackBe's Presto compliment each other very well. #5 on Ross's 'ESB Selection Criteria' list even says 'Do you need to publish services for consumption by other applications? This is a good fit for Mule as it provides a robust and scalable service container...'. And that's exactly where an EMP comes in! One of JackBe's best customers uses ESB-based web services instrumented with Web Services Notification to send data a number consuming applications, including Presto. Presto mashes the inbound data and delivers the mashup via mashlet to a user-facing portal in real-time.

As my 9th-grade shop-class teacher used to say: 'A good carpenter knows his tools'. Makes me wonder if Maslow was a friend of his.

The Texas Dependency Injection Massacre

2009-07-17T23:55:40Z

Since I think I’ve beaten the “I think what most people call architecture is really engineering” meme to death, let’s move on to something else. Eric Smith of The Limber Lambda blog (love that name!) commented:

I'm a little concerned with the intimation that use of interfaces, respect for visibility of type members and use of dependency injection equates to "over-engineering". As with everything, it depends on what you're trying to achieve, and generalisations in this regard, especially when junior people who may not understand what's at stake are reading, can be damaging.

I find it an uphill battle to engender a constructive mindset in developers who have established bad habits and whose pride lies in the way of addressing those habits.

Anti-"process" talk by Joel Spolsky and the "pragmatism brigade" makes it harder. A while ago I had a new developer refuse to write unit tests despite it being an established practice in our team because "... Jeff and Joel said they were bad in the StackOverflow podcast ...". Yikes.

Let me be very clear. I never suggested that techniques such as interfaces and dependency injection are over engineering. These are good engineering practices, and every software engineer should understand them. And if Joel and Jeff really said unit tests were bad, well that would be about the dumbest thing I’d have every heard either of those two say. Yikes indeed.

But as Eric writes, “it depends on what you're trying to achieve”. Engineering techniques like dependency injection, polymorphism, encapsulation are tools, and there are many good reasons to use them. But like many tools, they can also be used for evil.

In other words, the tools themselves are always innocent – you have to look at how and why they are being used by the people who are using them.

Let’s take dependency injection as an example. Externalizing a software component’s dependencies enables you to test it isolation from the rest of your system. For example, it’s very common to inject a dependency that writes to a durable store, such as a logger or a data access component. In your unit tests, you inject a mock durable store instead of the real dependency. The mock will be faster (no need to actually write to disk), cleaner (no need to clean up the files on disk between test runs) and will behave exactly to the spec (bugs in the dependency component won’t create false failures in the component you’re testing). Those are all good engineering arguments for using DI, full stop.

Furthermore, DI helps insulate a software component against changes in its dependencies. I may not be able to predict specific changes with any precision, but it’s probably safe to assume that there a given component’s dependencies aren’t going to remain completely static. DI doesn’t insulate you 100% from possible changes – in particular, it doesn’t help if the dependency’s interface changes.

But I would argue that you can go too far with DI. Let’s go back to the logger component example I described above. Maybe, the over engineer thinks, we’ll want the logger to write to the database instead of the file system in the future. Or maybe we’ll want the logger to write to a different database. And if it’s supporting a different database, then maybe the logger should support different back end databases. Or maybe, Or Maybe, OR MAYBE..

We’ve gone from a simple component that logs to the file system and turned it into a engineering monstrosity with multiple points of variability and extensibility. When you start saying “maybe we should” or “this could change in the future” or stuff like that, that’s when you start over engineering something.

Unfortunately, there’s only one way to know when you’ve started over-engineering: Experience. Sorry Eric, I can’t help you with your junior engineers. As David Lee Roth once sang, Experience is the “worst teacher goin’”. But if there’s a better way to learn, I don’t know it. In the meantime, I suggest code reviews and pair programming.

Tuning ConnectionBufferSize

2009-07-17T12:00:00Z

Poor network utilization for large TCP data transfers is often a symptom of an overly small ConnectionBufferSize. The ConnectionBufferSize is the size of the send and receive buffers used by the connection oriented transports, and in particular the TCP transport where the default size is 8 KB.

If all of the following factors are present, then you might want to investigate whether tuning the connection buffer size will improve the performance of your application:

Using the TCP transport in a binding
Bottleneck is in data transfer rather than data computation
Regularly transferring large messages or experiencing high connection latency
Network utilization is noticeably lower than file copies or other data transfers between the same two machines

On the other hand, the following factors might discourage you from increasing the connection buffer size:

Memory pressure
Interactive applications

I’ve previously recommended increasing the ConnectionBufferSize up to 64 KB once the network speed exceeds 100 Mbps and keeping the default for slower networks. I’ve found recently though several deployments where performance has benefited from bumping up the connection buffer size more aggressively. The exact optimal values for a deployment depend on many factors besides the bandwidth, such as the frequency of dropped packets, transmission latency, and the presence of routers or bridges.

I’m now considering recommended ranges of 8 KB-32 KB for connection up to 100 Mbps and 32 KB-256 KB for faster connections. If you’ve tried tuning the ConnectionBufferSize of a WCF application and seen optimal values significantly outside these ranges, I’d be interested in hearing from you.

CMMI Version 1.3 Plan Update

2009-07-18T03:52:29Z

CMMI Version 1.3 will include updates to the models in all three constellations: CMMI for Development (CMMI-DEV), Acquisition (CMMI-ACQ), and Services (CMMI-SVC), as well as updates to the Standard CMMI Appraisal Method for Process Improvement (SCAMPI) method. These updates will focus on high maturity, more effective generic practices, appraisal efficiency, and consistency and commonality across the three constellations.

Coping with Bugs on an Agile/Scrum Project

2009-07-16T02:39:00Z

An often asked question is how does Scrum recommend a team to handle bugs? Should they be placed on the product backlog? Or on a separate bug list? If they’re on the backlog, does the Product Owner get to set their priority or are they automatically the most important items? Should there be a separate bug fixing sprint? By Mark Levison

Hybrid2 public bike concept promises to help power city buses

2009-07-16T06:38:00Z

Hybrid bikes are one thing, but designer Chiyi Chen looks to have something far grander in mind for his Hybrid2 bike concept, which he says could one day help power fleets of city buses. To do that, the hybrid part of the bike (a regenerative braking system) wouldn't be used to help power the bike itself at all, but would instead store the energy in an ultracapacitor that'd then feed the energy back into the grid when its parked at a special bike stand, which would in turn be used to help charge the hybrid electric buses. Not one to overlook an ingenious little detail, Chen has also devised a special card RFID card that would not only be used to unlock the bike, but keep track of the energy that each rider generates -- build up enough credits and you can ride the bus for free. Intrigued? Head on past the break for a video overview from the man himself.

[Via Inhabitat]

Continue reading Hybrid2 public bike concept promises to help power city buses

Filed under: Transportation

Hybrid2 public bike concept promises to help power city buses originally appeared on Engadget on Thu, 16 Jul 2009 01:38:00 EST. Please see our terms for use of feeds.

Read | Permalink | Email this | Comments

Don’t Choose – Combine Your Practices

2009-07-16T13:22:21Z

I have been “preaching” about why you should combine software engineering management practices for a long time (almost all of my presentations end with this as a call to action). A lot of people get stuck in the religion of their current practice (or the practice that they see in the industry that holds the most promise). Today Scrum is hot. Kanban techniques are also gaining traction – especially in the blog/twitter spheres. I’ve been a big proponent of Feature Driven Development in the past – and there are a barrage of other techniques stemming from eXtreme Programming, Test Driven Development, Behavior Driven Development … the list goes on.

These days, I have to admit I’m very happy to see more emphasis on Lean techniques, such as Kanban systems, in the management of software engineering teams. Don’t get trapped in the Kanban religion though…it’s too easy. I’m also a big fan of something I’ve called “Feature Driven Scrumban Development” – which demonstrates the merging of a few different mindsets. I actually used to call it “eXtremely Lead Feature Driven Scrum Based Development” – however, that was just too much of a mouthful ;-) I made it up to prove a point – what we need to be “without form” – similar to the end goal of a practitioner of martial arts – when it comes to applicability of team and project management techniques.

If you’re interested in some of these concepts go out and read this book:

Scrumban - Essays on Kanban Systems for Lean Software Development

http://www.amazon.com/Scrumban-Essays-Systems-Software-Development/dp/0578002140

The book is easier to read if you are already familiar with Scrum and FDD and even a lot of Lean concepts…

Putting Max on the Back Burner

2009-07-17T00:54:50Z

This note is to inform JUnit Max subscribers that I will not be actively developing Max further (extraordinary events notwithstanding). Customers wishing to cancel their subscriptions are welcome to do so, as long as they stop using Max. Anyone wanting a refund will be granted one, no questions asked. Subscribers are welcome to continue using Max for the duration of their subscription, but no further subscriptions or renewals will be accepted.

Reasons

Max addresses a genuine need: dedicated TDDers really do need better, faster feedback. Max’s approach of risk-first test execution and an embedded UI modeled on syntax error reporting successfully deliver that feedback. There is an audience who can effectively use what Max delivers.

The business problem with Max is that there just aren’t enough of us who rely on tests minute-by-minute. Even allowing for my marketing and sales mistakes, the data suggests that there are at most a few thousand Java programmers actively practicing TDD. That’s not a large enough market to sustain a business, especially as the market turned out to be price sensitive. Even if every Max subscriber successfully convinced 100 of their friends to sign up, there wouldn’t have been enough revenue.

And so, much as I love the concept of better, faster test feedback and extending the benefits of developer testing, I will henceforth be investing my time in other products and in my consulting clients. I am not closing the door to future business–if someone comes along with a 10,000 seat order or offers to buy Max in order to donate it to the Eclipse foundation, I’ll be glad to discuss the possibilities.

Lessons

I learned a lot developing, marketing, and selling Max. One thing I learned is that I love being involved in the entire business. There were days in the middle where I felt fully engaged, fully alive. I’d like more of that in my life.

Here are some more specific lessons I learned:

Release early and often. The phrase, “If you aren’t embarrassed, you waited too long to release,” was very helpful to me. My first release was the tiniest sliver of what I imagined, but it gave me market validation (even if Erik and Saff were the only two people to buy it) and, more importantly, energy. The rapid release cycle let me experiment with features that I hoped would drive sales (none did) and fix blocking defects.
Ship it and fix it. The product needs to provide value from the first, but it doesn’t need to provide everyone with value all the time. Early sales answer questions about the market size. Early users accept some rough edges if they get to be first and you fix the problems. Not reflexively fixing every defect was a hard transition for me, but, often, answering the next business question was tmy highest priority. I would recommend installing real-time remote error reporting for anyone bringing client software to market. It was nerve-wracking at first to see all the errors, but in the end the feedback was invaluable.
End-to-end. I wish I had implemented a tiny sliver of the server functionality earlier. My ambition for Max was always larger than what I delivered, but implementing something of the service offering, however small, would have communicated that vision more clearly. Given the market size problem, it wouldn’t have made a difference in the end, though.
Cut your losses. The exponential revenue growth turned down in May (negative second derivative). June was worse. It was six weeks from the first signals that the market was saturated to today.
Lean startup. The most helpful advice I got came from Eric Ries and his Lean Startup ideas. Balancing time between the problem team and the solution team is a challenge for me, as I like making things.

Until We Meet Again

This is a sad kind of post to write. I had a vision of programming for my living, making tools to help make the work of other programmers go more smoothly. That’s not going to happen, not now with this product.

I’d like to thank all the Max subscribers for their feedback and forebearance with a young and rapidly evolving product. Ilja Preuss, David Gageot, and Eric Lefevre-Ardant were particularly generous with their time. Most of all I’d like to thank David Saff for telling me exactly what he thought of Max and for giving up some of his precious JUnit programming time listening to me vent.

It’s never easy to give up one dream and move to the next. My problem monitoring the behavior of Max in the field turns out to be common, and there are few good general solutions available. I’ll soon be launching a new product for developers of Eclipse-based products to deliver real-time feedback about errors. We’ll see if I get enough revenue to get the wheels off the ground of this one. If you have an Eclipse-based application deployed and want to find out what’s going on “out there”, drop me a line.

Navigating the Myriad SOA Standards and Specifications

2009-07-16T17:58:52Z

Blogger: Anne Thomas Manes

This just in:

"Members of OASIS, OMG, and The Open Group announce the publication of a new white paper, 'Navigating the SOA Open Standards Landscape Around Architecture.' The paper was produced to help the SOA community at large navigate the myriad of overlapping technical products produced by these organizations with specific emphasis on the “A” in SOA; i.e., Architecture.

This joint white paper explains and positions standards for SOA reference models, ontologies, reference architectures, maturity models, modeling languages, and governance. It outlines where the works are similar, highlights the strengths of each body of work, and touches on how the work can be used together in complementary ways. It is also meant as a guide to users of these specifications for selecting the technical products most appropriate for their needs, consistent with where they are today and where they plan to head on their SOA journeys."

Effective Concurrency: The Power of “In Progress”

2009-07-15T23:08:59Z

This month’s Effective Concurrency column, The Power of “In Progress”, is now live on DDJ’s website.

From the article:

Don’t let a long-running operation take hostages. When some work that takes a long time to complete holds exclusive access to one or more popular shared resources, such as a thread or a mutex that controls access to some popular shared data, it can block other work that needs the same resource. Forcing that other work to wait its turn hurts both throughput and responsiveness.

…

Let’s look at the question from another angle, suggested by my colleague Terry Crowley: Instead of viewing partially-updated state with in-progress work as an ‘unfortunate’ special case to remember and recover from, what if we simply embrace it as the normal case? …

I hope you enjoy it. Finally, here are links to previous Effective Concurrency columns:

The Pillars of Concurrency (Aug 2007)

How Much Scalability Do You Have or Need? (Sep 2007)

Use Critical Sections (Preferably Locks) to Eliminate Races (Oct 2007)

Apply Critical Sections Consistently (Nov 2007)

Avoid Calling Unknown Code While Inside a Critical Section (Dec 2007)

Use Lock Hierarchies to Avoid Deadlock (Jan 2008)

Break Amdahl’s Law! (Feb 2008)

Going Superlinear (Mar 2008)

Super Linearity and the Bigger Machine (Apr 2008)

Interrupt Politely (May 2008)

Maximize Locality, Minimize Contention (Jun 2008)

Choose Concurrency-Friendly Data Structures (Jul 2008)

The Many Faces of Deadlock (Aug 2008)

Lock-Free Code: A False Sense of Security (Sep 2008)

Writing Lock-Free Code: A Corrected Queue (Oct 2008)

Writing a Generalized Concurrent Queue (Nov 2008)

Understanding Parallel Performance (Dec 2008)

Measuring Parallel Performance: Optimizing a Concurrent Queue (Jan 2009)

volatile vs. volatile (Feb 2009)

Sharing Is the Root of All Contention (Mar 2009)

Use Threads Correctly = Isolation + Asynchronous Messages (Apr 2009)

Use Thread Pools Correctly: Keep Tasks Short and Nonblocking (Apr 2009)

Eliminate False Sharing (May 2009)

Break Up and Interleave Work to Keep Threads Responsive (Jun 2009)

The Power of “In Progress” (Jul 2009)

Posted in Concurrency, Software Development

Key Person Won't Participate

2009-07-15T17:13:33Z

Unable to gain the participation of a key person for a project retrospective?

A retrospective, sometime called a post-mortem or lessons learned session, is an event designed to discover and preserve the aspects of a project that were managed well; and discover and change the aspects that weren't. I consider this event a fundamental feedback mechanism for a learning organization.

My friend Mitch is a case in point. He won't show up for a retrospective.

Mitch builds business software infrastructure. For more than a decade, he has been doing this job for the same IT organization. The company that employs him is huge. Its business is complex. Its executives are demanding. Its systems must scale. But the investments in business software infrastructure consistently fail. And the failures disappear from view.

We're talking about the squandering of hundreds of millions of dollars. You would think that magnitude of failure would have everyone's attention, especially those demanding executives. Not so. According to Mitch, elaborate stories are constructed to explain the failures and important people invoke the magic words, "That's water under the bridge," which make the failures disappear.

Mitch clearly sees the failures and would like to do something about them. But he sees retrospectives as an equally large failure. When he looks back on the his participation in retrospectives for failed projects, he sees a consistent pattern. Each retrospective would find that a root cause for the failure was insufficient or inadequate sponsorship. And in every case, nothing ever happened to improve project sponsorship.

The past. Each project team reaches an organizational river that they cannot cross without additional resources. They need a bridge built because the river is wide, fast and rocky. No sponsor exists, however, with the power to authorize its construction.

The project team valiantly goes into the forest, cuts down trees, and lashes them together to create a makeshift raft. They try to cross the river at the exact point where they wanted the bridge built. Like a scene from a Western movie, the raft crashes into the rocks, the contents scatter and tumble downstream.

The exhausted members struggle to shore. They eventually reconnect. They realize the project is dead. Someone suggests a retrospective.

The present. You are organizing the retrospective. As you send a meeting request to the members of the team, you tug at your ear attempting to release the water trapped inside.

Mitch receives the meeting request. He thinks to himself, "It's best to not make a big thing about this. I'll tell her what she wants to hear. But I'm not going." He clicks "Accept."

You see Mitch's acceptance. You recall an earlier retrospective where Mitch hadn't participated. You remember he accepted that meeting request too. And you also recall that there were a few others who behaved the same way.

What can you do to gain Mitch's participation in this retrospective? I suggest the following strategy:

0. Review the results from previous retrospectives. What should have resulted but didn't? Explicitly redesign the retrospective to address those deficiencies. A design element to show that this time will be different is to bring in an outside facilitator to lead the event.

1. Gain the sponsorship of someone who has the power to do something with the results of the retrospective. Show this change by adding an agenda item for the team to share its findings with the sponsor at the end of the retrospective.

2. Make time for a face-to-face discussions with Mitch and all participants before the retrospective.

3. Ask each person, "What could prevent you from being fully present during the retrospective?"

4. Be willing to make adjustment, such as rescheduling or changing more elements of the design, to ensure the highest level of participation.

5. Review the retrospectives objectives and agenda with each person. Highlight the differences between this retrospectives and earlier events.

6. Ask each person for suggestions about what would make the retrospective even better for them. Consider them and change the design accordingly.

7. Remind each person that if the team wants things to change it must show the data, its meaning and significance in ways that will compel the sponsor to respond. Ask them to find relevant data and bring it with them to the retrospective.

8. Ask each person, "Is there anything that I should have asked you that I didn't?" For anything mentioned, asked them how they would answer their own question.

9. Tell each person how important their participation is to you personally.

10. Say, "Thank you."

Let's assume that Mitch is right about the root cause of the problem. What better venue is there for gaining sponsorship for the next project than a project retrospective? But breaking this kind of viscous cycle is always difficult.

Gaining sponsorship is more than having someone listen to the findings and recommendations. Success hinges on creating superb findings with data that backs them up and connecting them with the personal success of the sponsor. The more things that personally connect with the sponsor, the more likely that that person will be to take action on the recommendations.

Any attempt to force Mitch to participate is stupid and counterproductive. You may be able to force his body into a chair, but he doesn't have to bring his mind. Your best chance is through showing him that this retrospective will succeed and trigger action. So ensure that the person who agrees to be the sponsor will take action when presented with a compelling business case. Otherwise, cancel the retrospective because it will waste your time and everyone else's.

Architecture Astronauts and Over Engineers

2009-07-16T00:12:32Z

Since it’s apparently Architecture Week™ [1] here at DevHawk, here’s another of my favorite Dilbert cartoons of all time – relevant to the discussion at hand.

Two interesting comments on yesterday’s post:

Architectural thinking is a necessary (and very important) part of software development - but beyond the systems level (which is systems administration and not software architecture) I have a hard time seeing divorcing architectural thinking from the actual development as anything but a terrible thing. Although I see that your definition of architecture (at the functional level) does not match my caricature of the 'architecture astronauts' which I do think can be endemic in languages that encourage additional layers of architecture. [Michael Foord]

So based on the definition of architecture I'm reading into your post, you wouldn't consider the choice of object-oriented versus functional programming styles from an architectural perspective? I'm trying to understand what level of architecture you mean here. Like Michael, I usually think of architecture even down into the implementation patterns level (hence the architecture astronauts), but that seems to be included in what you might be calling an engineering concern. [Ryan Riley]

Let me be very clear. Using my definition, there is no such thing “architecture even down into the implementation patterns level”. I’d argue that the implementation patterns level is engineering, not architecture. From what I’ve seen, the terms “architecture” and “engineering” tend to be used interchangeably in the software industry, and frankly I think that’s a mistake. I said as much in yet another post I wrote four years ago:

Architecture is the intersection between business and IT.

If a decision doesn't effect a business person, it's not an architecture decision. I'm not saying it's not important - I think the role of the software engineer is critical in large-scale enterprise system design and construction. And I will readily admit that often a single person is responsible for both architecture and engineering. But that doesn't make them the same activity. As long as we continue to confuse the two disciplines, we hold them both back.

Michael and Ryan (or anyone else for that matter) are welcome to disagree with my definition of architecture. I often joke that if you asked ten architects to define “architecture”, you’d get twelve answers. But that’s my definition and I’m sticking to it.

But what of the Architecture Astronauts? Both Michael and Ryan mentioned them. Unsurprisingly, I think that term is used too broadly as well. If you go back and read Joel’s original post of Architecture Astronauts, there wasn’t much reference, if any, to the implementation layer at all.

The Architecture Astronauts will say things like: "Can you imagine a program likeNapster where you can download anything, not just songs?" Then they'll build applications like Groove that they think are more general than Napster, but which seem to have neglected that wee little feature that lets you type the name of a song and then listen to it -- the feature we wanted in the first place. Talk about missing the point. If Napster wasn't peer-to-peer but it did let you type the name of a song and then listen to it, it would have been just as popular

[Joel on Software, Don't Let Architecture Astronauts Scare You]

I feel that my definition fits very well with the way Joel writes about architecture in this paragraph. The Architect Astronaut is trying to solve a real business problem - people need access to information besides music. But the mistake they make is thinking they can solve multiple problems with a single solution. So they abstract higher and higher until they’ve lost sight of the original problem and can only focus on the abstractions. If you look at what Joel has to say about technologies like Hailstorm and Jini, you see the same pattern emerge.

This isn’t to say that similar problems of over-abstraction don’t happen at the implementation layer – they do. But they happen for very different reasons. Astronaut Architects are trying to solve multiple problems with a single solution. But when over-abstraction happens at the implementation level, it because someone thought they could predict the future.

We’ve all seen our fair share of over-engineered systems that introduce significant unneeded complexity on the off chance that the development team can successfully predict the kind of change likely to come in the next version of the product. Invariably, the team’s precognitive abilities are revealed to be as poor as everyone else's, so they’re left with a bunch of extra layers of software cruft that has to be maintained but provides zero additional value to the system. I’ve blogged about that problem before as well: Kitchen Sink Variability.

Since I’m big on keeping the terminology of architecture and engineering separate, then I’d argue that we need a different term than Architecture Astronaut for people who want to introduce additional layers of abstraction at the implementation layer on the off chance that they don’t suck at precognition. Since we call such systems over-engineered, wouldn’t that make the people who build them “Over Engineers”?

[1] It’s like Shark Week, but with white boards and even more terrifying.

Google zaps Chrome updates with shrink ray, and there was much rejoicing

2009-07-16T02:10:00Z

Filed under: Google, Open Source, Browsers

One of our perennial complaints about annoying apps like iTunes is the fact that they require us to reinstall the whole app every time there's an update. Apart from being annoying, it's inefficient.

Google seems to feel the same way, which has led Chrome/Chromium developers to implement a new binary diff algorithm (dubbed Courgette) which greatly reduces the size of its updates. Courgette delivers updates that are .08% the size of the Chromium mini installer (about 80Kb as opposed to 10Mb). In the official blog post, they state "It is an anathema to us to push out a whole new 10MB update to give you a ten line security fix." Can I get an Amen?

The blog also offers several reasons Google is excited about the new, Liliputian updates:

We want smaller updates because it narrows the window of vulnerability. If the update is a tenth of the size, we can push ten times as many per unit of bandwidth. We have enough users that this means more users will be protected earlier. A secondary benefit is that a smaller update will work better for users who don't have great connectivity.

Now if we could just convince other application developers to adopt this kind of attitude toward patching. I'm not going to hold my breath.

Google zaps Chrome updates with shrink ray, and there was much rejoicing originally appeared on Download Squad on Wed, 15 Jul 2009 21:10:00 EST. Please see our terms for use of feeds.

Read | Permalink | Email this | Comments

CLR 4: Making the AssemblyResolve event more useful

2009-07-15T22:04:00Z

In the introductory post on CLR Binder (‘Understanding the Binder – Part 1’), we listed the set of steps that the CLR Binder follows, in order to locate an assembly and bind to it. On reading this, an obvious question comes to mind. What happens when all of these steps fail to locate the assembly? Does the binder simply quit looking?

It eventually does, but not before firing the AssemblyResolve event. The user can register an event handler for the AssemblyResolve event and then load the assembly that was intended to be loaded in the first place (or execute some other code appropriately).

The AssemblyResolve event itself has been around for a while now. So what’s changed in CLR 4? Prior to CLR 4, if an assembly A has a reference to another assembly B, and an AssemblyResolve event occurs for the referenced assembly (in this case, B), there is no means to know the identity of the parent assembly (or the referencing assembly, or A).

Why is this problematic? Let’s take this example. Let’s assume that an assembly FirstParent.dll references Child.dll. Let’s also assume SecondParent.dll also references Child.dll. On failing to load Child.dll, AssemblyResolve event is fired.

Now, while loading FirstParent.dll and SecondParent.dll using LoadFile(), an AssemblyResolve event is fired for Child.dll. Looking at the AssemblyResolve event, it is unclear as to which parent assembly actually triggered loading Child.dll. This is not helpful if the user wants to execute different code as a part of the ResolveEventHandler, depending on the parent assembly that caused attempting to load Child.dll.

In CLR 4, the AssemblyResolve event provides richer information that includes the parent assembly’s identity. System.ResolveEventArgs now has a property called RequestingAssembly, which is the assembly which requested the (unsuccessful) load of the child assembly, triggering an AssemblyResolve event.

With this new property, when AssemblyResolve event is fired, apart from the current assembly (which could not be located by the Binder), the parent assembly is also provided.

Thus, using this property,

(a) The user now knows which assembly caused the load event

(b) The event handler can now make use of the parent assembly that is passed.

You can read more about this new property here.

Banshee as a Platform?

2009-07-16T01:01:00Z

Aaron Bockover recently discussed Banshee's modular architecture. Aaron discusses using Banshee as a foundation for managing not only music, and videos but merging with F-Spot to manage photos as well:

The slides from his talk go into a bit more detail.

Jonathan Pobst takes the platform idea one step further and suggests that Banshee's core could also be used to replace YAST's Software Installer UI:

Jonathan says:

Every time I use YaST's Software Manager, I wonder if it would be better implemented using Banshee. Banshee's interface has been tuned for usability, both on its own, and what it borrows from iTunes. Software management is a naturally scary operation, and using an interface that the user is already familiar with could help reduce user fear.
Of course, it would just be the interface pieces of Banshee in a new app, you wouldn't actually start Banshee for software installation.
Another feature I would like to see taken from iTunes/Banshee is downloading/installing in the background. Once I hit Install, go ahead and download the application in the background, and install it in the background. I can click on the "Downloading/Installing.." menu item if I want to see what's going on. Most of the time, I'd rather be looking at other things to download.

I think this is a brilliant idea.

Open Atrium: customizable, open-source team portal

2009-07-15T20:00:00Z

Filed under: Business, Office, Productivity, Social Software

Open Atrium is a new open-source "team portal" that looks like it's going to be running a lot of intranets in the near future. It comes with six basic features: a blog, a wiki, a calendar, a to do list, a shoutbox, and a dashboard. You can also create smaller groups within your team, and they'll each have members-only versions of those features. That's pretty useful, but where Atrium separates itself from the pack of other offerings out there is in the ability to build your own features.

It looks like the folks behind Open Atrium are pushing to create a feature-developer community, where people can build and share features for everyone to take advantage of. If you've always felt like every collaboration option out there was missing one feature or another, Atrium might be able to scratch that itch. My favorite basic feature is the iGoogle-like dashboard, which looks like the glue that holds all the other parts together. On the technical side of things, the core of Atrium is Drupal, and also uses PHP, Apache and MySQL.

Open Atrium: customizable, open-source team portal originally appeared on Download Squad on Wed, 15 Jul 2009 15:00:00 EST. Please see our terms for use of feeds.

Read | Permalink | Email this | Comments

Mark Needham: F#: What I've learnt so far

2009-06-30T13:09:35Z

I did a presentation of some of the stuff that I've learnt from playing around with F# over the last six months or so at the most recent Alt.NET Sydney meeting.

I've included the slides below but there was also some interesting discussion as well.

One of the questions asked was around how you would deal with code on a real project with regards to structuring it and ensuring that it was maintainable. I'm not actually sure what the answer is to this question as I haven't written any code in F# that's in production but there are certainly applications written n F# that are in production - the main one that I know a bit about is one which Amanda Laucher worked on which she spoke about at the Alt.NET conference in Seattle.
There was some discussion about dynamic v static languages - Phil spoke of not caring about what type something is rather caring about what it does. I pretty much agree with this and I think when using languages which have quite strong type inference such as F# (and more-so Haskell from what I hear) then I think we do move more towards that situation.
Erik raised the point that functional languages aren't the solution for everything and I certainly feel it's niche is probably around operations with heavy data parsing/mining involved. I'm not sure I'd fancy doing an ASP.NET MVC application only in F# although I've seen some WPF code written using F# (unfortunately can't remember where) which looked reasonable so I'm not sure we should write it off just yet.

I've put the code that I walked through in the presentation on bitbucket.

F#: What I've learnt so far

View more presentations from markhneedham.

Small Steps Are Good; Be Careful What You Call Those Steps

2009-07-14T20:56:03Z

I love it when my readers challenge what I’m saying, as in Plunge In or Dip Your Toe? (for Projects).

I do believe in small steps for projects. I’ve long been an advocate of inch-pebbles, of standup meetings, of iterations and incremental development. I love knowing what done means, for the project and for features inside a project. You don’t have to be agile to use those ideas.

The idea I was *trying* to get across is that if you call something agile, please make it agile. If you dip your toe into it, you have not made the commitment. If you vary your timeboxes depending on whether you finish work in the timebox, that’s not agile; that’s incremental development. You can say, “We’re experimenting with incremental development. We choose to vary the length of the timebox so we can practice getting to done. Maybe after we practice it for a while, we’ll fix our timeboxes and see how that works.”

That’s perfectly reasonable. I encourage you to do so, if you haven’t tried incremental development yet. But don’t call it agile. It’s not.

The idea of plunging or toe-dipping for some of my clients is about management’s expectations. “We paid for this 2-day workshop last week. Why aren’t you agile now??” is a more typical management reaction (which is why I have a draft about what management needs to do).

Maybe I’m too particular about what I like to call things. But if you’re toe-dipping, say you are. If you’ve moved to something, such as agile, commit to the principles and practices that make it up. But don’t call it agile if all you do is continuous integration. If you’ve never done continuous integration before, congratulations! You’ll never go back But continuous integration by itself is not agile. I know of many successful waterfall projects who used continuous integration (merging the coding and integration phases) who would never claim to be agile, but used that practice because it fit for them.

Baby steps for change is a good idea. It’s difficult (impossible?) to make a wholesale change. Taking baby steps, trying them out, seeing what works is a great idea. Just be careful what you call it. If you’re not doing it all, you’re not agile. You are making changes. You are likely making it better. In my experience, at some point, you will need to make the plunge. At that point, commit and plunge. Just don’t call it agile until you make that plunge.

Tweet This Post

Links for 2009-06-15 [del.icio.us]

2009-06-16T00:00:00-07:00

Ruby at ThoughtWorks
Martin Fowler summarizes ThoughtWorks experience in 41 ruby projects

Links for 2009-06-06 [del.icio.us]

2009-06-07T00:00:00-07:00

Gojko Adzic » Oracle Coherence vs Gigaspaces XAP
datagrid options

Links for 2009-05-29 [del.icio.us]

2009-05-30T00:00:00-07:00

SOA Myth and Mystery
Nice article (from 2007) on myths and rules for SOA
Scrum vs. Kanban
A comparison

Links for 2009-05-09 [del.icio.us]

2009-05-10T00:00:00-07:00

The (lack of) design patterns in Python - PyCon 2009
DyradLINQ
MS research project for distributing linq queries

Links for 2009-05-06 [del.icio.us]

2009-05-07T00:00:00-07:00

eoinwoods.info - Articles
Articles by Eoin Woods

Links for 2009-05-02 [del.icio.us]

2009-05-03T00:00:00-07:00

Links for 2009-04-18 [del.icio.us]

2009-04-19T00:00:00-07:00

Test Heuristics cheat sheet
List of things to test on various contexts