But Tasktop has even deeper roots with ICSE. A fundamental aspect of Tasktop’s vision has always been to improve communication and collaboration amongst the people involved in software development so as to truly connect the world of software delivery. The initial step Tasktop took towards this vision was to embed the concept of a task into the IDE as part of the Eclipse Mylyn project. When Mik Kersten, our CEO, started the Mylyn project in the UBC Software Practice lab, the need for tooling to connect the IDE to common issue repository systems quickly became evident. Luckily, a connector that allowed issues from Bugzilla to be brought into the Eclipse IDE was available within the Software Practices Lab. This connector had been built as part of the Hipikat project . Hipikat recommends items from a software project’s history, such as past bug reports, source code commits and email messages, that might be useful to a developer currently trying to perform a task on the project. In essence, Hipikat serves as a memory of the entire project built from the project repositories so that it can answer a question that you might have asked someone at the water cooler if they had only not left the project. The starting point for many Hipikat queries is an issue or a bug. For instance, a developer may select a bug he or she wants to work on and ask Hipikat for similar bugs that have been solved in the past. As a result, Hipikat needed a means of having bugs in the IDE, which caused the initial development of a Bugzilla connector. Shawn Minto, who built the Bugzilla connector, is one of Tasktop’s most experienced software engineers.

On Friday of the conference, Davor Çubranić, who conceived and built Hipikat as part of his Ph.D. work at UBC, and myself, will receive the “Most Influential Paper 10 Years Later” Award for the paper about Hipikat. Our paper is receiving the award, as it catalyzed substantial work on recommenders for software engineering, some of which are finding their way into practice today. For example, more and more sophisticated code completion recommendations are finding their way into the Eclipse Java editor.

Hipikat’s name means “eyes wide open” in the West African language Wolof. Keeping your eyes wide open is as critical today for tackling the hard problems of software engineering as it was ten years ago. Each day, Tasktop strives to keep its eyes wide open when tackling the challenges that come with with connecting the world of software delivery. Will you be attending the ICSE Conference? If so, please tweet me at @gail_murphy.

Support for Clarity PPM

For many developers, the world of the project office is an alien one, with its staff talking about investment portfolios, resource pools and demand management. The same can be said of the PMO when trying to understand developers who work in scrums and talk about CI and GitHub. But with the advent of faster delivery times and Agile methods, development and the PMO need to work together in more dynamic, flexible and aligned ways. That means traditional integration approaches, such as spreadsheets and email, need to be replaced with automated integration. This need led us to develop a connector for CA Clarity PPM, which enables the two teams to work together more effectively sharing work across organizational and tool boundaries. The development of this connector also reinforces our strong partnership with CA and demonstrated our support for CA Clarity Agile and CA Clarity requirements.

Building the connector has reminded us yet again that the technical side of integrating the process and data is often the easiest part. It also reminded us that getting agreement on how the artifacts flow between these two organizations is actually much harder. As we worked on the early version of the connector with a customer, it became very clear that though at the highest level the PMO and development had shared objectives, the reality of day-to-day operation was very different for the two groups. We learned a lot about how the PMO and Development can work together during this process. This learning will form the basis of a webinar titled ‘Connecting CA Clarity PPM with Development Tool Stacks from IBM, HP MS and more’,which not only will demonstrate CA Clarity PPM integrating with the development stack, but will also describe the integration patterns that make sense and the key decisions you need to focus on when building the integration. The best practices of integration continue to drive our investment in Software Lifecycle Integration, where we hope to codify and share these ideas.

Improvements to the RRC connector

As more and more people improve their requirements processes and start adopting tools like IBM RRC, it is clear that requirements can never exist in isolation and that integration is key to delivering software effectively. Requirements tools are great at improving the discipline of requirements, but without linking them to a broader ALM tool stack, the requirements start wrong and just get worse. The key to good requirements is flow and collaboration; flow, meaning that the requirements seamlessly flow between management, the business, development and test, and collaboration, meaning that every stakeholder involved has the ability to comment, discuss and more importantly disagree with the how and why a requirement adds business value. We at Tasktop are heavily involved in this dialogue and continue to improve our requirements connectors as we understand how this interaction plays out. For example, a key improvement in the 2.7 release is the ability to sync into folders between RRC and HP QC / ALM. For many organizations, a folder is more than a way to group large list of requirements; it actually includes some level of business semantics. By adding this capability, we now can share context across tool boundaries.  This is a great example of something that we learned from our customers and partners as we enable better requirements flow and collaboration with Tasktop Sync.

Our partnership with IBM forced us to re-think team collaboration from the perspective of inter-organization communication while we were implementing an outsourced testing model. We know many of our customers are attempting the same model — but how do you accommodate different tools, different processes and different styles of communication inherent in different companies? That’s where Software Lifecycle Integration (SLI) principles come into play. SLI allows for choice — choice of tool which allows for choice of process which results in better communication and collaboration. And happy teams.

But it is not without its challenges. The greatest challenge we faced in our second phase was the recognition that when data is being shared across companies, you have to be careful about what and how you share that data. Confidentiality responsibilities became front and center as we figured out our SLI strategy for the outsourced testing integration pattern. Come join me for part 2 of our series on “Drinking Our Own Champagne” and learn how we overcame these challenges — and proved again the fact that choice results in flourishing teams and great products!

When outlined on a whiteboard or diagram, the architecture of today’s ALM stack resembles a half-eaten bowl of spaghetti, with meatballs corresponding to the various tools selected by stakeholders. We have two choices, either find a way back to the homogenous and single vendor solution of the 1990s, or embrace heterogeneity and come up with an infrastructure that provides end-to-end collaboration and traceability across best-of-breed tools.

Not long ago, we witnessed a similar transformation of the app dev stack. Once the services, databases and app server space enabled heterogeneity, the middleware category materialized and the Enterprise Service Bus (ESB) emerged, along with the new title of “Enterprise Architect”. History doesn’t repeat, but it does rhyme. It’s now time to create the role of the Lifecycle Architect, and to define an architectural framework for connecting the software lifecycle. Just as the notion of services was key to enabling the ESB, and file and documents abstractions were to sharing data, we now need an abstraction to connect the software lifecycle and to create a Software Lifecycle Bus. That abstraction is the social task.

Today Tasktop is a kicking off an effort to bootstrap the SLI discipline, with a series of whitepapers discussing the technical architecture, common data model, and technical tools. We are proposing the Eclipse Mylyn m4 open source project as a home for collaborating on a de facto implementation of the SLI data model, which will leverage what has been learned from the adoption of Mylyn, integrations that build on it, and new efforts around the open standards of Linked Data and OSLC. Later this week we are also launching an SLI integration patterns catalog, based on existing input from our customers, and open for all to contribute to. By the end of the week, with input from key thought leaders present at the ALM Connect subconference of EclipseCon, we plan to release a first draft of the SLI manifesto. To learn more and to participate, see:

	Whitepaper: Building the Business Case for Software Lifecycle Integration
	Whitepaper: Software Lifecycle Integration Architecture
	Software Lifecycle Integration landing page
	Eclipse Mylyn m4 Project Proposal Draft
	SLI Patterns Catalog Wiki (to come)

Learn more about SLI

For many 21st century companies and government agencies, software is the not-so-secret sauce for business innovation and competitive advantage. But with 30-70 percent of software projects delayed or failing, delivering successful software projects is still a game of chance. In response, software delivery is always under a constant state of flux, with the practices of development, test, requirements and deployment being modified and improved. But the improvement is often wasted or limited without a way of seamlessly connecting the disciplines of software delivery into an integrated, automated business process. That integration requires a new discipline, the discipline of software lifecycle integration (SLI). SLI is the ALM discipline focused on solving the problems of disconnected, fragmented software delivery. Justifying SLI requires a systematic understanding of the costs and problems associated with software delivery. In this paper, we introduce a five step process for building the business case:

1. identify integrations
2. obtain real financial numbers
3. create annual costs
4. factor in soft benefits
5. measure and learn

By applying a measured approach to SLI, it is possible to build a strong financial motivation, which can then drive direct improvements to the business process of software delivery.

Tools and processes are not integrated

Increasingly, a firm’s competitive differentiation is greatly affected by their ability to deliver innovative, high quality software at a low cost.  Accordingly, for many organizations, delivering this software is increasingly becoming a key business process. The defining characteristics for many products are not determined by the actual product but rather the software that surrounds and runs it. Given the sheer costs and potential value, programs to improve an organization’s software delivery capability are easy to justify, with benefit statements associated with reduced time to market, higher quality, improved predictability, and/or reduced cost.  However, when examining the efficiency of the business process of delivering software, it is not enough to look at the process alone; it is also important to understand the people and the tools involved in the end-to-end delivery of software.  The tools often focus on improving particular disciplines or automating gaps in the process, and the people employ their own unique working practices using the tools in a certain way.  Automated testing, model-driven deployment, build, IDE’s, requirements and project management tools have traditionally been easy to justify, but they alone do not provide the benefits promised. The missing component to success is integration. Integration, the glue that enables complex tool chains to be connected, has emerged as new tools category.

Integration is hard to justify

In the case of Software Lifecycle Integration, integration is focused on enabling different tools to connect. For many organizations, integration is similar to collaboration, or conceptually being valued, but it can be difficult to prove that value. Integration is even more closely tied to collaboration when you realize that integration is all about moving information between disparate stakeholders, i.e., integration provides the telephone lines needed for collaboration. Integration and collaboration are inherently linked, which is both good and bad.  We have all heard it, ‘improving collaboration adds real business value,’ but for many organizations who are increasingly managing growing business needs with shrinking budgets, investing in collaboration or integrations between groups is hard to justify. It is much easier to invest money in improvements to existing processes that clearly serve one role rather than breaking down the barriers between roles. Unfortunately, doing more of something that you’ve proven is inefficient is just a way of producing more of the same inefficiency.

Despite the negative connotation around silos, optimizing for the silo is easier to do than optimizing for the system as a whole.  In the past ten to fifteen years, the pendulum has swung from command and control, hierarchical, centralized IT organizations to Agile, Dan Pink’s knowledge worker freedom, and decentralized reality. In this time, we’ve had tremendous innovations in the silo e.g., Agile programming methodologies, virtualized testing, open source tooling, etc., but that has come at the cost of optimization across the silos.  With Dev Ops, test-driven development and social coding, we are seeing the pendulum swing back as organizations again try to find ways of ensuring the system is successful. Agile methods have in part caused many organizations to rethink their discipline boundaries, but Agile itself can be hard to justify outside the boundary of the development team. Hence, the reality of Agile adoptions for many organizations is ‘water-scrum-fall’. With disconnects between planning, operations and testing being left to other initiatives to solve, 30 to 70% of software projects result in failure or delays despite the advances in the silos.

Despite overwhelming opportunities for improvement and efficiency, for many organizations, justifying collaboration and their integrations is undermined by:

• Ownership.  It is easy to define who is responsible for improving one discipline, but who is responsible for the interaction between two disciplines? For example, who would be responsible for improving the relationship between development and test? Often it requires a third party to get involved, but that in itself can pose a problem as neither group trusts the input of a third outside party.

• Geographical, organizational and political boundaries. The reality of any large organization is that organizational structures evolve and are supported by both managerial and political boundaries. Breaking down these barriers is often very difficult when you are pursuing approaches that by their very nature bring down barriers between groups.    In our example between test and development, most organizational hierarchies have separate testing and development functions and the common manager is often too senior to be close to the challenges that are blocking the improvement.  With outsourcing and remote work a reality of software development, it is rare that QA and development are under the same roof, let alone on the same continent.

• Measurement.  Software delivery has a history of poor measurement, but even that limited measurement often focuses on one discipline such as testing, development or planning. Integration by its very nature changes the flow of work, adding value in unknown ways; thus its value does not necessarily fit nicely into existing measurements.  Additionally, measuring across fiefdoms is always a challenge to get the right information and to get that information to match up so that it is meaningful.

• Inertia.  Change is hard, and organizations have a certain inertia which slows down or stops change from happening. Integration often means organizations need to both think about the overall business process of software delivery and each discipline and make changes to their process models to integrate more effectively.

Vendor integrations are limited and hard to use

Connecting tools into one integrated business process sounds like the responsibility of each tool vendor.  After all, the tool vendors are supposed to offer Application Lifecycle Management (ALM) tools.   Tools provide individual value for one discipline but miss the much greater value that comes when integrated into a broader process, and many vendors provide out of the box, and sometimes free, integrations. But integration is not the focus for those vendors. Integrations are often created to appease clients, support migration, or make a sale. This leads to integrations being limited and hard to actually deploy in the complex, customized, and unique working environment that is the norm in today’s enterprise. The reasons tool vendors struggle to provide the best integrations include:

• You get what you pay for. For the majority of commercial software companies, integration is not a profit center.  Integration teams are funded by other revenue generating groups or paid for out of services and support budgets.  That leads to these teams not being able to keep up with the maintenance and support of integrations, and it often means that when new versions of other tools come out, the integrations are not updated or tested.
• Integration and migration are synonyms. Many ALM vendors have broad and comprehensive stacks of development tools. Integrating with competitor products is hard to motivate when management would much rather move those customers to their own product. This leads to competitors closing up API’s and making it harder and harder to get access to the underlying data. In turn, the competitive vendors do the same with their tool, and a vicious cycle continues with the customer losing out.   Even when tool vendors build their own integrations, they are typically set up as a one-way integration, bringing data to their tools from a competitive alternative.
• Licensing and competitors make it hard. In general, competitors do not like working together.  Licensing is one example of where vendors make integration difficult by explicitly stating that competitors cannot use the product or purchase it.  This leads to lawyers reviewing where and when a competitor’s tool can be used and reduces the likelihood that the vendor will build an integration.

Building your own integrations is complex and a maintenance nightmare

For many organizations, the need to integrate point tools has led to them creating internal integration teams who build custom integrations between tools. Even if the building of the initial integrations are outsourced to a systems integrator, the organization often still requires an internal team to maintain the integrations as the point tools update, as required fields change, and servers are renamed, etc.  These tool integration teams provide custom-made software, sometimes extending vendor-based integration frameworks or using integration platforms such as Tibco or Websphere. Though the initial cost of building the custom integration is easy to justify, the ongoing overhead of running these teams is often not accounted for when doing ROI and resource analysis and planning. What often starts out as a simple connection between two systems for one team with very specific requirements slowly evolves into a business critical operational system with a variety of use cases never foreseen when the initial integration was conceived. This is made harder when:

• You have to build skills in multiple tools. Each tool comes with its own quirks, implementation models and connection patterns. The more tools you integrate, the more skills the integration team needs to manage. For many tools, the integration points are poorly documented and require extensive ‘experimentation’ to implement. This learning is hard to document and often leads to custom built integrations being difficult to maintain by people other than the original creator.

• Update the tools when new versions come out. Traditional tool vendors release new functionality once or twice a year (the pace of new releases has gotten faster in recent history) whereas SaaS vendors release software with a much higher cadence. Keeping custom bespoke integrations up to date with the latest functionality is a large overhead. Also, many organizations support a number of different versions of a particular tool requiring integration to support multiple versions of a particular product.

• Testing and support are a burden. Ultimately, creating the integration is far less effort than supporting those integrations over time. Over time, integrations become a mission critical back end service requiring rigorous testing prior to release to avoid data corruption, missing records and process breakage.  That level of testing is a huge burden requiring multiple test environments, test plans and associated test infrastructure. As soon as the software is considered mission critical, any operational and release overhead will have to be applied to this software, requiring formal review and sign-off before the software goes live.

The value is obvious; it just needs the numbers

The value of connecting up separate tools, disciplines and processes seems obvious to many software delivery professionals. After all, the business processes for sales, logistics, accounting and operations were integrated and automated in the 90’s, ironically largely through software as a medium for the automation and integration, but there is a big leap between knowing something is of value and being able to prove it. There are numerous examples that demonstrate the value of integration. An often cited research study from IDC states that that the cost of not finding information is $3,300 per employee per year. But the IDC study focuses on finding the right information and its effect on productivity.  It ignores the powerful added value that collaboration creates when two groups that traditionally did not see each other’s information are presented with it. For example, by connecting the developer and the tester, existing requirements can be re-evaluated, giving the opportunity for a smarter solution. Daniel Moody and Peter Walsh describe the increased value of information in their work ‘Measuring The Value Of Information: An Asset Valuation Approach‘ as ‘in general, sharing of information tends to multiply its value – the more people who use it, the more economic benefits can be extracted from it’. By focusing on the value of sharing, connecting and being able to report on information, application development professionals will be able to provide context on the cost associated with integrating this information.

Give the right people the right information at the right time in the right form

Providing instant, correct information to the right people is the dream of any information system, and the business process of software delivery, like any other business process, is an information system. The majority of software delivery processes rely on manual processes, rekeying information, meetings, or email to connect key disciplines. Processes such as planning, requirements, development, test and deployment are disconnected, working in different tools and using different approaches at different cadences to solve the same problem. Not only does this provide massive process disconnects between groups, but it also undermines the information that should flow between them. Key artifacts such as requirements, defects, tasks and builds go through several transitions between inception and implementation. For example, a requirement needs to be transformed into a plan to be managed that can then be transformed into a set of activities for developers. The requirement also needs to be transformed into a set of tests to ensure that the business requirement is met.  Currently, in most companies, each transformation is manual and error-prone with key context information being lost. Transformations are undermined by: 

• Rekeying information. A common solution as an artifact is transformed across disciplines is to manually enter information into each tool.  Although this seems like a simple activity, as work moves from planning to analysis, design, development, test and deployment, each artifact is never quite translated completely.  In many cases, the individuals used to handle the manual data entry are either too junior and don’t understand the business processes involved, or those who do understand the business process are wasted doing work they abhor that should be automated.  Even if the individual doing the data entry is at the right level, manual data entry is fraught with mistakes, and the individual’s perceptions often cloud the transformation.  Context, intent and history are often lost when it is moved between disciplines. A great example is a requirement; when defined for planning, it means one thing and is described in that way. The plan that is used to drive requirements work has a simple list of requirements on it but no context and thus the business analyst will take that requirement and enter the details they believe to be true, which may be very different from the original intent of the requirement. All the context from planning is lost as it is translated into the requirements discipline.  The next transition to design again can suffer from these transformations. The process of software delivery starts to resemble the child’s game of Chinese Whispers or Telephone, where everyone converts what they hear to their own perception and changes its intent.

• Email is a nightmare. For many organizations, the glue that connects the disciplines is email. Artifacts are handed off via email, bugs discussed, and requirements described in often complex and sometimes heated email exchanges. Email however, is a poor proxy for capturing real collaboration, and the results of interactions are often poorly communicated and documented.  Email overload often results in missed communications as different people are included on the ‘to’ and ‘cc’ lists and documents are branched in different email threads.

• The amazing spreadsheet. Spreadsheets are used to store requirements, defects, priorities, issues and many other artifacts that drive software delivery.  But often these very important documents become large and difficult to work with. Also the challenges of version control and integrating information from a spreadsheet to other artifacts makes spreadsheets a great way for getting started but difficult to use for long-term integration. A great example is the defect spreadsheet bemoaned by everyone, as it quickly becomes out of date and difficult to work with. How many times have you been on a call where the first ten minutes is spent working out which version of the spreadsheet everyone should be working from?

• Status Meetings.  A normal solution to the challenges of spreadsheets or confusion in emails is a meeting.  Meetings are great for driving collaboration and clarification but are often difficult to schedule with disciplines in different time zones, language barriers, and busy schedules.  As the cadence of software delivery increases, having meetings in a timely matter becomes that much more difficult. Also, add to that the reality of team members being involved in multiple projects and initiatives; not only does this mean that an individual will be attending far too many meetings, but also they will have to context switch, which may mean losing information about a particular artifact.

Agility requires reporting from many sources

Today, the term Agile is used as the catch-all term when describing any improvement to the practice of software delivery. But at the heart of Agility is responding to the environment or building in a feedback loop. Feedback requires rapid, real-time data on how the team is operating, the software they are delivering, and what the customer wants. Daily standup meetings and backlog maintenance requires an integrated understanding of the project. When you add distributed teams and complex software supply chains, it is clear that some form of integration is required. Some of the challenges that arise when integrating for Agility include:

• A person with a clip board is expensive. Manual processes for gathering project status information and customer feedback are often the preferred method for many Agilisters, but the overhead associated with capturing this information is large. Add to that the increased cadence of Agile projects, and you have created a huge cost for the project manager, scrum master or team. This leads to the scrum master being a full time status checker rather than adding any value to the team, they spend their time updating the whiteboard, spreadsheet and communicating progress and status information to their management.

• Out of date information means wrong decisions. Agility is about responding to the situation in the most appropriate fashion. Response requires information, and that information needs to be up to date to enable teams to operate in the most effective way. Running a standup without current defect lists or customer feedback reduces the value of that standup and may lead to wrong decisions being made by the team.

• An individual updating a spreadsheet is error prone. Adding stories, defects and tasks to a ‘super’ spreadsheet is for many teams the answer, but the reality for most software teams is that the spreadsheet is always an afterthought, and the information it holds is never quite up-to-date. The spreadsheet also suffers from version control problems, as many members of the team wish to update it at the same time. Add distribution to the mix, and misunderstandings start to creep in on what people mean by the values on the spreadsheet.

Traceability and compliance is hard to take

For many software projects, the record of the truth is actually held in the heads of the development team. Agile practices accept that realization and encourage practices to share and communicate that tacit knowledge throughout the team. Tacit knowledge is not an adequate way of ensuring compliance and auditability. Having clear, connected software artifacts is key when external parties assess a projects adherence to company or regulatory policy. Also, that information is important when projects have been implemented and problems occur. Software development compliance is often undermined by:

• Manual steps. Having a clear process that is documented is one thing, but actually ensuring that people follow it is quite another. Manual process steps are likely to be ignored when time becomes of an essence and problems with the plan occur. To ensure compliance, manual steps need to be replaced with automation which is enables ease of audit and manageability.

• The breadth of information residing in many tools. The sheer complexity of many software projects makes it difficult to provide clear and concise information flows.  Multiple tools, coupled with external organizations and reliance on external APIs make reporting from one tool near impossible without integration.

• No one really thinks bad things will happen. It is true that the likelihood of litigation for most software projects is small, but as software plays a more important role in a business’s ability to operate, then transparency becomes more important to senior managers. Not only is the risk of litigation or audit a key motivator for improved traceability, but the underlying value of the business is reduced when management cannot really see the impact of change X or the ramifications of change Y.

A five step method for valuing integration

The value of integration will vary greatly based on each organization’s unique situation. Thus, it is impossible to say making defects available to both developers and testers in their respective tools has value $X. Instead the value of the information can only be determined by that particular integration, for that group with one set of data. For example, the integration between development and test would focus on the bugs and builds being generated from each group. That information would drive work for both groups. It is then possible to evaluate the costs / value of different types of integration (see table 1).

table 1 – example cost/value table

Step 1 – Identify integrations

It seems simple, but there are many different integrations possible in the software delivery flow, including requirements to development, development to test, test to requirements, and project management to all disciplines. Each integration has value and risks associated with it. For example, the value of connecting test to development is huge, but when every developer uses a different tool, the integration can be very difficult and thus increases the risk of the integration working or the associated cost. A quick scan of all the possible integrations and a risk, value gut feel is a great place to start (see table 2).

For many organizations, the integrations to focus on have been pre-decided because of existing process problems, audit concerns or serious communication problems. Thus, any holistic identification of all the integration opportunities is pointless; instead concentrating on the integration required tends to be the focus. However, reviewing the application lifecycle as a whole rather than a series of disconnected disciplines provides insight that is often missed.  It is too easy to focus on one particular integration or aspect of the lifecycle without thinking about how that flow connects to other flows. For example, considering test and development is a great start, but reviewing that in the context of operations and requirements will help you identify business changing epiphanies.

Step 2 – Obtain real financial numbers

Determining value in terms of high, medium and low is a great way of defining which integrations to focus on, but integrations have a cost, thus it is important to identify the potential fiscal value of integration.  The costs of integration are a great place to start in identifying information value. For example, by using a similar process that IDC employed in how they calculated the cost of finding information, ask a team how long it takes them to find the right build or defect list. Look at the number of emails asking questions about requirements and build status. That information should then convert into real time and effort, which has a real cost for each role (see table 3).

table 3 – example value

It is also important to factor in the cost of geographic distribution and organizational separation. For example, when teams are in different time zones, traditional techniques for aggregation such as spreadsheets and emails become quickly out of date and cause friction between groups. Distribution is therefore an amplifier of any cost.

Step 3 – Create annual costs and look for overlaps

Often the numbers that surface when looking at the costs and value of locating, aggregating and tracing get everyone very excited, but this needs to be balanced with reality. Factors such as release frequency, distance, and cost overlap can amplify or reduce the value. Also, annual costing implies some assumptions as to the work year, staff utilization and the general flow of work.  Rather than spending too much time calculating the perfect model, concentrate on a best case, worst case scenario, allowing for risk to be calculated into the model (see table 4).

table 4 – example costs and benefits

Step 4 – Factor in soft benefits

Not all benefits can be measured in terms of monetary value. Some benefits are not predictable; for example, the “a-ha” moment created by providing additional information to a developer about the test plan is hard to predict as it is dependent on the information and the developer paying attention to certain data trends. Also integration provides support in areas such as compliance and governance, but measuring the value of governance on a business is difficult to achieve at the micro level. Everyone is able to appreciate the value of compliance, as everyone can tell a story of a company who did something that was not compliant and dealt with the consequences.   For some organizations, compliance is mandated, and thus the value of integration is easy to define as it is in replacement of manual compliance reports and processes.  For example, the consequences of non-compliance in regulated industries can take the form of fines, limitations posed on a business, and negative branding.  Even in non-regulated industries, reserves for potential lawsuits, errors and omissions insurance, potential for business process optimization, etc., all have costs and benefits that may result in the soft benefits outweighing the hard, easy-to-measure benefits that we describe above.  Qualitative soft benefits can be found by asking the team and other stakeholders involved in software delivery what their biggest problems with collaboration and process flow are and then using integration to solve those problems.  Additional intangible benefits include:

• Improved visibility and transparency. Insight and intelligence gained from information across the value chain being available to all disciplines to improve each individual discipline not just individually but as part of a whole

• Replacing internal development team.  By adopting a systematic approach to Software Lifecycle Integration, it is possible to reduce the overhead of running the business process. For example, moving some of your best programmers off of internal facing IT / DevOps and on to far more valuable opportunity creation has an associated value.

• Long term trending and a data warehouse. By exposing an end-to-end process and applying automation to the solution, it is possible to use the information in different ways. One great way of using the information is within a data warehouse, which can be used to explore trends and do analytics which you could never do without a consistent integrated view of the software delivery process.

Step 5 – Measure and Learn

The initial business case is a moment in time and is created by spot analysis and time and motion studies. To achieve true long-term process improvement, a continuous measurement process is required, allowing that measurement data to feed into improvement initiatives and Kaizen opportunities. It is therefore important for any software delivery organization to put in place a repository of measures over time. There are many metrics that add value to software projects, but from an integration point of view those metrics include:

• Batch size – How many requirements, defects, tasks or test plans are moving around the system. Larger batch sizes make sense for well understood systems, smaller are more appropriate for applications where there is a large amount of unknown? By measuring batch size, it is possible to identify opportunities for improvement based on the relative size of the batches.

• Flow – How does work flowing between the different tools provide insights into bottlenecks and disconnects between tools? Flow is crucial for process success. When an artifact is created in one system and does not flow quickly into its next state or system, those disconnects are often indicative that there are problems with the process and integrations.

• Artifact change rates – Defects, requirements, test plans and tasks will all change during their lives on a project, but it is important that this happens in a continuous and managed way. Heavy requirements churn or defect creation spikes can indicate certain issues with the process.

What SLI means to you

Software Lifecycle Integration (SLI) is an emerging but important discipline that requires organizations to invest. SLI is the glue, connecting the disciplines that enable software delivery to be treated in the same way as other business processes.  By doing so, SLI provides direct returns on the investment but also allows greater returns on the investments made in each discipline on its tooling.  However building the business case for SLI is always difficult, as integration is less tangible than other tools such automated testing, but the ultimate value is much greater as information is made available in multiple places and different contexts. Application development professionals should:

• Treat software delivery as a key business process. For many organizations, software delivery is the last key business process to be evaluated, optimized and automated. By treating this business process in the same way as other key business processes such as sales, operations, accounting and logistics you will not only focus the organization on improving its value, but you will also be able to apply the decades of learning on business process improvement. That kit bag of ideas and focus enable you to apply financial management and build a stronger case for automation and integration.

• Look at the end-to-end flow. The key to building the business case for integration is to consider the end-to-end flow, from inception to implementation and maintenance. By not looking in detail at each discipline, but instead focusing on the transitions between the disciplines, it is possible to optimize the software delivery process without massive process change.  The beauty of SLI is that integration facilitates improvements without causing disruption in any one silo.  Each group can continue to work with their own tools, with their own workflows, but still gaining the visibility and benefits that comes from information being synchronized in real time.

• Apply the five step process for building the financial model for integration. By understanding the opportunity, the current cost and associated benefit of integrating each discipline it is possible to build a financial model for integration.  Spreadsheets, manual processes and email are easy opportunities for elimination to be replaced with automation via integration.

Look to company initiatives to connect to. Agile, Lean, DevOps and enterprise PMO initiatives are great candidates to associate the cost of change with. These initiatives often have strong leadership looking for tangible benefits that can be achieved with integration. Many of these initiatives require improved flow, better collaboration and cross discipline / tool reporting that software lifecycle integration will provide.

Download the whitepaper

v0.5.0, March 25, 2013

Download the whitepaper

Overview

Software delivery is plagued by disjointed siloes of information spanning software stakeholders and suppliers. The tool diversity that is the norm of today’s Application Lifecycle Management (ALM) stack brought with it productivity benefits for stakeholders such as developers, who have influenced the selection of the tools that make them most productive. However, the stakeholder-specific selection of best-of-breed tools has also undermined the decade-old promise of ALM. The lifecycle stacks of today are more fragmented than they ever were.
Imagine an integrated fabric that allows information to flow freely and in real-time across the stakeholders, tool siloes and vendor boundaries that define software delivery. We call that vision Software Lifecycle Integration (SLI). When the application development stack became overly complex, the role of the Enterprise Architect became clear, and a new discipline and middleware tool category formed. Just as connecting the back-end services that power our organizations required the creation of an Enterprise Service Bus (ESB), connecting software delivery requires an Application Lifecycle Bus (ALB). In addition, for the SLI category to form, we need:

	A role to be responsible for it, the Lifecycle Architect
	A taxonomy of integration concepts, types and approaches
	A common data model for SLI
	A lifecycle bus architecture
	Integration patterns

The goal of this whitepaper is to provide an initial version of the above for the emerging Lifecycle Architect role, to define a set of architectural concepts than the Lifecycle Architect can use to plan SLI deployments, and to bootstrap the conversation evolving around the SLI discipline.

Lifecycle tool diversity

Software integration is not new. In the 1990s, when application development stacks became more complex by combining application servers, legacy systems and databases, often from different vendors, Enterprise Architecture and Middleware came of age. A decade later the Enterprise Service Bus (ESB) became a core part of the enterprise stack. These technologies focused on the integration of application runtimes and of application data. With today’s rise of the API Economy, we are now seeing a resurgence of both the need to connect application and behavior in the cloud.
On the application and data integration layer, we’re in good shape. The trouble lies on the Application Lifecycle Management (ALM) part of software delivery, which is not about getting the application to run, but about developing and evolving the application over time. In ALM, we have seen a decade of promises of a connected lifecycle, with many a great vision outlined but never a realization of ALM that has lined up to customers’ needs. Where we are today is that organizations who have more than 500 developers have ALM stacks that are connected by brittle, manual, and combinatorially exploding point-to-point connections between vendors’ tools. This approach has not scaled and has now become the bottleneck on ALM adoption. Worse yet, the end-to-end software delivery cycle is only as productive as it’s bottleneck, and while Agile, mobile, cloud and open source are transforming the way developers build and deploy software, little has been done to address what organizations now see as the main bottleneck on software delivery, the disconnection and communication impediments in the software lifecycle.

Accidental vs. fundamental diversity

The root cause of the disconnected lifecycle is tool diversity. If an organization could procure the ultimate ALM suite from a single vendor, and meet the needs of every stakeholder, there would be no need for SLI. However, tool diversity is now the norm, not the exception. Diversity is also the sign of a healthy ecosystem. We have witnessed this in the coming of age of Agile as well as the new breed of developer tools, where competition, low-cost and open source entrants, and innovative vendors abound. The result is that two types of lifecycle tool diversity exist:

• Accidental diversity: This includes all of the heterogeneity in the tool stack that does not support organizational needs. Tools inherited as a result of M&A activities, or independent selection of similarly functioned tools due to a lack of centralized governance, fall into this category. For example, an organization could have three bug trackers, one a 20 year old legacy tool created in house, another the new developer-favored issue tracker, and the third an open source issue tracker that resulted from an acquisition.
• Fundamental diversity: All heterogeneity that adds value to the lifecycle by adding to the productivity of software delivery. For example, an organization developing a Ruby on Rails app may be using Redmine, while Java developers within that organization are using JIRA, and .NET developers TFS.

Benefits of tool diversity

Lifecycle tool diversity allows organizations to assemble best-of-breed stacks that meet the varying needs of stakeholders in the lifecycle, and that align to platform demands. Key benefits of fundamental diversity include.

	Specialized tools: The various stakeholders of software delivery require different tools to be effective at their particular discipline. For example, developers need customizable tools integrated with their development environment. In contrast, business analysts may want browser-based tools with the latest user interface mockup capabilities.
	Right-sized tools: Some tools have become specialized to organizational size. For example, a new Agile planning tool may be great for a dozen teams, but a traditional PPM tool is often needed for planning across very large numbers of teams. Similarly, a cutting edge lifecycle tool may be suitable for new development projects done by a small set of teams while enterprise-scale lifecycle tools remain the corporate standard for core products.
	Platform affinity: Lifecycle tool vendors who provide a development platform often provide an ALM on-ramp onto that platform. For example, an PaaS platform can provide deployment automation functionality, while a mobile platform can provide app store integration. As such, an organization creating a hosted back end for a mobile client may end up with two sets of ALM tools, each optimized for the development platform experience.
	Vendor diversity: Different vendors specialize at different stages of the lifecycle, and target different development styles. For example, some vendors specialize in supporting cutting-edge Agile trends, while others focus on tried and proven features. An organization may choose to have both deployed if the goal is to prove out a new flavor of Agile in a part of the organization, especially across different horizons of R&D.
	Supplier diversity: As outsourcing and consumption of open source software increases, it becomes either impractical to expect software suppliers to use the same ALM tools as the sourcing organization. For example, open source projects tend to use open source ALM tools, while small suppliers tend to use lightweight issue trackers in place of the enterprise ALM tools needed for large scale software delivery.
	Support for legacy: The cost and disruption moving away from a legacy system, such as an older tool or in-house defect tracker, can be overly high to take on during a lifecycle modernization. Allowing the legacy system to interoperate with the new systems enables a more incremental migration path.

Costs of tool diversity

A core challenge of software delivery has been that, without a suitable integration layer, the business benefits of diversity are thwarted by the costs of a disconnected lifecycle. These costs boil down to undermining the promise of ALM itself, as the benefits of ALM tools can only be realized with a sufficiently connected tool chain. Key problems caused by a heterogeneous stack that lacks integration include:

	Lack of traceability: Due to information being siloed into disparate systems, it is impossible to get cross-system traceability. For example, business requirements are disconnected from user stories which a disconnected from source code. Once a support ticket is filed with a major defect affecting a particular line of source code, the affected requirement cannot be determined. The less traceability there is in software, the more expensive the software is to maintain.
	Lack of visibility: Reports, dashboards and planning tools only have visibility into the repositories that they were built on. For example, reports in an ALM tool fail to see defects tracked in the Quality Management tool and dashboards connected to a data warehouse are missing data from the lifecycle tools that do not provide a direct integration. Lack of visibility makes it impossible to effectively manage a software portfolio.
	Limited governance: Governance, workflows, or compliance processes cannot be implemented across tools. For example, a compliance workflow implemented in a centralized ALM tool is not enforced in a lightweight issue tracker used by a large portion of the organization’s developers. Without governance, software delivery cannot be adequately connected to other business processes.
	Manual processes: Without an integration layer, end users often need to enter data into two tools, reducing end user productivity and affecting data accuracy, and rendering automation impossible. For example, creating requirements links between one tool and another, or entering time sheets into a PPM tool while the Agile tool is intended to be the definitive record of time tracked at a finer level of granularity. The lack of automation has individual stakeholders wasting time on manual entry, and impedes the deployment of lean methodologies.

Signs of lifecycle failure

Without an adequate integration layer, the following anti-patterns occur in the software lifecycle. Each of these contributes to the costs listed above by preventing the end-to-end flow of information across the lifecycle stack.

	Manual processes: Duplicate data entry and any other manual entry becomes the bottleneck of the lifecycle. These include excessive status reporting meetings, or status updates via email and spreadsheets.
	Batch jobs: Attempts and connecting the lifecycle that periodically reconcile data models across tools, but result in numerous conflicts and make cross-silo collaboration impossible.
	Custom solutions: Integration provide by services vendors or internal teams that result in brittle, costly to maintain and upgrade with frequent releases of ALM tools, creating lock-in to old versions
	Point-to-point integrations: Integrating two systems directly, with no abstraction in between, results in a brittle system that does not scale, resulting in batch jobs, conflicts, and other workarounds.

In the following section, we will examine the architecture goals and principles of a connected application lifecycle.

Software integration architecture

This section provides an overview of the SLI architectural concepts, including goals, the role of the Lifecycle Architect, and the integration types and data model.

Goals of integration

The goal of SLI is to help organizations create a lifecycle that seamlessly connects all of the disciplines, stakeholders and suppliers involved in software delivery. This enables the benefits of diversity while mitigating the problems. The benefits of an integrated software lifecycle include:

	Insight: The ability to deploy metrics, reports, and lean methods across the lifecycle. For example, implementing a lean build-measure-learn loop, from idea to customer validation.
	Traceability: End-to-end and cross-system linking of lifecycle artifacts. For example, all code, tests, branches are linked to business requirement, regardless of system heterogeneity.
	Flexibility: The ability to deploy new lifecycle management systems, and to retain legacy systems while modernizing the stack. For example, if a new mobile platform is being adopted for a new application, the lifecycle tools specific to that platform can be used.
	Flow: Real-time flow of social and status information ensures stakeholders work within their system of choice while being connected to collaborators in adjacent parts of the lifecycle. For example, as soon as a tester comments on a defect, requesting more details on steps to reproduce, the support engineering sees the comment within their system.
	Governance: While end-users get to work in the systems that make them most productive, the organization needs cross-lifecycle governance and workflows to function across systems. For example, policies and workflows can be implemented to define the cross-repository status and record updates that a creation of a defect results in.

The role of the Lifecycle Architect

When the complexity of software applications and the enterprise systems that they depended on grew, middleware formed as that glue that could connect end users to systems and data. In this transition, the role of the Enterprise Architect (EA) emerged and became a key technical stakeholder in the delivery process. It’s now time to define the Software Lifecycle Architect role to own this role, and to coordinate the activity of mapping the business of process delivery onto the Software Lifecycle Architecture. This architect’s goal is to define the stakeholders, workflow model, feedback loop that spans the systems and stakeholders that make up the lifecycle.

Lifecycle integration types

SLI breaks down into the following four types of integration, delineating the different silo boundaries and layers within a typical software delivery organization. The applicability of integration types varies with the size and amount of software delivery being done by the organization. For example, a small organization should have little or no “within discipline” integration needs as developers should be using a single tool. But it may have a need for “cross lifecycle stage” integration if testers are using another tool, or “cross-organization” integration if testing is being outsourced.

Integration architectures

Since the software lifecycle is about connecting stakeholders, a distinction needs to be made between the tools used by practitioners and where the data is stored. This distinction originates from Geoffrey Moore’s discussion of Systems of Record vs. Systems of Engagement.

• System of Record (SoR), aka Repository: Encapsulates all back-end services and data storage, typically exposed via API. For example, the database and REST APIs of an Agile planning tool.
• System of Engagement (SoE), aka UI: End-user facing system that supports interaction and provides a user interface. For example the mobile client, browser-based UI, or desktop-based developer IDE.

Part of the challenge of SLI is that most of today’s lifecycle systems have a coupling between the SoR and SoE. In contrast, Microsoft Outlook is an SoE that’s decoupled from its SoRs via email protocols, such as Microsoft Exchange and IMAP.
All of the integration types listed above can be achieved by each of the following integration architecture types. However, each has a set of tradeoffs, and potentially tool-specific limitations. For example, an Integrated Development Environment (IDE) based integration connect developers to testers to provide “cross lifecycle stage” integration. However, it is limited to developers if testers do not use a federated SoE such as an IDE, but instead use an SoE specific to the testing SoR, such as the web UI or rich client of the testing tool.

These integration architectures are not mutually exclusive and can be applied in conjunction. For example, a federated User Interface integration can augment an integration bus to show cross-system artifacts. An integration bus can, and should, build on open standards where possible.

SLI data model

Independent of integration type, any non point-to-point integration must support a common data model for the purpose of querying and interchanging lifecycle artifacts. We decompose this model into two main types of artifacts. The focus of SLI is to provide a data model for connecting SoRs via “tasks”, their social communication streams, and contexts that link the relevant assets.

Two main types of relationships exist between these artifacts:

• Containment: Artifacts of one type can contain artifacts of another. Eg, tasks can contain subtasks, requirement definitions can be broken down hierarchically.
• Linking: Tasks can be linked to assets, and vice-versa. Eg, a source code commit links to a user story, a requirement links to all tests that cover it and code changes that implement it.

Software lifecycle bus

Integration is not a new problem. However, solving the problem for the software lifecycle imposes a new set of constraints. In addition to connecting data sources and schemas, the flow of communication must be integrated. The software lifecycle is a conversation, and as such, the primary concern for SLI is to connect conversations across siloes and to model this flow. To do this, we propose that in addition to other integration architectures being deployed a software lifecycle integration bus is a required part of the architecture needed to meet the integration goals of large-scale software delivery. This new kind of bus infrastructure for the software lifecycle is something we refer to as the Software Lifecycle Bus (SLB), a new product category that enables the real-time and event-driven flow of information across the software delivery chain. In this section we discuss the requirements of the bus, as well as deployment and implementation concerns such as storage topology and failure tolerance.

Bus requirements

In order to meet the goals of SLI, the lifecycle bus must meet the following requirements.

	Vendor neutrality: Capable of connecting to a tool from any vendor whose API supports the SLI data model.
	Real-time communication: Support activity streams (eg, comment threads), identity and real-time activity propagation across tools.
	Interoperability: Must support common query and data interchange formats (eg, OSLC, Mylyn).
	Invisibility: End-users do not need to change their work practice or technical tools for the bus to function.
	Manageability: Must run unattended with automated conflict resolution and error notification system.
	Extensibility: Must support addition of connection points, business rules, and transformation semantics.
	Openness: Must be query-able and expose its data model using open standards, and support transformations to other data formats.

Bus capabilities

The bus needs to support connecting the following capabilities between the tools that it connects.

Storage elements

Since the bus is not a new lifecycle SoR, it cannot act as the storage for lifecycle artifacts. Instead, it should only store state needed for data consistency, workflow transition transactions, and performance needed to query and flow data across the tools that it connects.

Bus operations

The lifecycle bus enables a broad range of operations to be implemented on the information flowing through it. Some of the common operations are listed below.

Failure tolerance

While not a new SoR, the lifecycle bus connects systems critical to software delivery, and as such becomes mission critical to software delivery. If the bus supports write operations, it must also ensure data consistency for those operatinos. As such, the following failure scenarios must be supported.

Extensibility

Since the lifecycle bus is intended to integrate unforeseen end-points and mapping scenarios, it needs multiple layers of extensibility.

Conclusion

In today’s software lifecycle, information is locked into stakeholder-specific repositories. Our inability to make information flow both within the organization and across organizational boundaries is preventing the realization of Lean Manufacturing methods in software delivery, and in establishing the infrastructure for a software supply chain. To achieve this, we need an integrated fabric that allows information to flow freely and in real-time across between stakeholders, across the tool silos and vendor boundaries that define the software lifecycle. Creating the Software Lifecycle Integration (SLI) discipline, architectural frameworks and technical tools will provide organizations with the lean methods that will scale software delivery, and pave the way for an integrated software ecosystem.

Appendix

Data model validation

We have used the following sources of information and architectures in creating the models described below.

	Open source Eclipse Mylyn implementation and ecosystem of extensions: significant portions of the architecture have been implemented in Mylyn and vetted by it’s user and vendor community, which now includes over 100 extensions and 2M monthly downloads.
	Open OSLC standards: These standards are compatible with the underpinnings SLI data model.
	Commercial implementations: the Tasktop Sync tool implements the lifecycle bus architecture described here, and the data model has been validated by large-scale customer deployments of Tasktop Sync, as well as a customer-driven data model working group.

Open standards & OSLC

We are still in the early days of ALM interoperability standard adoption. However, standardization efforts have been improving, with the leading efforts supporting key aspects of the SLI data model:

	W3C Linked Data: preferred format for artifact linking
	OSLC: REST format for queries, cross-repository linking with rich tooltips OSLC-Core: Implements core query and… OSLC-CM: Implements the Task model OSLC-RM: Implements the Requirement and Requirement Definition artifacts in the model OSLC-QM: Implements the Test and Test Plan artifacts in the model
	File-based interchange (eg, ReqIF): limited in the task data that can be exchanged, typically simple fields only.

Eclipse Mylyn

The original goal of the Eclipse Mylyn project was to redefine the developer experience around the collaborative and social workflow of task-focused collaboration, and insulate them from the overloaded of non-development related features in ALM tools. The goal of the newly proposed Mylyn m4 project is to leverage the proven interoperability of the existing Mylyn model and to create a new de facto SLI model and reference implementation.

Download the whitepaper

For us at Tasktop, being named to the Ready to Rocket list for the second time highlights the sustained opportunity we have in front of us and reinforces our excitement for what is to come in 2013.  It’s so fulfilling to see that outside groups have just much faith in our success as we do, and we’d like to thank the selection committee at Rocket Builders for acknowledging us.  As a bootstrapped business, we are proud of what we’ve accomplished so far; we experienced a 250 percent growth in 2012 with the help of our 50+ staff across two continents, and being recognized by Rocket Builders makes all our hard work even more worthwhile.  Tasktop is certainly “Ready to Rocket,” and we have great expectations for our continuing growth in the coming year.

Tasktop is always looking to add the best and brightest to our team, so contact us if you’re interested in joining a company that’s passionate about connecting the world of software.

The real answer to the question of “What do you do?” is that I work for a company that provides an integration platform to connect up the tools and people that produce software from idea to coding to deployment. Our customers are some of the largest companies in the world, and in nearly all cases, they build software not for the sake of commercializing it, but rather as a means to solving a problem or driving a competitive advantage in their respective industry, from banking, to insurance, to healthcare, to retail, to manufacturing, to government. We’re helping traditional businesses turn into software delivery organizations. Cheers to that.

When you think of integration, its just not sexy. Period.

Even though we are participating at SXSW Interactive this week as a stop on the SXSW Startup Crawl, and our CEO Mik Kersten has a talk on Monday evening, the reality is that we just aren’t as cool (or as young, truth be told) as a lot of the folks who do social or mobile or cloud. We do know what those words mean but our lot is all about creating a new kind of collaborative infrastructure for software delivery. We are at SXSW because our US HQs are located in Austin and because of my love of the SXSW event, even though Tasktop doesn’t fit neatly into an event that I often joke is short for “SeXy SoftWare”.

I promised Mik that I would pitch his SXSW talk, so here is the pitch… We’re also here to demonstrate how crucial it is to create a multi-vendor platform for software delivery collaboration if we are to make the next order of magnitude improvement in our ability to deliver software, as Mik will outline in his talk Social Code Graph: The Future of Open Source. Seriously, it is a marriage of a geeky topic with cool visualizations, and I would encourage you to attend and experience some of Mik’s passion and boundless energy.

So, this blog is my feeble attempt to explain what we do and explain why it is important. I wish it was containable within 140 characters. I wish it was containable in some pithy one-liner that I could use at the bar that would result in me being the life of the party. Every time I get it to 140 characters, it sounds like some sort of marketing jargon. So, what we have at Tasktop is great business that solves a really big problem that still takes too many words to explain.

So, if your company produces software in any sizable scale, I’d encourage you to at least give this blog the ten minutes it takes to read, even if you aren’t directly or even indirectly involved in the production of software.

What is Integration?

When we talk about integration, we are talking about moving data from one system to another and keeping each system synchronized. That is exactly what we do at Tasktop. It’s all about the flow of information between the constituents who drive the creative processes around software delivery. We help companies automate the flow of information between all of their tools that are used in the production of software, ensuring that the right information is at the right place at the right time.

When you think about how software is produced, especially in large companies, it generally involves some hair-brained idea from a manager (I am a manager, so I feel comfortable making this assertion), some business analysts who have to convert that idea into a semi-sensible set of requirements, a product/project manager who must convert those requirements into a project plan, developers who get the pleasure of struggling to make it a reality, quality assurance personnel who figure out where they screwed up, and operations who determine why it won’t scale.

Generally speaking, each constituent above uses their own tools, often times from different competitive vendors. The various disciplines are often separated geographically, commonly with cultural and language barriers. The vendors are usually not interested in effective integration, as each is competing for grabbing as much of the software lifecycle as possible. Even if the products are from one vendor, they often are not integrated with each other. Adding fuel to the blaze, culturally, the different disciplines of software do not always get along. It is no wonder that, despite all of the great advancements in each silo, software projects fail or are delayed 24-68 percent of the time. At Tasktop, we do integration, specifically the integration to make information between the five disciplines of software delivery (requirements, development, testing, deployment and project management) flow between the various tools that each discipline uses. Whether you call it Application Lifecycle Management (ALM) or Software Development Life Cycle (SDLC), we are putting the “L” back into this business process of software delivery.

Why is Integration Important?

Why is moving data around between various systems something you should pay attention to? As we’ve been doing deployments of our Tasktop Sync integration solutions in enterprise situations over the past couple of years, we’ve learned that the data moving around is really just a proxy for the automation of the software delivery business process. As we kick off the deployments in our customers, we are often bringing the key representative stakeholders of the tools (e.g., QA and development) together for the first time to talk about not just their silo’d business processes, but more importantly, to talk about how the two (or more) disciplines should work together to build better software in a predictable and repeatable manner.

So what does this mean? The best way to think about this is by wondering what the world would look like when the software delivery process is not integrated. Over the past 15 years, we’ve seen each of the five disciplines focus internally as a silo. During those 15 years, each silo experienced tremendous innovations, such as Agile planning and development, Continuous Delivery, DevOps, functional programming languages, and test-driven development. The challenge was that as each of the silos optimized, the valleys between them grew ever deeper. Testers would often batch up all of the defects they discovered in a spreadsheet and share the defects every month or six weeks with the development team. Requirements were often not tied back to the activities that developers did and the tests that QA professionals ran, so a change to a requirement was often not caught. Reporting and analytics across the entire software delivery value chain was impossible without significant manual work. As more industries faced new regulations, increased reporting requirements, compliance from their stakeholders, and demands for more governance, the ramifications of not meeting these needs had real financial consequences. No matter what industry you are in, having an integrated software delivery chain will ensure higher software quality, faster cycle times for application development and delivery, and less rework.

Another reason why integration is so important is because it connects up different worlds. Over the past ten years, we’ve seen some pretty dramatic changes in the power of the individual. Whether it was populism driven by Apple devices being purchased by the individual and brought into the workplace, even though the individual had to pay for it out of their own pocket, or it was the freedom of inexpensive tools that an individual could easily purchase via a credit card (e.g., JIRA, Github or free e.g., open source ala Hudson/Jenkins, Git, Bugzilla, etc.), more and more software development had the inmates running the asylum. Even though Central IT groups or management had mandated the use of a particular tool stack or technology set, individuals were defying those mandates and going with the tools or devices that made them the most productive and effective in their jobs. So, one of the big problems that integration allows our customers to solve is to connect up the world of the individual, where the newest, coolest, most productive, least heavyweight usually wins the day, with the needs of the enterprise and what were viewed as heavyweight tools and technology that were trying to manage risk, gain visibility, ensure governance and compliance, and generally protect the organization. You can see one example of how Tasktop does this in the JIRA / QC webinar.

Hope to See You at SXSW or a Watering Hole Soon

Integration may not be sexy, but it’s now the main bottleneck that we as an industry have on scaling our collective ability to delivery software by 10x.

So if you see me out and about this weekend at SXSW, I am happy to talk about TechGirlz, or my kids, or Capital Factory, or DreamIt, or bootstrapping a business to 50+ people, or professional blackjack, but I hope you also ask me about integrating your software delivery stack. I promise I won’t scare you too much.

I think everyone agrees that software delivery is not an ancillary business process but is actually a key business process, and the ability to deliver software faster, cheaper, and of a higher quality is a competitive advantage. But delivering software is difficult, and if you believe the Standish Chaos report, anywhere from 24 to 68 percent of software projects end in some form of failure.

Even the criteria for success has been questioned by many, as ‘on time, on budget, delivering the functionality requested’ can still mean software that fulfills requirements but adds no business value. Billions of dollars a year are spent on software development tools and practices in the desire to increase project success and reduce time-to-market. Each year, development, testing, requirements, project management and deployment roll out new practices and tools. Many of these additions bring value, thereby increasing the capability of each individual discipline. But ultimately, the problem is not the individual discipline; the problem is how those disciplines work together in pursuit of common goals and how the lifecycle is integrated across those disciplines.

It has been a year since I joined Tasktop, and during numerous customer visits and partner discussions, two things are very clear: 1. the landscape of software delivery tools and practices is going through a major change, and 2. to be effective in software delivery you need to automate flow and analytics.

The ever-changing face of software tools and practices

Add Agile, Lean Startup and DevOps to a large amount of mobile, cloud and open web, and not only do you have the perfect book title, you have all the ingredients necessary for a major change in the practice of software delivery. Agile and Lean encourage rapid delivery, customer feedback and cross-functional teams focused on delivering customer value. Mobile and cloud are changing the landscape of delivery platforms, architectural models and even partner relationships. Never before have we needed to build flexible development processes that encourage both feedback and automation. Imagine spending three months writing a specification for your next mobile application when your competitors deploy new features on a daily basis. Imagine not connecting your new sales productivity application to LinkedIn, where your sales people have all their contacts. Our development approach needs to not only include partner organizations and services but also deliver software at a much higher cadence.

Automation of Flow and Analytics (reporting) is key.

I have noticed a strange relationship between increased speed, reporting and integration. When you increase the speed of delivery, traditional manual processes for reporting and analytics stop working or become an overhead. For example, one customer spent two days compiling the monthly status report spreadsheet across development, test and requirements. This two day effort required meetings with numerous people and emailing the spreadsheet around for comment and review. When the organization adopted two week delivery sprints, this work was an overhead that no one wanted to endure. Now the company had a choice: drop the status report, or look to an automated solution. Because more frequent releases meant the need to collaborate better, they opted for an automated solution that connected the test, development and requirements tools, providing a report that described the flow of artifacts among these three groups.

The automation not only resulted in creating the report but also improving the flow between these different disciplines. Suddenly there was clarity as to the state of a story or when a defect should move into test. This clarity was avoided in the manual approach, which left large amounts of ambiguity. The report drove the creation of automated flow, which resulted in a better process, which then fed the report with better data.

That means there is a sixth discipline in software delivery

Lifecycle Integration is emerging as a key discipline for ALM. It provides the glue that enables the disconnected disciplines of requirements, development, testing, deployment and project management to connect. It unifies the process and data models of the five software delivery disciplines to enable a unified approach to Application Lifecycle Management (ALM).

Without integration, many of the disconnects go unrecognized, and the flow between groups is never optimized. The larger your software delivery value chain, the more pronounced the impact of these disconnects. Factor in external organizations, either through outsourcing, application integration, service usage or open source, and these impacts can mean the difference between not just project, but business success and failure.

Perhaps we in the software industry are suffering a bit from the ‘cobbler’s children syndrome,’ with integration being a first-class citizen in the traditional processes we have integrated for our business clients for years. But the time is right to apply these lessons and build a discipline around lifecycle integration for the practice of software delivery.

Deep Dive Into Content

All the sessions in the program are great, but I would like to draw your attention to a few that caught my eye during the call for papers.

• Moving towards ALM 3.0 by Forrester Analyst Jeffrey Hammond. Not only is Jeffrey a great speaker, but always fills his presentation with lots of data that you can use back in the office. In this talk Jeffery is highlighting the major shifts in the fabric of ALM. Looking at the data this talk is based on we might see a redefinition of the ALM category, which is very exciting!
• What ALM knowledge you can expect from CS graduates by Gary Pollice, professor at WPI. The emerging skills crisis in software engineering is going to affect us all so I am excited to hear how you can better hire CS graduates and make them more productive. This talk also helps to remind us that ALM is more than just tools, but includes processes and people.
• Continuous Integration at Google Scale by John Micco, Google. CI has emerged as the lifeblood of modern software delivery, and on paper seems easy, but for the majority of large organizations with complex builds, heavy dependencies and nasty test environments building a workable CI environment is difficult. In this talk John describes how a very complex CI environment can be built and maintained in a very changeable business.
• Building Mylyn 4.0 by Mik Kersten, CEO of Tasktop. Mylyn has defined how Eclipse developers interface with systems of record such as bug trackers and project manager tools, but the underlying model has not changed for over 5 years. Mik is going to describe what needs to change to get Mylyn ready for next generation ALM.

Bring your boss to ALM Connect Executive Event

ALM Connect will provide a rich set of ideas for practitioners to take back to their teams, but without management support many of those ideas will never be implemented.

Above: EclipseCon 2012 ALM Panel with Mik Kersten, Dave West, Melinda Ballou and James Governor

Solving this problem was the motivation of running an Executive event on Wednesday the 27th of March. This event is aimed at decision makers and is by invitation only.

Content highlights include:

• Twitter and Github kick off the event talking about the future of ALM and how the next generation of software development is being undertaken. These presentations will show how you can marry innovation, rapid delivery and complex development teams into an Agile delivery capability.
• ALM in action case studies. Still working on the fine print, but we will have two high profile companies who are going to present their experience with ALM and how they are using ALM to form a competitive advantage. These sessions are aimed at telling all the dirty secrets of ALM, the motivation for adopting ALM and the reality of ALM in their companies.
• The user is the center: Apps in the world of engagement by Lee Nackman from HP. Lee has been involved in ALM for many years and was one of the executives responsible for IBMs involvement with Eclipse. In this talk Lee will describe how systems of engagement have changed the face of ALM and how that is only to get worse. I expect some sneak previews of the HP’s future ALM strategy in this talk.
• Agile 2.0 software development in the era of the social graph by Israel Gat Fellow at the Cutter Consortium. Israel a leading management light on ALM and the economics of software delivery will describe the social side of development. Describing how social graphs and other mechanisms can be used to better manage and enable software delivery.
• Scrum – Success ends with middle management by Ken Schwaber co-creator of Scrum. Having Ken, one of the drivers of the Agile movement at the event will add a level of Agile pragmatism to the proceedings. In this talk Ken will present the audience with framework for taking Agile to the next level, but be warned this path is not for the faint of heart.
• The future of ALM panel – This session includes Sam Guckenheimer from MS, Lee Nackman, Jeffrey Hammond and Mik Kersten on what the future of ALM looks like and how it will affect the audience. I will moderate, so expect an exciting and thought provoking session.

If you are interested in attending the executive day, or have boss who is interested then please let us know. You may also wish to view the Executive Day landing page, or the event announcement from the Eclipse Foundation for a detailed agenda. I can promise the event will be exciting, thought provoking and inspiring. It is rare to get so many leaders on ALM in one room and at one event.