Adventures of a Technical MasterMind

Blogger is done

2019-02-20T10:12:00.001-05:00

I'm too annoyed with blogger these days, and too annoyed with Medium's policies so you can find my current posts in two places:

Tumblr : https://jkowall.tumblr.com/
LinkedIn : https://www.linkedin.com/in/jkowall/detail/recent-activity/posts/

NPM is Broken

2019-02-20T09:26:00.002-05:00

As someone who bought and implemented NPM solutions, covered them as an analyst, and now watches the industry, one cannot help but notice that NPM(D) is broken. According to Gartner themselves, the data center is rapidly changing, the data center is going away, maybe not as quickly as Capp states, but it’s happening. This is apparent by the massive public cloud growth posted by Amazon, Microsoft, and Google in their infrastructure businesses. This means that traditional appliance-based NPMD offerings will not work, nor will traditional ways of collecting packet data. Many of the flow offerings do not handle the new types of flows which these services generate, but most importantly they do not understand the internet, which is the most important part of assuring services in cloud hosted environments.

The network itself is not just moving to overlay a-la NSX and ACI, it's moving inside of orchestrated containers, and new proxy/load balancing systems typically built off components or modifications of NGNIX, Envoy, and other layers for caching. Applications are evolving just as quickly, and you see a race in APM to keep up, but NPM seems to just stay the same. NPM is broken, since all of the vendors in this Magic Quadrant are making the majority of their revenue selling hardware, and most of the cloud solutions are not feasible at scale as they rely on getting packets into a centralized analysis point.

One of the ideas, when we came up with the NPMD market at Gartner versus the previously used NPM terminology, was an emphasis on diagnostics. The goal was that as infrastructure became more complex, the diagnostics and workflows must evolve significantly. Unfortunately, this hasn’t happened in the 5 years since this Magic Quadrant has begun. Why is innovation not happening? I have a hypothesis.

After reading the new Magic Quadrant for NPMD released in February 2019, there were some rightful moves in the ratings, for example, Extrahop's vision has always been sound, and leading most of the laggards in this Magic Quadrant. Extrahop has always been considered somewhat of a hybrid as they have been able to fulfill many APM use cases, including working with AppDynamics for specific requirements. Of the 10 vendors who remain:

5 companies are public

Broadcom, Cisco, NetScout, Viavi (JDSU) have been for a very long time. NetScout is the only of these vendors who make the bulk of their revenue in NPM technologies.
Congratulations to SolarWinds and the wizards at Thoma Bravo for the re-spinout!

1 company is venture capital-backed (Extrahop)
5 companies are owned by private equity (Accedian, RiverBed, LiveAction, LogicMonitor, SevOne)
Can't figure out who owns Colasoft, Zoho, and H3C. H3C is very large and diversified, as is Zoho.

If this market needs to evolve with the infrastructure why is there so little deployed growth capital? When looking at the last round of funding for Extrahop it was in 2014, nearly 5 years ago. If there is such heavy investment in other technologies to support the migration to cloud architectures, why not in NPMD?

In the next blogs, I'll be covering what happened in APM over 8 years ago, and what has failed to happen in the NPMD market to date. A second topic will be the evolution in buyers of NPMD solutions, and finally a further analysis of the cloud trends as applied to networks and network performance.

If you want to read the Magic Quadrant you can download a copy from the AppNeta or Extrahop websites, thanks to you both!

Like it or hate it? Just engage on LinkedIn or Twitter.

AIOps is a feature

2018-08-01T07:18:00.002-04:00

Amazing that it’s been two months since last writing, but it’s been a bit busy launching new products and getting customer implementations under our belts. Since the well-oiled machine is accelerating, and additional help is inbound, there is more time in the day to analyze the market and the rapid changes occurring. There has been a lot of AI washing, even more so than 10 months ago when this was penned. As expected, AI is the new cloud (these Microsoft AI commercials are driving me crazy). If you don't have AI in your message, you are not relevant. The buyers of technology are eating up the wording even though this is far from reality, as I have explained. The outcomes associated with generically attaching to a word is a good marketing move but creates a mess for the users of technology, and those analysts who cover and define markets. We already are seeing many end users of technology asking why they need so many different "AIOps" tools.

For those who see this term, but maybe don't quite grasp the meaning. The concept of AIOps is a new-ish Gartner market which makes much sense and resonates with the executives of most IT organizations. We are all suffering from information overload, and increased complexity. These are problems which can be better solved by computers than people, whether you call this AI (which it's not) or machine learning, the result and impacts are game-changing for those operating digital businesses:

In this figure, the flow is to collect data, use it to engage with others internally and externally, enacting changes deemed suitable by the algorithms. This automated platform could do this unassisted and ultimately result in increased business value.

The concept of a closed loop for the Enterprise IT Operations has been around for decades. Billions of dollars were spent by CA, BMC, IBM, and HP to create an entire closed loop automation system which looks a lot like what is shown above. If anything we should learn from our historical failures. The main difference in AIOps is that "Big Data" and "Machine Learning" now exit inside a "Platform." The problem is that the typical enterprise uses dozens of automation tools, over a dozen monitoring tools, and trying to collect this data and make sense of it has been nearly impossible. The closest we have gotten the use of generic tools to manage events from monitoring or logs from other components in the infrastructure. So, what did Gartner do? They created 11 categories to describe this group of AIOps tools, but they are so broad that any monitoring tool can fit into this schema.

Historical data management: Every monitoring tool must handle historical data, this is required to build reports and dashboards or understand trends.
Streaming data management: Most monitoring systems use a combination of batching and streaming to collect, analyze and send monitoring data. Streaming data collection and analysis is a requirement for most APM and NPMD solutions.
Log data ingestion: Due to the uptake of solutions built on top of the ElasticSearch project log analytics has become a feature of most monitoring tools. By my count, there have been at least 10 new products introduced this year which collect and analyze logs.
Wire data ingestion: Although packet data is a useful data source, it becomes hard to scale in public or private clouds. Nonetheless, many products out there use packet data, but in today's environments, often this data is captured on each OS instance via distributed analysis versus taps/spans, which come with additional challenges or limitations. Interestingly enough, Gartner only mentions taps/spans and not distributed collection and analytics.
Metric data ingestion: Every monitoring tool must collect time-series metrics.
Document text ingestion: This is less well understood in the market since many of the log analytics tools do an excellent job of ingesting human-readable documents and apply natural language processing to that data. Much of the advances are beginning to become table stakes as the ElasicSearch platform now has more advanced capabilities built into it, which continually improved.
Automated pattern discovery and prediction: Just two years ago very few monitoring tools had automated pattern discovery, but today many solutions have capabilities to reduce the human effort involved in problem detection. The challenge in this category is that many data sources once ingested lose the relationship or structure of the data, yet many can infer them from the data, but do so with a high number of false positive correlations.
Anomaly detection: Similar to the last category this has been around for a long time but has become more common since manually managing thresholds is not scalable or sustainable. Some products on the market have had this capability for a decade or longer.
Root cause determination: This is probably the area in this model where the most innovation is occurring. There are very few products on the market which do this effectively. Most of the offerings are using rule-based expert systems, but these have many flaws when they see new problems they were not programmed to handle. If this problem were solved well, then we wouldn't see massive multi-hour outages occurring regularly. In the next two years, this area will evolve quickly.
On-premises delivery: This one is fascinating as almost all software is becoming SaaS-delivered. Gartner is even now saying that By 2025, 80% of enterprises will migrate and then shut down their traditional data centers, versus 10% today. As those data centers close, SaaS delivery becomes the de facto method of purchasing the software used to manage applications and the infrastructure running in public or managed clouds.

The primary drivers of any technology are the use cases it solves or the business benefit it creates. Within AIOps the use cases are variant since role and persona are broad, requiring countless differing and sometimes conflicting requirements. The AIOps platform must be infinitely configurable and customizable, which makes it a jack of all trades and a master of none. Specific personae need prescriptive workflows and out of the box, functionality to show value quickly. The net result is that AIOps technologies are in fact features of existing products to meet the ever-changing data requirements. Most of the monitoring tools out there have added more data stores and capabilities to try to become this platform of choice. Adding analytics capabilities on top of the data stores, and ingesting additional data is now a requirement. In my work running technical partnerships at AppDynamics there has been a significant uptick on integrations between products to try to fill this gap and become "the platform of choice," but in reality, they are point solutions solving one or two key use cases.

More recently, in July of 2018, there was quite a shift within Gartner around AIOps. They have re-focused on cross-domain analysis, even further broadening the use cases for the data. When a solution is too broad it can no longer meet the user requirements, this is likely the case for AIOps. Assuming one can extract and store all of the relevant data (which is a challenge unto itself) having a generic platform interpret data without a specific model coming from the originating tool is nearly impossible. Once you lose the meaning of the data, it is generic and disconnected from the business impact. More advanced tools do not collect everything, but self-tune the way and how they collect, this means there are intentional gaps in data based on the design of the tool. AIOps systems cannot process data from tools which do not emit all of the data, the reason all data cannot be captured or exported is to scale the systems. The intelligence of more advanced monitoring tools is distributed within a system (agents, aggregators) and not merely centralized in the backend. Similarly, when IT projects which attempt to build large generic data lakes containing unstructured information do not yield the expected results, not to mention exceeding the estimated costs.

Vendors who overrotate towards AIOps show a limited understanding of the dynamics of the buyer and persona within an organization. Many vendors aspire to sell to new users, but their heritage prevents this from becoming a reality. There are various requirements in a large enterprise, which drive the buyers towards different solution sets. Assuming that:

“In five years' time, Gartner envisions that, for a number of leading enterprises, today's disjointed array of domain-specific monitoring tools will have given way to what is fundamentally a two-component system. The first component will be an AIOps platform, and the second component will be some kind of DEM capability.”

“Deliver Cross-Domain Analysis and Visibility With AIOps and Digital Experience Monitoring” Published 5 July 2018 - ID G00352799

A grand vision, but not realistic. The use of DEM has limited deployments today. Also, many marketing organizations use DEM technologies, but they are disconnected from other monitoring tools or teams, often due to siloed teams. None of these tools are integrated. The concept of a generic platform to analyze this data is not realistic or obtainable. Disjointed tools will remain the reality in 5 years unless the enterprise magically retires legacy applications and architectures and moves to common, modern, cloud-based application architectures. We will have an array of domain-specific tools (likely even more tools), but different organizational structures along with different employee skill sets. The change must happen in people and culture before the change in technology can begin to occur, beyond incremental improvements.

Please leave comments on the blog here or @jkowall on Twitter. If you liked or disliked this, that way I can judge what topics are most relevant to you, my readers. Thank you!

Misunderstanding "Open Tracing" for the Enterprise

2018-03-12T10:53:00.001-04:00

When first hearing of the OpenTracing project in 2016 there was excitement, finally an open standard for tracing. First, what is a trace? A trace is following a transaction from different services to build an end to end picture. The latency of each transaction segment is captured to determine which is slow, or causing performance issues. The trace may also include metadata such as metrics and logs, more on that later.

Great, so if this is open this will solve all interoperability issues we have, and allow me to use multiple APM and tracing tools at once? It will help avoid vendor or project lock-in, unlock cloud services which are opaque or invisible? Nope! Why not?

Today there are so many different implementations of tracing providing end to end transaction monitoring, and the reason why is that each project or vendor has different capabilities and use cases for the traces. Most tool users don't need to know the implementation details, but when manually instrumenting with an API, the implementation must be well understood along with the use cases for the data. When you look at language-specific deployments there is even more variable, what you can do in Java is worlds apart from what is available for Golang and Python. It gets even more complicated as every vendor and open source tool uses a different language to describe what they do and what they call things.

The Enterprise uses a wide range of technologies which must be cobbled together to make their applications work. Some of the custom apps written in Java, .NET and other languages, much of it a decade old. Other parts of the stack are packaged applications such as those provided by Oracle, Microsoft, SAP, and many more. These often work with messaging systems which span both open source and commercial tools using proprietary protocols such as those offered by Tibco, Mulesoft, Oracle, Microsoft, and open source projects such as ActiveMQ and RabbitMQ. Finally, there are modern technologies built in a newer language, for example, Golang or Python which often use native cloud services and PaaS platforms. All of these must work together for high-quality user experience and expected performance, resulting in desirable business outcomes. Don't forget the fact that each of these technologies often has data store and database requirements which must also work for them to function. The result is that identification, isolation, and remediation of problems is a big challenge.

In many Enterprise organizations, each of these "hops" are managed by a different team or subject matter expert who often uses another siloed tool for monitoring and diagnostics. The enterprise APM providers build end to end views across these technologies, both old and new. Unified views are incredibly hard to do technically and culturally, and even more difficult in production, under heavy load, with minimally affecting the performance of the instrumented transactions.

The retrospective to the Enterprise is smaller more modern companies who build things differently. They create customized stacks of open source, develop tools and technologies which can go very deep into the infrastructure layers. This subset of companies are not the Enterprise; they have typically been running in cloud or containers since the inception of the company. They make different decisions and are happy to leverage open source projects which are often customized extensively. They are writing their instrumentation for various purposes including monitoring and root-cause analysis. OpenTracing attempts to standardize the custom coded instrumentation with a standard API and language. Once the developer writes the instrumentation, how a tool or platform consumes data is not part of the standard, it's not a problem OpenTracing is trying to solve.

The result of this decision is that if a user has implemented OpenTracing with a specific vendor and language, let's call it "Tool A" they must release libraries or implementations for OpenTracing for each language. Yes, each of the dozens of languages needs an implementation from each tool with specific language features to match what the tool does. If I were to switch to a different tool, such as "tool B" that would mean changing the libraries which the vendor hopefully supplied for each language. Implementation of a new tool requires code changes associated with connecting the library and any language specific implementation changes. For a mature language such as Java, this would require changing the library and the implementation at the same time since the propagation formats to the tools are incompatible. OpenTracing doesn't solve the interoperability problem, so what does the "open standard" attempting to solve? Well, for one thing, is that it allows those making gateways, proxies, and frameworks the ability to write instrumentation. That should, in theory, make it easier to get traces connected, but once again the requirement to change implementation details for each tool is a problem.

Enterprise APM tools do a lot more than tracing, for well-understood languages like Java, APM tools collect metrics from the runtime (JMX), from the code (BCI), they capture call stacks, SQL calls, and other details which are correlated back to the trace. Advanced tools can even do runtime instrumentation without code change using BCI. If you were doing this with OpenTracing, it would require code changes to make the API calls. These code changes are then part of your code base and must be maintained and managed by your team. Don't forget that Enterprise APM tools do a lot more than just backend tracing, and they capture metrics and traces on the front end (mobile/web), infrastructure, log capture, and even correlation to other APIs. These additional capabilities are out of scope for the OpenTracing project.

So if the goals are solving any of the problems an open standard would solve, OpenTracing doesn't do a lot. If you read the marketing put out by various vendors and foundations, one would think its the panacea, but the reality is far from the truth. OpenTracing does not standardize metrics, logs, or other structured data which tools consume.

For some reason there is a presumption that swapping agents out of an APM tool is difficult, this is entirely not the case. Vendors replace each other all the time, if interested in these stories, many published, and countless others remain behind closed doors. It's much easier to change instrumentation done at runtime versus those hardcoded into your products. This is especially the case when the API and standards are always changing, and many of the changes in OpenTracing have been breaking changes to the APIs.

Enterprise APM vendors must build a lot of instrumentation to support these various technologies, languages, and frameworks. Each of the companies had teams of people just maintaining framework instrumentation. Each tool vendor has dozens of engineers dedicated to writing these, which is a waste of resources, and could be more efficient if there were an actual open standard.

In the past six months, a working group formed thanks to the primary open source developers at Google, Pivotal, and Microsoft who created the OpenCensus project, focused on solving these issues including interoperability. OpenCensus is a single distribution of libraries that automatically collects traces and metrics from your app, displays them locally, and sends them to any analysis tool. OpenCensus is a pluggable implementation/framework that unifies instrumentation concepts that were traditionally separate, consisting of three major components: tracing, tagging, and metrics.

As part of this project, we are defining TraceContext which is a standard header which tool vendors and open source projects can use to propagate information, otherwise known as a wire protocol. We are building this for HTTP currently in the first version. The organizations behind this standard include your major APM vendors (for the most part), all major public cloud vendors, and some private cloud vendors.

Finally, although OpenTracing gets a tremendous buzz, it's not as popular as one might think. When looking at the distributed tracing Github OpenTracing is not heavily mentioned, in fact, the most used OSS projects are tools are versus APIs. Leading OSS projects include OpenZipkin and Jaeger (Uber/RedHat) along with Skywalking (Huawei). All of these projects have a wide range of commiters, especially OpenZipkin which started at Twitter, and is being community driven, while having ties with Pivotal. These organizations are all intimately involved in OpenCensus along with the main committers to these tools. Most of the marketing fails to mention these successful projects even though they have higher adoption, and are already compatible with many frameworks, API gateways, PaaS, and public clouds, unlike OpenTracing which has limited use cases aside from a handful of small APM vendors and RedHat who have a vested interest in its adoption.

Since this is a wire protocol the public cloud providers, frameworks, proxies such as linkerd and envoy can implement it, and the data would be natively consumed by any tool, creating true interoperability. The value in this instrumentation is the analytics and context, not the instrumentation (for the most part). Getting behind this standard is vital for the APM industry as it will solve user concerns with open data exchange and interoperability. Please join the OpenCensus project and help us address the real problem, not create new future pain.

This blog post is the opinion of Jonah Kowall and is not affiliated with his employer or other parties.

Artificial Intelligence in Digital Operations

2017-09-20T08:51:00.001-04:00

Artificial Intelligence in Digital Operations

Not a week goes by where I don't see a vendor claiming that they have applied Artificial Intelligence (AI) to running digital businesses (there was a new one this week, but I began writing this beforehand). The list of vendors continues to increase, and when digging into the technology, the marketing is often overstepping the use of the AI term. Let's take a step back and understand what AI is from a computer science perspective.

The traditional problems (or goals) of AI research include reasoning, knowledge, planning, learning, natural language processing, perception as per Wikipedia. The other interesting trend is the application of the AI effect. This term explains that as we apply models and data to problems, and create algorithms to solve these problems, we move the problem outside of the scope of AI. Thus, abusing term AI to describe much of what we do in technology. Most of the technology itself is not complex when it comes to building self-learning models, but yet is described in mathematical terms.

Users expect AI to do the work for them, meaning providing insights and knowledge they may not possess or may require them to do additional learning and studying to come to these similar results. The net effect is we expect magic from these intelligent machines, hence the term AI. We see this very often in our digital assistants, whether you use the common ones Alexa, Google Assistant, Siri, or Cortana or the more advanced offerings such as x.ai or zoom.ai. Unfortunately, there are far fewer companies outside of a couple dozen building actual AI. Outside of the powerhouses AI today represents programming to produce outcomes we expect a human to come to, not providing us with additional insight. When implementing these poor AI solutions, they are often represented by a decision tree and focused knowledge along that tree. The downside is that if the decision tree or knowledge has gaps those become issues which the AI can no longer solve. These expert systems show theirs seems quickly. The intention would be for the AI to adapt, but that has yet to occur in the market. Humans, on the other hand, can react to and derive new creative decisions based on reasoning, broader knowledge (such as technical fundamentals), and perception. Computers do not have these capabilities unless they are programmed, which is a challenge today. This use of humans versus machines is one reason that we do not have automated heavy machinery when human lives are at risk (flights, military, cruise ships, and others). While people do make mistakes, they can react to the unknown, and computers cannot typically do this well.

When it comes to applying "AI" to IT Operations (monitoring and automation are my key focus areas), the industry is immature. For the most part, we've seen textual analysis, anomaly detection, topological analysis, regression analysis, and historical analysis. Some of these can be quite advanced by incorporating supervised learning and neural networks. As more of these capabilities are combined, they can almost seem intelligent and can help humans manage today's scale requirements.

The downsides are that today's AI solutions fail to incorporate general knowledge of how systems operate, they cannot do research and provide advice which is not programmed, and they lack "intelligence" aside from helping humans cope with challenges. The unsupervised learning capabilities are limited today and cannot create newly learned outcomes versus those which are programmed. These defined rule books consist of expected outcomes, which the computer then follows, versus intelligence required to create new outcomes. Incorporation of broader sets of knowledge such as StackOverflow, Quora, vendor knowledge bases, or other great sources of knowledge accessible on the internet would allow computers to derive new outcomes and advice. The use of these rich data sources is in its infancy, once this knowledge becomes integrated and automated it would provide a massive benefit to those complex operating systems.

As things evolve and real intelligence starts to take hold this will change, but we are a long way from that point, especially in digital operations today. We have some customers at AppDynamics thinking about the orchestrated enterprise of the future, and how these intelligent agents can help them make better and faster business and operational decisions. We are eagerly working with these customers as a trusted partner, and we look forward to hearing your aspirations, ensuring our alignment on the future state of AppDynamics.

Digital Business Operations : Fire Your CMDB

2017-08-24T20:46:00.001-04:00

As outlined in a prior post, Digital Business operations require new thinking and new technologies. As Operations evolves to meet the needs of Digitalization, so too must the core systems of record for operations. When running a data center with physical assets, or even user assets such as laptops, printers, and desktops the CMDB was a useful construct to understand what you had, things were static, and thus the problem was more easily solved. In reality, almost no one had an accurate CMDB, they most often hovered around 80% coverage, based on the beliefs of staff, and often were driven by a combination of automated and manual processes. The use cases for the CMDB are often tied to ITSM processes such as request, incident, problem, and change management. By having good data capture to record asset and component ownership and configuration it made these processes more robust, reliable, and accurate.

By using discovery tools which crawled technologies or leveraged the network for data discovery was accurate, leading to a well maintained CMDB. In my personal experience, I found the network approach to be great (I was a very early nLayers customer at Thomson Reuters), but it had challenges around packet capture and aggregation. These are challenges today with any packet based collection and are made significantly worse in public cloud environments, which were not an issue last decade.

When virtualization entered the fold, the number of workloads increased and became more dynamic, but it wasn’t a major problem for the existing systems to handle this change, aside from an increase in scale. As applications evolved and configuration moved from being defined within the application server configuration, for example, database connections, connection pools, message queues, Memcache systems, and other components. Those have transitioned to being defined in the runtimes, and discovery tools increasingly have issues collecting configuration information. Most enterprises have dozens of configuration management tools and automation stacks. Ranging from legacy solutions provided by BMC, HP, IBM, CA, and others, but these teams have also added new-ish opens source such as Chef, Puppet, Ansible, Salt, and more. Today these teams are looking at orchestrating infrastructure and creating new layers, and examples include the open source project terraform. The reason for the fragmentation in configuration management is evolving applications and associated infrastructure, which also rely and depend on older applications, running on classical infrastructure (ex: VMWare, Tibco, and Mainframe). Over time the stack becomes complicated and costly. None of the new players in this space seem to support legacy technologies, and the legacy vendor solutions make it cumbersome to support modern architectures. The result is that we have a mess, and no path forward to reduce debt.

In today’s world with a high degree of self-service (public and private cloud including SaaS), containers, and orchestration these discovery tools do not work. The processes which consist of an ITSM strategy, often underpinned by ITIL (more on this later) no longer function in an efficient and highly automated system Finally, building dependency maps and graphs are no longer possible or feasible via a centralized repository. Adding technology support to a non-functional process would not fix the problem. For example, CMDB discovery tools which add Docker support or attempt to handle Kubernetes or Swarm are missing the point and lack the capability to collecting data from ephemeral systems. The technology is not the only thing which has changed, but also the desired business outcomes. The net result is agility is paramount, implemented by cross functional product engineering teams. That shift requires a culture change within IT, and yesterday's solutions do not support these initiatives.

The business demands rapid innovation via incremental but continuous improvement, which results in a high frequency of change across infrastructure (physical and logical), applications, and environments. Discovering and controlling these systems is a challenge from both a security and audit perspective, but also from a service assurance perspective. Decentralized IT organizations driven by the business need to move quickly, and experiment and the mandate of innovation often contradict centralized IT organizations methodologies. Technology which relies upon access to systems that are often outside of the area of control within IT is an ever growing challenge. These issues require us to shift data collection from an approach of crawling and cataloging data towards instrumentation (scraping web services is fine if your data is lightweight, but typically depth is not captured in these exposed API endpoints). The approach of instrumentation provides a more accurate understanding of dependencies and user experience, and a dynamic way to understand relationships between physical and logical components, and allow us to create new use cases for this data to solve some of the problems the CMDB was designed to address. The next generation of CMDB will be dynamically modeled. I have seen sprinklings of this in a couple of startups, but they are missing the right type of instrumentation to provide real-time capabilities. Events, logs, and metrics are not enough, ingesting topological models is not sufficient, this must be instrumentation across application and network.

My next post on this subject will focus on ITSM, which today is disjointed with both customer support and development. I am a strong believer in fixing these issues the right way with teamwork, and I’m looking forward to sharing my thoughts on this topic.

CIO Insights: Gartner CIO Survey Shows Canadian growth increasing

2017-07-17T09:48:00.002-04:00

Being a Canadian, and to celebrating Canada turning 150 years old, I wanted to call out some interesting trend data I recently came across. Having spent a good amount of time visiting with customers in Canada in 2017, as I do each year I have noticed changes in the country. Quite possibly due to the changes in government and the political climate, or change and progress in the private sector.

In Canada, there are economic challenges today, mostly due to a high reliance on natural resources, and low market prices today. Even with these difficulties there are increased IT budgets in Canada, surpassing the global average growth rates based on the Gartner IT survey data.

When breaking this down further, only 38% of government employees projected budgetary growth versus 64% of non-government respondents in Canada. Comparing these to last year the numbers were 16% and 57% respectively. Canada sees increased investment from a technology perspective based on the survey group. Much of that investment is coming from the private sector versus the public sector

which is encouraging for the Canadian economy.

I still will be making a couple of trips to Canada in 2017 including speaking at the full stack conference in Toronto (http://2017.fsto.co/ ). I'll be presenting new content including new thoughts around monitoring and instrumentation. I'm looking forward to finalizing my presentation over the next couple months. I wish you a happy birthday Canada and look forward to seeing the results of these changes. See everyone in Toronto.

Digital Business Operations the new mode for IT Operations

2017-06-30T06:28:00.000-04:00

Recent conversations I've had with others in the IT Operations space helped formulate an idea I have been working on for a while concerning the future of Ops. The old mode of operating, you can call it ITIL based, or mode 1, or whatever you prefer is not going to be the primary mode of operations in the long term. Senior leadership has acknowledged this and made concerted efforts to change their habits of hiring, spending, and roadmaps for technology and delivery of new functionality.

More advanced teams operating in an agile manner, or mode 2 consist of smaller more integrated teams made up of individuals with different skills. These teams are meeting today's digital business challenges. In most enterprises, these teams are part of a bimodal strategy, but bridging the gap between mode 1 and mode 2 is something few have solved. I'm personally not sure this is possible due to cultural and fundamental differences in beliefs and trust. In many organizations, there is a high degree of variability on the level of investment between mode 1 and mode 2, but most leaders agree the future is moving more of the staff towards mode 2 due to business demand. Listening to customers is key.

There is a fundamental shift occurring between both modes of operation that we need two terms to explain how these teams operate and what they require from people, process, and technology perspective. Bimodal is not quite bifurcated enough compared to what is happening in these enterprises today. I'm coining the term Digital Business Operations and IT Operations as these two teams. There will be an acceleration of new technologies and capabilities which will further separate the way these teams operate, making unification even more complicated than it is today. Thankfully there will be better infrastructure abstraction technologies which will allow each of these teams to operate independently (naturally for root-cause we'll need bridges, which is what our goal is at AppDynamics). Many believe the answer to this will be the adoption of private and hybrid PaaS technologies, but I find these are too complex, rooted in yesterday's problems. A better lighter weight approach will emerge built upon containers and orchestration making infrastructure abstraction simple versus the complexity we see today in private PaaS.

The changes in infrastructure, application architecture and management are still pockets of enterprises, and often experimental in nature. Similarly digital business and agility must allow for experimentation, but at some point, the experiment solidifies into a core business tenant. This solidification is what will occur within Digital Business Operations which will result in new more repeatable (or industrialized) ways to handle processes, toolchains, and workflows which today are implemented inconsistently between organizations.

Digital Business Operations requires a fundamental change in specific areas we've taken for granted in mode 1 IT Operations. These include process frameworks such as ITIL, service management (ITSM, Ticketing, Bug Tracking), automation, and configuration management (especially the concept of a CMDB). Each one of these is a topic I'll hopefully cover in future posts. I'll share some of the challenges, ideas as to how the vendors may or may not solve these issues, and some insight into what practitioners or first mover organizations are doing to address these problems.

Instrumentation is Changing... Finally!

2017-06-16T13:08:00.002-04:00

I've always been a fan of trying to solve complicated problems. As an end user, I applied various technologies and tools to diagnose some strange ones over the years. Applications have become increasingly decoupled and distributed requiring the monitoring and diagnostics to change significantly. Let's look at a short timeline of the changes which have occurred, why these changes were necessary, how they helped solve a technological shift, and which challenges remain.

Phase	Why	How Overcome
Component monitoring	Distributed systems became pervasive	High-end solutions (CA, BMC, HP, BMC) became commoditized (Solarwinds)
Event correlation	Too many monitoring tools (still applies) created information overload	Enterprises rely on antiquated tools, many have given up.
Log analytics	Diagnostics too challenging in distributed systems	Splunk unlocked it, but ELK has commoditized it.
Front-end monitoring	Customer centricity is the key for digital business	Hasn’t yet been well adopted, penetration is still under 20%.
Time-series metrics	Microservices created too many instances creating too much data	Has not been solved, but open source is now growing up to handle scale.
Tracing	Complex and distributed systems are difficult to diagnose	Beginning to evolve.

In each of these phases we've had commercial innovators, and over time they have been replaced by either open source or commoditized commercial solutions which are basic and low cost. Typically as there were frankly bigger problems to solve.

I would say that 2010 to 2015 were the era of log analytics. At this point the technology is pervasive, 95% of companies I speak to have a strategy. Most of them use a combination of tools, but inexpensive or free software, not necessarily a trait which is shared by hardware or cloud storage, are becoming the norm. Most enterprises are using at least two vendors today for log analytics. Typically one of them is Splunk, and very often the other is ELK. When this market was less mature ELK was more fragmented, but not the solidified ElasticStack platform which Elastic codified has made it a viable alternative.

The era of end-user monitoring peaked in 2015. Everyone was interested, but implementations remain rather small for the use cases of performance monitoring and operational needs. End user experience is a critical area for today's digital businesses, but often requires a level of maturity which is a challenge for most buyers. I do expect this to continue growing, but there will always be a gap between end user monitoring and backend monitoring. Most vendors who have tried to build this as a standalone product have failed to gain traction. A recent example of this is SOASTA, who had good technology, and decent growth, but failed to build a self-sustaining business. Ultimately SOASTA was sold to Akamai.

2016 was the era of time series metrics, where that market peaked, and we saw a massive amount of new technology companies flourish. Examples of this were increased traction by vendors like DataDog, or newer cases including Wavefront, SingalFx, and others. We've just seen the start of M&A in this area Solarwinds buying Librato in 2015, and more recently VMWare buying Wavefront in 2017. We also have the commercial monitoring entrant Circonus who have begun OEMing their time-series database under the IronDB moniker. The next phase of this market will be the maturity of time-series databases in the open source world. Examples may include Prometheus, but I'm keeping an eye on InfluxDB. As these options mature for monitoring, IoT, and other time-series use cases especially solving easier data storage at scale. The other missing component is front-ends like Grafana improving significantly with workflows and easier usage. Ultimately we may see an ELK like stack emerge in the time-series world, but we'll have to wait and see.

Due to the increase in microservices and the resulting Docker containers, orchestration layers are beginning to take hold in many organizations. As a result of these shifts in application architectures, 2017 seems to have more of a focus on tracing. The importance of tracing is great to see take hold, as root-cause isolation in distributed systems requires good tracing (at least until ML matures in a meaningful way). Tracing also allows you to understand impact when you have service outages or degradations. Additionally, tracing can be used to tie together IT and business metrics and data. The trace is essentially the "glue" of monitoring so to speak. By tagging and tracing you are essentially creating a forensic trail, although this has yet to apply within security, it will! Gartner even began talking about the application to security not long ago in the research note titled Application Performance Monitoring and Application Security Monitoring Are Converging G00324828, Cameron Haight, Neil MacDonald. The detailed tracing is what companies like AppDynamics and Dynatrace do, and has been the core of their technologies since they were founded based on these concepts. These tools solve complex problems faster and perform much more than just tracing, but the trace is the glue in the technology. Unfortunately for buyers, these monitoring and diagnostic technologies come with a typically high price tag, but they are not optional for digital businesses.

Today's open source tracing projects require developers to do work, this is different than how commercial APM tools work which auto instrument and support common application frameworks. The open-source tooling is getting better with standards like OpenTracing, and front-ends such as Zipkin continually evolving. The problem is these technologies still lack the automation for capture that you see in commercial tools. How often do I trace? What do I trace? When do I trace? If you expect developers to make these decisions all the time I think there will be issues. Developers don't understand the macro performance of how their component will fit into the bigger picture. Also once a service is written and reused for other applications, it's hard to understand the performance implications. I am interested and currently experimenting with proxy level tracing to help extend tagging and tracing in areas where you may not want or need heavier agents. I hope to share more on this on a future blog.

The long-term goal, however, is to combine all of these siloed data sets and technologies by using more advanced correlation capabilities, in addition to applying new algorithms and machine learning to the problem, which is currently in its infancy. Over at AppDynamics, we do this based on a trace itself, but we are evolving new capabilities, and doing so in a unified manner, going back to a trace. Monitoring of digital businesses is going to continue to be an exciting space for quite a long time, requiring constant evolution to keep pace with evolving software and infrastructure architectures.

I’ll be giving an updated talk on monitoring and instrumentation which will cover a lot of what is in this article. We will also go into more depth around instrumentation and tracing. I will premiere this new talk this October at the Full Stack Conference October 23rd and 24th 2017 in Toronto http://2017.fsto.co for tickets. I am Looking forward to contributing to this great event.

That's too Expensive, the Pricing Battle

2016-10-10T14:34:00.000-04:00

Pricing is a tricky beast. I've seen a lot of models out there, and each has pros and cons. Firstly it depends on who are your target customers. My experience is in Enterprise software, which typically has a larger transaction price and volume, yet the number of deals is smaller. When focusing on SMBs as a target, the models change along with the selling motion. For good reasons most software companies want to have both models, but I haven't seen many companies able to execute this strategy. You end up with vendors adopting one model and causing fractures in the way tooling is licensed, many times the economics don't equate to good business decisions for the end user or the vendor. Here are the various licensing models I've seen in the IT Operations Management space:

Application footprint or infrastructure footprint based pricing

Per node, per CPU, per application server, per JVM, per CLR, per runtime

User-based pricing

Per concurrent user, per named user, per monthly active user, per page view

This model can track the users being monitored (in the case of end-user experience), or the users using a tool. For example, in Service Desk it could be the number of help desk agents.

Storage based pricing

Per gigabyte consumed, per event consumed

The measurements become more challenging in highly dynamic or microservices environments which cause additional issues regarding usage of specific technologies. Most apps consist of both legacy and modern technologies; hence the value is different from a solution to manage them.

Then there are the terms of the license which can be anywhere from monthly billing to 5-year commitments. These commitments can be for a minimum and/or a burst model. I've had some great discussions with analysts around value-based pricing. Although this is a very loosely defined term, building a pricing model based on the value someone gets from the software sounds perfect, in theory. How many problems are you detecting? Solving? Although this makes sense, calculating the "value" is a guess in most cases. With APM you can determine the amount of time/money/revenue saved, but it's still a challenge to build and measure clear ROI. Measuring ROI becomes even harder with other less customer-centric technologies.

In my career I’ve now seen pricing from three distinct angles, I’m going to summarize what I’ve found in each of my roles. These are personal experiences, and your mileage may vary.

End User

As an end user, I bought tens of millions of dollars’ worth of software over my 17 years as a practitioner. I always tried various tricks to ensure I was getting the best deal possible for my employer. Licensing and pricing is a challenge. How do I pay as little as possible for the best solution for my needs? When can I afford to buy an inferior product to meet my budgetary needs? When should I request more budget to select a technology that will differentiate us as a business? Regardless, the net is that everything is overpriced, and I could never get the pricing low enough to remain satisfied. It doesn't matter if I were paying based on application, infrastructure, consumption, or on-demand pricing. Although my technology providers were my partners, I also felt I needed to extract the most value for the least amount of capital from them to serve our shareholders.

Analyst

As I transitioned over to an analyst role, I learned yet more tricks around pricing and deal negotiation. Most technologies go through a cycle of immaturity to mainstream, and finally into obsolesce. Gartner uses two models to describe this; one is the Hype Cycle which shows the technology trigger through it being a productive technology. Additionally when coupled with a Market Clock the lifecycle is visible. Market Clocks address the lifecycle of a market and how technology becomes standardized, commoditized, and eventually replaced. These constructs are useful to both end users and vendors to understand how technologies mature, and the related pricing and competition to be expected within a particular technology market. I often gave advice to vendors due to the number of licensing and pricing models I had seen, and what end users were asking. Clearly, there was always complaints by end users about any pricing model aside from open source. Everyone loves the idea of free software, yet there are many hidden costs to take into account. Which technology providers can deliver results is often more important than the licensing model.

Vendor

I was fortunate enough to run the AppDynamics pricing committee for about six months, and I learned a lot about how to license and price. During this experience was the first time I had studied margins. AppDynamics software is available both on-premises and via SaaS delivery which makes the model especially interesting. Each delivery model has different margins and costs to consider. These have to be taken into account when we determine a pricing model and our discounting. I also learned, first hand, the struggles customers had with our pricing model. The net result is that regardless of how you price a solution the users complain about the pricing. There is no way to solve this problem that I have found, and it was rather depressing. The opinions are strategies across sales, product management, and marketing are all different, and each group has a differing perspective which is very challenging to rationalize. I am not sad to have moved along from that responsibility :)

My Take

Every model is flawed; pricing models are inflexible and software costs too much. If you don't bundle products, then quoting and licensing become complex, yet if you bundle then, you end up with shelfware. End users want to pay for what they use, and yet they don't want commitments, they want "on-demand" pricing. Without commitments, most vendors have issues predicting revenue or demand. This is often an issue if you are using traditional hosting for the product, which most SaaS companies do to some extent for cost reasons. Software, unlike hardware, doesn't have the same type of fixed cost to deliver. The margins are different. End users and salespeople also want the model to be simple to understand, calculate, and rationalize.

I may share more secrets later, maybe around how to negotiate licenses and different ways to get leverage. Leave your comments here or via twitter @jkowall on what interests you on this topic.

Industry Insights: The Cycle of Innovation: The Rise and Fall of HP Software

2016-09-20T13:41:00.001-04:00

I find the cycles in technology fascinating, and it's an unfolding lesson of historical cycles. Although we believe our industry moves at a rapid pace there are many macro cycles which occur over decades, the patterns do not change much. The first and current example is HP Software (with more focus on IT Operations). Let's rewind to the foundational pieces of HP Software, which came from the acquisition of Mercury Interactive in 2006. HP spent $4.5b to purchase Mercury and built a large well-established business off the platform in both Quality Assurance (QA) and IT Operations. Over time HP failed to invest, in what at one point was the market dominance of QA and a substantial footprint in ITOps, these once large market shares eroded as technologies commoditized and the buying shifted to best-of-breed. HPs solution set became difficult to implement (even for HP engineers), and ongoing management is hard requiring consulting and many resources. Having managed this portfolio at scale in the past, I felt this pain first hand. I speak to customers regularly today who use these tools, and none of them are satisfied users. The inability for these technologies to meet buyer demands and the burden of maintaining them resulted in significant market share decline. Based on the Gartner market share data (Market Share: All Software Markets, Worldwide, 2015) HP software's revenues from 2013 to 2015 went from $4.4b to $3.4b showing a major issue in execution.

This kind of decline is highly abnormal within healthy businesses, and may be overstated by the Gartner data, but it creates an opportunity for companies who have inexpensive access to capital. Due to the low-interest rates, and the need for debt investments, there is ample access to cash. Just yesterday Thoma Bravo raised another $8b fund, which was oversubscribed. Capital supply is outstripping the amount of companies available to be acquired and streamlined. The criteria for these acquisitions focus on taking advantage of the massive amount of software maintenance generated by enterprise software businesses. There are examples of companies who's business goal is collecting software maintenance and lack innovation.

Analyzing HP's earnings statement filed on September 9th the software margins were only 17% (http://investors.hpe.com/~/media/Files/H/HP-Enterprise-IR/documents/q3-2016-quarterly-results.pdf page 11) meaning the optimization would improve profit. Most of this profit comes from maintenance versus new product sales or innovation.

Micro Focus is a good fit for this kind of business. Micro Focus has acquired loads of once innovative assets that have large install bases collecting maintenance revenue. Micro Focus' business includes software assets resulting from the merger of Attachmate, along with new additions. Much of the product set is a plethora of Mainframe software but also includes SUSE Linux, NetIQ, and other software graveyards. I still see quite a bit of NetIQ out there, but it's large legacy install bases especially in the Government Vertical. Replacement of these legacy management tools is a challenge; the result is a long tail of software maintenance. These customers pay vast sums of money for maintenance while getting no technology advances.

Analyzing the Micro Focuses annual report, maintenance outpaces license sales by over 2x. License sales and maintenance both declined between 2015 and 2016 with maintenance suffering a larger decline. That decline is a result of customers switching providers or technologies, and stopping maintenance payments.

Other notable examples of privatization and PE was the acquisition of business model innovator Solarwinds. Solarwinds IPOed in 2009 with $116m in annual revenue and had grown the business to $428m of annual revenue before being privatized in 2016. Solarwinds created not only simple and cost-effective tools for engineers but did so by only leveraging inside sales, driven by aggressive email marketing. Solarwinds figured out this formula long before companies like Marketo (and others) made is easy to build similar models. For many years, SolarWinds had a business advantage in the selling and marketing of its offerings to the technologist buyer. The tools weren't much better, but they were cheaper, easier to try and buy, and the value was there. Solarwinds ran into issues when others were able to replicate the business model they had, removing some of its advantages. As a response Solarwinds had to expand its portfolio, often too quickly to meet the demands of Wall Street investors. Several of SolarWinds acquisitions were not well thought through, and the efficiency of the business suffered. The net margin was over 30% in 2011, and when they were privatized in 2016, it was down to 18%. The SolarWinds portfolio today consists of many products which are not tied together, and highly complex to adopt. They still have several good technologies, but using them together provides no advantages. The current situation leaves a lot of room for current PE owners Silver Lake Partners and Thoma Bravo to optimize the spend and business, which will result in profits for their fund investors. Unfortunately, it will likely not result in a lot of investment in the technologies to keep place with the current market demands.

Similarly, Dynatrace was a highly innovative product at the time of the first public product in 2006 they focused on a gap in the market with an offering to address performance during the development and testing lifecycle. After expanding from Europe to the US, it was acquired by Compuware in 2011 for $256m and became part of a portfolio of products. After Compuware was acquired by PE in 2014, they split apart the mainframe business of Compuware (and kept the name) and brought back the Dynatrace name to handle the Application Performance Monitoring products. In the meantime, the PE firm also took Keynote private and combined them into Dynatrace. Over 14 years of being public Compuware's revenues went from a peak of $2.23b in 2000 to a low of $720m in 2014, which is less revenue than they made in 1997. Today the Dynatrace portfolio consists of several products which have different backends, user interface, and technologies. You are probably noticing a trend here, and it doesn't benefit the end users of these technologies.

A slightly older acquisition by PE, the privatization of BMC occurred in 2013. With the help of BMC's new owners, the optimization has taken hold; they are focusing on specific assets. The reduced spending has slowed revenue decline. From 2013 to 2015 based on the Gartner data BMC's revenue dropped from $1.87b to $1.84b. BMC's privatization required raising a massive amount of debt, affecting the risk rating of the company's ability to repay those debts. Based on Moodys over $6.7b in debt was raised (https://www.moodys.com/research/Moodys-downgrades-BMC-to-B3-rates-new-parent-notes-Caa2--PR_296668) causing an increase in investor risk. Similarly, when you look at the LinkedIn company data regarding BMC, the employee count has been reduced by 7% in the last two years, while the investment in engineering and sales headcount increased 1%. PE firms are cutting staff, and optimizing the business.

Dell/EMC is yet another example of two companies which have been taken private. Similar stories playing out at other large corporations including IBM, and CA. Each has similar stories, and the final shoe has not dropped yet around their software businesses. IBM has some successes in software, but the IT Operations software business is in a similarly dire situation to what has been seen above. In my past predictions, I expect CA to be taken private before the end of next year (2017), we'll see how accurate that turns out to be. Although Mike Gregoire has made some drastic improvements since taking the helm in late 2012, yet CA's portfolio continues to be legacy (heavy on the Mainframe) with almost no revenue coming from SaaS subscriptions.

In recently published research by Gartner's Gary Spivak (I&O Leaders Must Actively Respond When Key Vendors Face Activist Investor PressuresPublished: 29 March 2016 ID: G00291934) he states in his strategic planning assumption: "By 2020, eight of the top 12 publicly traded IT operations management (ITOM) vendors will respond to pressure from activist investors to sell all or parts of their businesses." As someone who studies the financial and market side, I'm quite happy to be siding with Gary's perspective on the future of this market.

You can probably spot the cycles I have highlighted, I would expect these to continue happening, especially if borrowing money remains cheap, and accessible capital remains high.

Thanks for the readership, please leave comments here or @jkowall on Twitter.

Breaking Down Engineering Investment at Innovative Companies

2016-07-12T03:31:00.001-04:00

I’m always looking for good ways to understand how companies invest money internally. When I was a Gartner analyst, I would keep an eye on data from Glassdoor and LinkedIn regularly to try to gauge the trends. I would use them in my understanding of companies versus what I was being told by the companies and end-users themselves. I always did this by checking content on the sites regularly and recording it into build patterns and trends.

Thankfully, LinkedIn has come to the rescue by releasing the Premium Insights feature on company pages. I’ve been looking at this data, and I’m finding some interesting trends regarding what it uncovers. I’m also going to compare the LinkedIn data with the Gartner data which shows market share and revenue.

I’m going to have a look at the larger companies in ITOM to compare those which are investing for the future of their customers and those who seem to prioritize other functions. I’m going to look at Support and Engineering percentages and revenue based on the Gartner market data.

Company	Employee Growth (2 yr)	Employee Tenure (2 yr)	Engineering (%)	Sales	Revenue change (3yr) - Software
Source	LinkedIn	LinkedIn	LinkedIn	LinkedIn	Gartner
AppDynamics	114%	1.7	32%	28%	531%
AppNeta	48%	2.6	22%	32%	162%
BMC	-5%	7.2	10%	17%	-2%
CA Techn	-13%	6.5	30%	16^	-10%
Dynatrace	37%	3.1	21%	23%	N/A
HP	-1%	7.9	14%	9%	-22%
IBM	-3%	8.1	16%	8%	-9%
Idera	18%	3,1	9%	49%	N/A
New Relic	90%	1.7	27%	29%	188%
Microsoft	-4%	6.1	25%	13%	3%
Google	20%	3.8	32%	12%	26%
Solarwinds	11%	3.1	15%	24%	52%
ServiceNow	67%	2.2	16%	20%	142%
Splunk	81%	2.2	25%	29%	116%
VMware	3%	3.8	32%	15%	27%

The reason I decided to analyze the turnover, is because happy employees shouldn’t be leaving. Then again, if you are in hyper growth mode, the turnover will likely be much higher. The other key number is the investment in engineering. If you are an end-user, it’s critical your vendor is investing in engineering. It seems the sweet spot for engineering is in the mid-twenties, but if you have a large portfolio of products, this could also potentially skew how much innovation is going on.

Disclaimer: This is my personal website and reflects my views and opinions only. Any comments made on this website by myself or by third parties do not necessarily reflect views or opinions of my employer.

Industry Insights: The Challenges Behind New User Interfaces

2016-06-21T13:45:00.000-04:00

The future of computing will be bifurcated. On one hand, there will be entirely new models for computing such as voice, autonomous agents, and bots, with no traditional user interfaces. On the extreme opposite hand, there will be new user interfaces augmented with our ‘real’ worlds, such as the innovation done by Microsoft holographic computing technologies, along with virtual reality platforms coming to market from Google and Facebook. Bringing these trends to fruition, though, will require some key enabling technological limitations to be overcome.

Voice

It’s been slowly happening for a while now: Voice recognition will change one of the key interfaces with today’s computing and applications. Apple’s Siri, Google Now, Microsoft Cortana, and the super-hot Amazon Echo, along with their smart agents, are the practical embodiments of a growing trend toward the application of machine learning to voice and data. Andrew Ng, chief scientists at Baidu, says that 99 percent accuracy is the key milestone for speech recognition. Companies like Apple, Google, and Baidu are already above 95 percent and improving. Ng estimates that 50 percent of web searches will be voice-powered by 2019.

Agents

The next natural step for this accurate voice recognition technology is the incorporation of a learning bot that learns all about your life and assists with your tasks, via voice recognition, of course.

These new technologies will require voice recognition access, data access, and interoperability with connected assets. These agents will continually learn, access new data sources, and provide you as a user with a significant amount of value. But these great innovations also come with many (sometimes steep) costs. They will generate ever increasing numbers of API calls, requiring vast amounts of infrastructure, and require new levels of scale and management.

Voice represents the new computing interface model — one that many point to as the interaction model of the future. And we are just a few percentage points away from achieving the technical prowess to make it ready for prime time. In Mary Meeker’s recently posted internet trends for 2016, she calls out an observation.

Performance

The performance of these interfaces is a key to adoption, but often the voice recognition system is hosted by a third-party on another network. Voice-driven applications must send data to one of these providers, get a response, and process it. Delivering results in under 10 seconds to the user? That’s quite a high service level, considering most transactions lack visibility from end-to-end. Transaction tracing capability and tying the user request through the dependent systems and APIs will be critical to meet this user response time requirement.

Another example is the wild success of Amazon’s Alexa voice services. Their first device, the Echo, is #2 in electronics in Amazon’s store today (June 2016) even after 20 months. In this short amount of time, there have been over 1000 integrations, known as “Skills.” Some of the most impressive Skills are the replacement of existing interfaces. There are useful apps from reference lookups, news and stocks, home automation, travel, ordering goods and services, and of course, personal and social data.

Among the most popular Skills are Capital One’s offerings. Capital One has a dedicated mini-site focused on this functionality. Capital One is one of an elite cadre of ‘traditional’ companies that recognize and embrace the imperative to innovate and leverage new technologies to the benefit of their customers. They’re paving the way with their efforts, example, and important contributions to open source technologies. At the same time, though, rolling out new interaction schemes also brings challenges when integrating existing backend systems to new API driven-functionality such as those required by Alexa. Troubleshooting and ensuring a high-quality user experience needs a proper end-to-end view across multiple systems and technologies. That 10-second threshold is ambitious given the complexity of the systems involved., But as we’ve seen with the traditional web, as consumers adopt and grow comfortable with new technologies, the bar quickly goes higher — never in the opposite direction.

Market Share or Market Adoption?

2016-02-04T11:31:00.003-05:00

End user experience monitoring is the most important dimension of APM. In order for IT to become business and user aligned they must understand user experience, user journey, and their customer and user constituency. Gartner’s recent APM survey “Survey Analysis: End-User Experience Monitoring Is the Critical Dimension for Enterprise APM Consumers” by Cameron Haight finds the same thing; 46% of survey respondents ranked end user experience monitoring as #1. I found the same thing across the thousands of end user calls I took as an analyst.

We’ve seen massive shifts in open source over the last decade, driven by highly robust projects driven by dedicated companies and individual contributors. Open source causes issues when trying to judge market opportunity. The analyst firms focus on looking at revenue, but there is an untracked ecosystem out there.

The best way to analyze market share and market opportunity, and many times what execution should be is to analyze what is actually in use versus what is likely sitting on a shelf collecting dust. Thankfully today’s user experience products can actually be measured by crawling the web, which is exactly what companies like Datanyze, Netcraft, Builtwith, and Similartech do.

The best open source resource for technology usage on the web is the http archive due to the fact that it’s backed by leading companies like Google, but increasingly is being sponsored by APM companies. The data is also publicly available and can be analyzed by anyone. I use this resource often when testing hypotheses about the market as it provides an easy way to get questions answered about technologies in use, trends, and other changes on the web.

This past weekend I decided to do some trend analysis on the “market leading” APM companies out there to see what the adoption trends are. While many vendors sell a lot of software, much of it is unfortunately shelfware. End users increasingly pay for it due to bundles which are common with vendors like IBM, CA, BMC, and HP.

I began by looking at various javascript libraries included on various publicly accessible websites. My interest was in analyzing usage of the following products:

AppDynamics Browser
BMC Truesight EUEM
CA Technologies APM (Specifically BRUM)
Dynatrace Application Monitoring
Dynatrace Gomez Real User Monitoring
Dynatrace Ruxit Real User Monitoring
HP Diagnostics
IBM Tealeaf and APM (which has the same Javascript used for both APM and Marketing use cases)
New Relic Browser

CA Technologies

I was unable to find a consistent way CA products were deployed for JavaScript instrumentation. The CA website itself doesn’t even use their own product (that should tell you something). If someone shares the secret here, I can do the analysis. CA still doesn’t have much in terms of SaaS delivered real user monitoring or transactional APM.

HP Diagnostics

Similar to CA, while there are specific filenames in use for the HP products; there are none which are found in the data within HTTP archive. This either means no one is using the products publicly (which wouldn’t surprise me), or they are not doing it via JavaScript includes which would be a typical best practice. HP still lacks SaaS delivered real user monitoring and transactional APM is in beta.

HP and CA both have customers using packet analysis for APM, but most continue to move off those platforms based on large hardware expenditures (packet aggregation, switching, and server hardware to do the analysis). Additionally these technologies don’t work in modern applications (web or mobile based), especially those behind a CDN, hosted on public cloud, or in highly virtualized or container based infrastructure.

Dynatrace

Dynatrace as a company has it’s own set of issues with loads of overlapping technologies. You can even see in the single use case of end user experience monitoring, they have 3 distinct technologies and product offerings showing the level of portfolio fragmentation. Where will they invest? Which one is the right choice? The portfolio is an increasing mix of overlapping technologies, which must be corrected if they wish to remain competitive with leading vendors with a unified strategy (AppDynamics and New Relic).

Here are the charts showing the remaining vendors analysis. We have no way to differentiate between paid, trial, and freemium offerings here, so please keep that in mind.

New Relic has a massive installed base, since they launched their browser product in 2013. You can see momentum has slowed throughout 2015. In the last earnings call New Relic only had 5,285 customers paying more than $5,000 per year. The base New Relic Browser product costs $2,388 per year for 500,000 page views, which is quite a small site.

If we remove the New Relic numbers we get this graphic.

You’ll firstly notice that legacy vendor technologies are all on the decline, this includes BMC, Dynatrace Gomez, IBM Tealeaf. The clear investment is in the newer technologies such as AppDynamics, Ruxit (which is new, but gaining traction), and some Dynatrace installs.

If anyone wants the queries I used to collect this data, it’s all open data available on Google BigQuery, or you can download the data, and load it into your own MySQL database.

Please leave comments!

Update : 2/6/16

Here are some added players in the final graph. I included Pingdom, Akamai RUM, and SOASTA. Acceleration by SOASTA, slowdowns of Pingdom (Similar to what we're seeing with New Relic), and Akamai is still rather small.

I've also posted the source code for my queries here : https://github.com/jkowall/APM-BigQuery

Industry Insights: My thoughts from AppSphere 15

2015-12-24T14:17:00.001-05:00

Stuck in Vegas for two weeks, longing to be back home, it was a trial of might and perseverance. The first week was spent at AppDynamics AppSphere, and the second at the Gartner Data Center, Infrastructure & Operations Management Conference. (Look for a post next week on the Gartner conference.)

The second annual AppSphere drew 1500+ attendees, doubling the 2014 attendance. There was lots of passion and wonderful user engagement, but most importantly lots of exciting product announcements. The change from 2014 to 2015 was noticeable and impressive.

Having spoken at the first AppSphere on behalf of Gartner and feeling the energy at the conference, the momentum and acceleration one year later was apparent in the content, scale, and depth of the conference. In 2015, AppSphere expanded from one track to four tracks, and there was a large increase in customer and partner speakers. Product announcements included Browser Synthetic Monitoring, Server Infrastructure Monitoring, and new enhancements for session management, the C/C++ SDK, and many new AWS extensions. The most surprising feature customers were excited about was the ability to tie log messages automatically to the transactions that generate them, providing context that has been missing in the ITOA industry.

On the sessions side, there was a highly engaging customer presentation from HBO (must watch if you are a Game of Thrones fan). John Feiler, senior staff engineer, provided insight on the challenges of video streaming at scale. AppDynamics was used to correct issues before the season five premiere, and is used through the application development and operations lifecycle. This ensures a high quality viewing experience for all customers (including me).

Other great sessions included Pearson’s Mike Jackson and Tim Boberg. Pearson is a global leader in learning, and they use AppDynamics across the enterprise, serving over 1.3 million daily logins via over one billion page views every two months. Pearson was challenged in providing a standard way to see and monitor across many different technology stacks. Their tool of choice is AppDynamics.

BarclayCard’s Peter Gott explained how they were able to remove silos and get the organization to modernize and share data effectively using AppDynamics technologies, working across a highly variable technology stack and ultimately greatly improving user experience and uptime. Barclaycard has many more digital initiatives, including wearables and new mobile payment technologies, where AppDynamics plays a key role.

There were many other compelling and informative sessions. I wish I could have attended more of these — the audience loved hearing from customers, as did I!

There were also two great panels. I was honored to chair the microservices panel, featuring product managers from Red Hat, Microsoft, Google, and a heavy microservices user, DreamWorks studios. My colleague Prathap Dendi led a panel on IoT, which was particularly interesting, featuring customers from Garmin, Tesla, and SmartThings. Our partner Red Hat also participated in the IoT panel.

Partner participation included RedHat, Microsoft, Trace3, Bigpanda, Apica, ExtraHop, Capgemini, Scicom, Column Technologies, Moogsoft, xMatters, Electric Cloud, Neotys, Orasi, and mainframe partner DG Technology Consulting. We thank them and look forward to working with all of our existing and new partners throughout 2016.

We are looking forward to another record breaking AppSphere conference in November 14-17th 2016, we expect to once again double attendance, fingers crossed.

Industry Insights: Gartner Survey Analysis Critical Dimensions in APM

2015-11-19T14:23:00.000-05:00

Gartner analyst Cameron Haight did a survey Survey Analysis: End-User Experience Monitoring Is the Critical Dimension for Enterprise APM Consumers (Gartner clients only) over the summer across users about their APM implementations. There are several key areas where the data is quite telling.

Users main concerns

Users of APM tools based on this Gartner survey, and in my experience indicate 3 primary issues with APM tools.

APM tools are too expensive

This is a valid concern, APM is not inexpensive, but when compared to the cost of APM 7 years ago, the license costs fallen over 75%. Typical APM tools would cost $15,000 – $20,000 per server. Today we are looking at pricing closer to $4,000. These are list prices, which we know people do not necessarily pay. Additionally the time to value, implementation time, and spend for services which used to be typical in the past are a fraction of what they were once. I would estimate the cost of deployment and maintenance with a modern APM tool is 10% of the work of a legacy tool. This cost savings is not only hard dollar amount but results in faster ROI, and a shorter time to value.

APM tools are not well integrated

While this complaint is certainly the norm in the industry, at AppDynamics our unified monitoring approach addresses this concern. Building a single user interface and single installation for the platform which is leveraged across the monitoring capabilities. Additionally, by building integrations and facilitating an open ecosystem, connecting systems together becomes easier than the past. With modern software applications this has changed significantly.

Tools are too complex

Deployment, and usability within tools has been a major issue. At AppDynamics, we address this issue with our modern web based UI, easy deployment, and platform administration. This is coupled with a much smaller footprint for our controller (the monitoring server we require), requiring less resources to deploy (or just use SaaS and you don’t have to deal with the controller at all).

Breadth of APM implementation

25% of the 207 respondents say that APM is on 25% or more of their applications. I would have guessed this number to be much smaller. The question was not very clear if it was targeting critical applications or all applications. Users also have misconceptions about what APM is. When a server monitoring product can capture metrics about applications or application health it doesn’t make the tool an APM tool. Similarly, the use of older synthetic monitoring tools often qualifies for most as APM, when it does nothing to monitor the performance of users or transactions.

APM Buyers

APM tools are bought by a lot of different people in most organizations, we see everything from single user buying, departmental buying, and enterprise-wide buying. For many of the reason outlined above the enterprise motions tend to happen over time as APM gains traction. Most APM tools 5+ years ago were bought by landing big enterprise deals, but today most land in smaller deals and expand across enterprises. There have been many swings of the pendulum between suites of tools, mini-suites of tools, and best of breed. We’ve been in a best of breed cycle for the last 5 years, and this is likely due to the level of importance technology is playing within businesses as they digitally transform.

Based on Gartner survey data, buyers are opting for best of breed 59% of the time, which seems surprisingly low based on my conversations, and the fact that suite providers are significantly behind market leaders within APM. The only explanation for this is that those taking the survey may not currently have active APM initiatives, or do not understand what is and what is not APM. Haight then goes on to state “Enterprise APM consumers should deploy best-of-breed approaches as skills and finances dictate, but make sure to account for the potentially higher costs of integration.”. This is another area which large suite vendors sell on powerpoint, but upon implementation the integration is extremely challenging. The integration within suites of tools is incredibly expensive and complex to deploy and maintain, most organizations require heavy services outlay to implement and maintain integrations, even with suites of tools. These large suite vendors provide illusions of well integrated tooling, but based on first hand knowledge this is not the case.

The buyers are broken down as follows, which is accurate based on my time as an analyst and my time at AppDynamics.

IT Operations = 67%
App Support = 11%
App Dev = 8%
Other teams = 15%

The App support function is often part of IT Operations, hence differentiating between these two teams is often difficult.

Users also express concerns for legacy APM solutions to deal with large transformational shifts occurring through digital transformation. Once again, many vendors which sell legacy tooling are now messaging towards digital initiatives, when their tooling is not suited to accomplish these things.

Major shifts in the way software is designed, implemented, and analyzed are also shifting. The Gartner survey points out shifts most concerning to users include IoT, Cloud (SaaS), Mobile, and Microservices. There is clearly a large market opportunity for visibility in these areas, especially SaaS and IoT where APM vendors have limited ability to support these environments today. Although AppDynamics has many great customer use cases in Microservices, Mobile, IoT, and Cloud there is still a lack of technology with APM vendors to handle deep monitoring of SaaS along with having limitations when instrumenting IoT devices built on microcontroller architectures. These all present themselves as areas where our innovation and focus will create new ways to create visibility.

The key takeaway is that APM still has a long way to go to be as penetrated as it needs to be, and there are areas where major innovation has yet to occur to deal with new computing platforms, new connected devices, and new software architectures which will drive the next evolution of computing, data capture, and analytics.

What CIOs need to take away from Gartner Symposium

2015-10-16T08:16:00.002-04:00

Gartner Symposium is the high level CIO conference to attend. In this venue Gartner unveils new thinking including new predictions, surveys, and other insights into how people and technology will solve the problems of the future.

Highlights

Introduction to the algorithmic economy

The kickoff keynote by Peter Sondergaard the SVP leading Gartner’s Research business begun by imploring businesses to invest and create algorithms as the key unlocking insight into the vast amounts of data collected and generated. Peter went on to say how algorithms describe the way the world works, but in software, this is key to allowing systems to exchange data with one another (otherwise known as M2M). These algorithms will be used by agents such as Google Now, Siri (Apple), Cortana (Microsoft), and Alexa (Amazon). These are highly advanced algorithmically driven agents which use vast amounts of data both publicly and privately to provide an interactive and intuitive interface allowing them to predict what a user would want. These types of agent systems will define the post app world. Peter went on to describe how these opportunities are the future of computing, and represent a $1 trillion business opportunity.

The Robots Are Here

Darryl Plummer presented the trends for 2016 and beyond. They included the movement towards automated agents or robots. This includes what Gartner calls the digital mesh (devices, ambient experiences, and 3d printing) along with the application of smart machines (information, machine learning, and autonomous agents and things). These major changes will underpin the new reality IT must exist within. The cultural issues which I’ve seen transpiring in the media is the relationship between people and machines, right now this relationship is cooperative. The future holds codependency and ultimately competitiveness says Darryl.

The prediction is that by 2018 20% of all business content will be authored by machines. These machines will be fed by the IoT ecosystem of sensors, transport, and analytics. Gartner predicts that 1 million IoT devices will be purchased every hour in 2021. These not only create tremendous opportunity, but also incredible amounts of risk both in terms of personal information and security. Darryl goes with several daunting predictions around risk and security which will occur due to this digitization.

The final and most interesting trend which came up several times throughout Symposium is the use of smart agents as I mentioned above, Gartner predicts by 2020 smart agents will facilitate 40% of mobile interactions which will be the start of the post-app era.

2016 CIO Agenda and Survey

Dave Aron spoke about the freshly published 2016 CIO agenda and CIO survey. There was quite a bit of evolution here, Gartner introduced the digital platform:

The CIO survey included 2,944 CIOs managing $250B in IT spend. Clearly the revenue is already shifting towards digital today with CIOs estimating 16% of revenues are digital today. This is just the beginning of the journey, in 5 years the estimate is 37% of revenues will be digital. These digitalization changes are done to build efficiencies in operations, along with direct revenue generation.

The platform Dave laid out included aspects of bimodal delivery (on the bottom of the diagram above). The talent platform was one of the largest struggles CIOs mentioned in the survey. Skills shortages were the critical gap, and the talent shortage is being felt by CIOs. With fewer graduates in Computer Science over the last decade this shortage is likely to increase and continue.

The leadership platform also included insights into the chief digital officer. Dave touched on the fact that CDOs are being put into place, but at a slower rate than Gartner had predicted. This means that CIOs are taking on a lot of digital transformation responsibilities based on the survey data.

Private Cloud

I attended a session led by Donna Scott focused on private cloud. Based on Donna’s polling at Gartner infrastructure and operations management in June of this year 40% of respondents said they would put 80% of workloads in private cloud and 20% in public cloud. The main reasons were agility, speed, compliance, security, application performance, protecting IP, reducing costs, and dealing with politics. She then highlighted common use cases for hybrid cloud, or the use of public and private clouds at the same time. These can be split into more simplistic use cases such as running development in public cloud, while production is in private cloud or advanced use cases like active/active application resiliency.

Most successful private cloud implementations are done by type A companies who embrace hyperscale and web scale computing. Donna highlighted examples from companies such as Paypal, DirecTV, Chevron, McKesson, and Cox Automotive. The progression most companies make in cloud is highly tied to application maturity:

Jeffrey Immelt — GE CEO

The interview of GE CEO Jeffrey Immelt was quite interesting. GE has over 15,000 software engineers. GE is changing its business dramatically to build a platform for the industrial internet. Immelt believes that if GE can build and create the platform to use this data and the analytics to create greater efficiencies around industry (such as transportation where GE is one of the largest manufacturer of jet engines and rail locomotives) they can create a digital differentiator within their products, and allow other manufacturers to use their platform. He also outlined how the analytics will be used to create a “digital twin” for the physical assets within the analytics layer including all relevant data and metrics for each instance of machinery. One great quote resonated with me after seeing all of the M&A within the IT Operations space: “In 20 years of M&A, more value was destroyed than created by software acquisitions”

I also tried out the new Microsoft HoloLens, which was pretty nifty. We look forward to supporting that technology within AppDynamics as it matures and is brought to the enterprise market, which is where Microsoft has been focusing.

Symposium is a great high level conference, having spoken here twice but never able to attend the sessions I’ve gotten a new appreciation for this event both in terms of breadth and quality. I still believe that it’s often too broad for many vendors and attendees, but it’s a wonderful overview to what’s happening in our ever growing world of IT complexity and nuance.

Software will power the Internet of Things

2015-10-11T09:51:00.001-04:00

Today’s connected world is moving from devices towards things, what this means is that by using increasingly low cost sensors embedded in devices we can create many new use cases. These span across use cases in cities, vehicles, home, offices, factories, retail environments, worksites, health, logistics, and health. These use cases rely on ubiquitous connectivity and generate massive amounts of data at scale. These technologies enable new business opportunities, ways to optimize and automate, along with new ways to engage with users.

These technologies have been enabled by a perfect storm of technologies converging. They include both hardware, transport, and analytics.

Inexpensive sensors – As highlighted by this research by the ITAC http://itac.ca/uploads/events/execforum2010/rob_lineback_10-6-10-2.ppt sensor prices have been continually dropping.
Ubiquitous internet access – Powered by the pervasive mobile technology devices and sensors can be connected to the internet. Less than a decade ago, the powerful computers in our pockets did not exist. Powered by standard connectivity and protocols such as Bluetooth, Wifi, and NFC driven by protocols such as Zigbee and ubiquitous APIs.
Cloud technology – High speed on demand processing, storage and capabilities enabled by public cloud create the backbone for information collection and analysis on demand. These resources and platforms are easily accessible to all to collect data and provide insight into the usage of the thing.

What is the glue which makes all of this possible? Software is the key to IoT, this makes everything function together and creates these new capabilities and opportunities. This is why we believe seeing inside the software is key to visibility for purposes of troubleshooting and creating insight into the IoT. The complexity and scale issues presented by IoT on both the backend (in the cloud) and the frontend (things themselves) is a major challenge for not only the systems themselves, but for the management tooling of these interconnected and fluid systems.

Other key cautions for IoT include:

Software is not being managed properly, both in terms of availability and performance.
Data ownership of the data collected and mined.
Data security of collected data which can be used for malicious purposes.
Battery technology has largely not evolved for 15 years or more, this is a major limitation to today’s devices and connected things.

As a company AppDynamics believes that IoT will be a key part of computing and interconnected systems of the future, our customers are increasingly applying our technologies to these use cases, and we look forward to becoming an integral part of both collecting and analyzing data within these systems.

Industry Insights: Regulating failure (Reg SCI)

2015-08-05T09:31:00.001-04:00

When examining the complexity in today’s applications and environments, and why APM technologies are becoming more critical by the day, those responsible for an application’s lifecycle must understand what an application does. Aside from providing application visibility, APM tools help troubleshoot issues. The failure to see and troubleshoot is a constant struggle for those supporting applications. When I speak publicly I’m always able to point to specific instances of failure which affect each of us; this month the hot button items were travel issues regarding airline IT systems, stock market failures, and others.

Within the securities market, the SEC has adopted regulations that attempt to improve US securities markets’ ability to handle systems compliance and maintain integrity. To that effect, on November 19th, 2014, the SEC approved the adoption of the Regulation Systems Compliance and Integrity (Reg SCI) under the Securities Exchange Act of 1934. The regulation requires compliance by November 2015, in a few short months. This new regulation was specifically created to prevent or better handle issues and incidents related to flash traffic crashing exchanges, security breaches, and other areas of system resilience. The financial markets are increasingly interconnected, making cascading issues a reality. These regulated entities include FINRA, trading systems, plan processors, and clearing houses. The requirements of Reg SCI include creating procedures, executing testing, monitoring effectively and reporting data and status to the SEC. The reporting must be done on a regular basis, and when major systems changes occur. The entities covered by the new reg SCI mandates must also perform annual reviews including the testing of disaster recovery procedures of secondary sites, and their ability to handle the same volume of transactions with the same responsiveness as the primary sites. The focus is primarily on production systems, but also includes development and testing processes.

When outages do occur, there are specific provisions as to what must be reported, including the root cause of these outages. This helps share the reason for issues which affect the technology that powers the financial markets.

In regards to what APM focuses on, the regulation requires that capacity planning must be accomplished, but interestingly the capacity planning must be focused on transaction accuracy and timeliness to ensure market integrity. Most IT Operations professionals focus on infrastructure capacity planning, but this regulation clearly shifts that focus to the application layer. Stress testing must also be accomplished with major changes, once again requiring measurements. Reg SCI specifically notes that the monitoring of any 3rd party provided software or services, and how those systems perform is a requirement. Capabilities around monitoring availability and performance of these services is an APM technology capability, as 3rd party performance often affects application performance and proper execution.

AppDynamics is a trusted APM provider in many of the world’s largest banks and exchanges, and many more globally. We are also used within several companies which fall under Reg SCI. As a result, many of our customers are reaching out to us in order to comply with this new regulation. We’re pleased to discuss how we can help, and how AppDynamics is evolving to handle new types of capacity planning models in the future.

The actual regulation can be found here http://www.sec.gov/rules/final/2014/34-73639.pdf with commentary here http://www.sec.gov/rules/final/2014/staff-guidance-current-sci-industry-standards.pdf

Dynatrace Growth Misinformation

2015-06-19T12:15:00.000-04:00

For my valued readers: I wanted to point out some issues I’ve recently seen in the public domain. As a Gartner analyst, I heard many claims about 200% growth, and all kind of data points which have little basis in fact. When those vendors are asked what actual numbers they are basing those growth claims on, often the questions are dodged.

Dynatrace, recently used the Gartner name and brand in a press release.

In Its First Year as an Independent Company, Gartner Ranks Dynatrace #1 in APM Market

http://www.prweb.com/releases/2015/06/prweb12773790.htm

I want to clarify the issues in their statements based on the actual Gartner facts published by Gartner in its Market Share data:

Dynatrace says in their press release:

“expand globally with more than three times the revenue of other new generation APM vendors”

First, let’s look at how new the various technologies are:

Dynatrace Data Center RUM (DCRUM) is based on the Adlex technology acquired in 2005, but was created in the late 1990s. The DCRUM architecture has largely unchanged since the acquisition. The UI however has evolved over time.
Dynatrace Synthetic Monitoring is based on the acquisition of Gomez in 2009; that technology was similarly built in the late 1990s. The user interface has evolved, along with the use of the technologies acquired with Proxima technologies in 2007 to use for Business Service Management (BSM). Dynatrace did try to sell BSM for several years, but has moved away from that; they do still however use the technology within their synthetic monitoring solution..
Dynatrace Application Monitoring was acquired ($256m) in 2011, and also has a separate user interface.
Ruxit is an organically developed and modern offering with a modern user interface, which was released in 2014.
It was announced this week that Keynote will be merging into Dynatrace Synthetic, giving yet another user interface for synthetic monitoring which was developed in the 1990s.

These products are not modern. Dynatrace is not a “new generation” company, as in reality much of its revenue comes from technologies more than a decade old. Ruxit is clearly a “new generation” product, but has little revenue since it’s a new offering which is SaaS only.

“Dynatrace’s 2014 growth rate increased to 20.6%, nearly three times higher than the previous year’s growth and 24% faster than the overall global APM market.”

According to Gartner market data (“Market Share: All Software Markets, Worldwide, 2014”) Dynatrace’s growth rate in 2013 was 20.5%, while the overall APM market grew only 13%

In 2014, the Gartner published growth rate for Dynatrace was 20.6%, while the overall market grew 15.8%. So it can be approximated that three-quarters of Dynatrace’s growth was the overall growth of the market, from which all APM vendors benefited. Based on this, a reasonable inference is that Dynatrace’s growth numbers benefitted substantially as a result of the overall market growth in these periods.

Wanted to set the record straight, and use the data to shed some additional light on the Dynatrace press release.

Performance Monitoring, another view

2015-05-20T07:07:00.002-04:00

Originally posted on devops.com : http://devops.com/2015/05/20/performance-monitoring-another-view/

This is my first post to DevOps.com, for those who haven’t read my other writings, I currently work as the VP of Market Development and Insights at AppDynamics. I joined in February 2015 after being at Gartner for 4 years as a Research VP covering all things monitoring. During my time as a Gartner analyst I would help buyers make decisions about purchasing monitoring software. When I read articles on the internet which blatantly disregard best practices I either smell that something is fishy (sponsorship) or they just didn’t follow a process. Conversely some people do follow a proper process. I specifically wrote this up due to reading a really fishy article.

Step 1 – Admit it

The first step to determining you have a problem is admitting it. Most monitoring tool buyers realize they have too many tools, and none of them help isolate root cause. This is a result of most buying happening at the infrastructure layer, when the application layer is either lightly exercised synthetically or disregarded. People soon realize you must find an application layer monitoring tool, or write your own instrumentation. The vendors who sell logging software would love for you to generate a lot of rich logs, so they can help you solve problems, but realistically this doesn’t work in most enterprises as they are using a combination of custom and packaged software, and forcing developers to write consistent logging is a frustrating and futile task.

Step 2 – Document it

The next phase is to document what you are using today, most organizations have over 10 monitoring tools, and they typically want to buy another one. As they consider APM tools they should document what they have today, and determine a plan to consolidate and eliminate redundant tools. Having tools within silos will happen until you can fix the organizational structure.

Step 3 – Requirements

I can’t tell you how many times vendors set requirements for buyers. You can tell when vendors do this versus actually understanding the requirements the user has, and if the solution will scale and be operable by the team. Purchasing a tool which requires full time consultants on staff to keep it running is no longer feasible today, sorry to the old big ITOM vendors (formerly the big-4) but people don’t want that anymore.

Step 4 – Test it!

If you are evaluating software you MUST test it, and make sure it works in your environment. Better yet load test the software to determine if the vendors claims of overhead and visibility are in fact true. I see far too many people buy software they do not test.

Getting back to the article which prompted this post:

http://devops.com/features/devops-style-performance-monitoring-for-net/

The author is located in Austria and I believe there is a connection to Dynatrace. Dynatraces’ R&D and technical leadership are in Austria, but this is just conjecture. The article is written with a bias, which is obvious to any reader who has used these tools. The evaluator did not test these products, or if he did he didn’t have a testing methodology.

Finally the author’s employer is new and all employees have come from Automic Software in the past year. I don’t need to explain to those who work with Automic the state of things there. http://www.glassdoor.com/Reviews/Automic-Software-Reviews-E269431.htm#

I think I’ve spent enough time on the background, here are my specific issues with the content.

The screenshots are taken from various websites, and are not current or accurate. If someone writes an evaluation of technology, at least test it, and use your screenshots from the testing!

This post was specifically to discuss DevOps, but yet the author doesn’t discuss the need to monitor microservices or asynchronous application patterns, which are clearly the current architecture of choice. This is especially the case in DevOps or continuous release environments. In fact Microsoft themselves announced new products last week at Ignite including nano server, and Azure Fabric Service for building these patterns. The difficulty with microservices is that each request externally spawns a large number of internal requests, these often start and stop out of order creating issues when monitoring these. This causes major issues for APM products, and very few handle this effectively today.

The article calls out user interface and deployment models inconsistently, and fails to mention Dynatrace is an on-premises software product. Dynatrace is also a thick Java client, built on Eclipse. This is the definition of a heavy client, which the author calls out for other products in this article.

Continuing the incorrect facts in this article the author calls out Dell (formerly Quest) Foglight. This is not Dell’s current product for modern architectures. Dell has moved to a SaaS product called Foglight APM SaaS, which is not what was evaluated. The evaluator should look at the current product (noticing a trend here).

Finally the AppDynamics review is inaccurate:

Editors note: the following is this authors opinions, not the opinions of DevOps.com. As the author states, he is employed by AppDynamics.

“code level visibility may be limited here to reduce overhead.”

The default in order to scale AppDynamics to monitor thousands of systems with one single on premises controller (something no other APM vendor delivers) is done based on advanced smart instrumentation, but every transaction is monitored and measured. All products do limit the amount of data they request in some way. AppDynamics has several modes including one which captures full call traces for every execution.

“ lack of IIS request visibility”

AppDynamics provides excellent visibility to IIS requests, I’m not sure what the author means here. AppDynamics also supports implementation on Azure using nuget.

“features walk clock time with no CPU breakdown”

Once again, I’m not sure what the author is referring to here. The product provides several ways of viewing CPU usage.

“Its flash-based web UI and rather cumbersome agent configuration”

The agent installation is no different than other products, showing the author has not installed the product. Finally the product is HTML5, and has been for a while, there are still some flash views in configuration screens, but those are being removed with each release. I’d much rather have a web based UI with a little remaining flash than a fat client UI requiring a large download.

AppDynamics has a single UI which goes far deeper into database performance than the other products in this review. AppDynamics also provides far more capabilities than what was highlighted in this article including synthetic. AppDynamics provides deployment of the same software via SaaS or on premises installation.

Finally this review did not look at analytics, which is clearly an area of increasing demand within APM. That being said this review is far from factual or useful.

Hopefully this sets some of this straight, please leave comments here, or @jkowall on twitter!

Thanks.

An Open Response to the Open Letter To Monitoring/Metrics/Alerting Companies

2015-05-07T13:49:00.000-04:00

Response : http://www.kitchensoap.com/2015/05/01/openlettertomonitoringproducts/

John we couldn’t agree more with this letter, the specific issue is context. Too many of the tools and systems in use both commercially and with open source are metric or event (log) collectors, providing dashboards with little context about what is happening. In order to provide the proper context and operational visibility one must understand relationships and data flows between metrics and events.

This well written letter makes many points we completely agree with at AppDynamics. The use of words predictive or fixing issues automatically are not something we prescribe. Gartner has also long condoned the use of predictive in ITOA scenarios (“IT Operations Analytics Technology Requires Planning and Training” Will Cappelli December 2012). The area we disagree with is having early warning indicators of problems which are escalating. If technology is employed which collects end user experience from the browser and that performance is baselined by geography, as degradation occurs across the user community this often is an early warning indicator that something is behaving abnormally. We have customers who have seen a vast reduction in complete outages (P1 issues), and an increase in degraded service issues (P2 issues). This means we have evidence that the use of AppDynamics can in fact reduce the number of outages by providing early warning indicators. We have other evidence showing legacy enterprise monitoring tools are far too slow, this is a coupling of older technology, and organizational or process issues. This prevents the alerts from getting into the right hands in a timely manner. For example in a enterprise with siloed teams and tools, a storage contention bottleneck on a particular array would often be seen by the storage team, but lack of application operations visibility and escalation in a timely manner would result in service issues. This of course can be solved by fixing organization issues, but that is a challenge at scale.

CIO: Changing Sourcing Strategies

2015-05-04T06:52:00.004-04:00

Sourcing strategies are evolving rapidly. CIOs have to handle this shift, where innovation isn't coming from the usual vendors. There is a clear movement toward those who can facilitate the most critical agendas to enable digital business. New organizational structures are required that allow for a culture of innovation and experimentation. In order to innovate, teams that typically consist of larger, slower-moving units must transform into small, startup-type or project-based units.

AppDynamics CEO, Jyoti Bansal, often speaks about his methodology for creating a startup within a startup. Many new product initiatives begin with a small team of two-to-four engineers and product managers tasked with building a minimum viable product. In this model startup teams are augmented with small acquisitions which come with talent. This new talent provides new perspectives, skills, and core intellectual property allowing the acquirer to meet customer needs quickly. This is similar to the model employed at Google, Facebook, Yahoo, or Twitter allowing for incubation and experimentation with new, unorthodox ideas. At Google, some of these new ideas become winners like Gmail and Google Apps for business, while many fail. Some of these new products take several iterations before they can be productized. Google Talk is one example that was allowed to “sunset” but was reborn as Google Hangouts, which has had rampant adoption with its second iteration and integration of other services. Google Hangouts on Android has over 500 million installations.

This is the new pace of innovation, something CIOs must now replicate in their own operations. A Gartner headline says, "Digital Business Economy is Resulting in Every Business Unit Becoming a Technology Startup." What this means is if you do not evolve your organization, thinking, and sourcing, you will be unable to compete. The furious and often large acquisition strategy, results in non-integrated technologies which fail to compete in a time where technology becomes a major advantage in digital transformations.

Within IT operations management, an area I've studied and tracked for over two decades, we saw many large acquisitions over the years from HP, IBM, BMC, and CA. These acquisitions provided growth and breadth, by adding new capabilities that were slotted into a portfolio. That growth strategy though, which had once been effective, ended up creating too much complexity. The complexity required services, which in turn led to challenges in staying current.

This technology debt and lag do not meet the needs of today's technology buyers. The large acquisition strategy has slowly fallen off; meanwhile, these big players have had major challenges making the transition to organic innovation. Gartner predicts that "By 2017, at least two of the "big four" IT operations management vendors will cease to be viable strategic partners for 60 percent of I&O organizations." We've already seen this play out with the privatization of yesterday's innovators and ongoing execution issues with others.

New innovative upstarts in ITOM run at a different speeds, allowing them to create, build, and provide the value customers want, without the heavy services, consulting, and integration burdens. These new vendors have highly differentiated strategies and approaches that keep them innovative and give them a competitive advantage.

The question remains for CIOs: Of these innovative vendors, which will be able to continue down the innovation path as they move from being medium-sized companies to become large companies? As revenues of these innovative vendors continue to increase substantially, innovation at scale remains an open question. But at AppDynamics, we're focused on culture and strategy to enable and retain our innovation as we build the next generation ITOM technology for software-defined businesses.

The End of my Affair with Apdex

2015-05-04T06:52:00.002-04:00

A decade ago, when I first learned of Apdex, it was thanks to a wonderful technology partner, Coradiant. At the time, I was running IT operations and web operations, and brought Coradiant into the fold. Coradiant was ahead of its time, providing end-user experience monitoring capabilities via packet analysis. The network-based approach was effective in a day when the web was less rich. Coradiant was one of the first companies to embed Apdex in its products.

As a user of APM tools, I was looking for the ultimate KPI, and the concept of Apdex resonated with me and my senior management. A single magical number gave us an idea of how well development, QA, and operations were doing in terms of user experience and performance. Had I found the metric to rule all metrics? I thought I had, and I was a fan of Apdex for many years leading up to 2012, when I started to dig into the true calculations behind this magical number.

As my colleague Jim Hirschauer pointed out in a 2013 blog post, the Apdex index is calculated by putting the number of satisfied versus tolerating requests into a formula. The definition of a user being "satisfied" or "tolerating" has to do with a lot more than just performance, but the applied use cases for Apdex are unfortunately focused on performance only. Performance is still a critical criterion, but the definition of satisfied or tolerating is situational.

I'm currently writing this from 28,000 feet above northern Florida, over barely usable in-flight internet, which makes me wish I had a 56k modem. I am tolerating the latency and bandwidth, but not the $32 I paid for this horrible experience , but hey, at least Twitter and email work. I self-classify as an "un-tolerating" user, but I am happy with some connectivity. People who know me will tell you I have a bandwidth and network problem. Hence, my level of a tolerable network connection is abnormal. My Apdex score would be far different than the average user due to my personal perspective, as would the business user versus the consumer, based on their specific situation as they use an application. Other criteria that affect satisfaction include the type of device in use and connection type of that device.

The thing that is missing from Apdex is the notion of a service level. There are two ways to manage service level agreements. First, a service level may be calculated, as we do at AppDynamics with our baselines. Secondarily, it may be a static threshold, which the customer expects; we support this use case in our analytics product. These two ways of calculating an SLA cover the right ways to measure and score performance.

This is AppDynamics’ Transaction Analytics Breakdown for users who had errors or poor user experience over the last week, and their SLA class:

Simplistic SLAs are in the core APM product. Here is a view showing requests that were below the calculated baseline, showing which were in SLA violation.

The notion of combining an SLA with Apdex will result in a meaningful number being generated. Unfortunately, I cannot take credit for this idea. Alain Cohen, one of the brightest minds in performance analysis, was the co-founder and CTO (almost co-CEO) of OPNET. Alain discussed his ideas with me around this new performance index concept called OpDex, which fixes many of the ApDex flaws by applying an SLA. Unfortunately, Alain is no longer solving performance problems for customers; he's decided to take his skills and talents elsewhere after a nice payout.

Alain shared his OpDex plan with me in 2011; thankfully all of the details are outlined in this patent, which was granted in 2013. But OPNET's great run of innovation has ended, and Riverbed has failed to pick up where they left off, but at least they have patents to show for these good ideas and concepts.

The other issue with Apdex is that users are being ignored by the formula. CoScale outlined this issues in a detailed blog post.They explain that histograms are far better ways to analyze a variant population. This is no different than looking at performance metrics coming from the infrastructure layer, but the use of histograms and heat charts tend to provide much better visual analysis.

AppDynamics employs automated baselines for every metric collected, and measures based on deviations out of the box. We also support static SLA thresholds as needed. Visually, AppDynamics has a lot of options including viewing data in histograms, looking at percentiles, and providing an advanced analytics platform for whatever use cases our users come up with. We believe these are valid approaches to the downsides of using Apdex extensively in a product, which has it’s set of downsides.

Back to my blog

2015-04-15T10:50:00.000-04:00

I'm back to posting content in my personal blog again, I was provided a personal blog at Gartner and could express myself there. I've made a couple updates to my old blog, and will begin posting new content here on a regular basis. Some of this will be posts written on the AppDynamics blog, while others will be new and original content.

Appreciate all of the readers!