This is especially important as the focus shifts from long QA/user acceptance testing cycles to rapid identification and resolution of issues in production, and deployment of the fixed application back into production. This rapid identify-fix-deploy loop requires adoption of new tools and processes to be successful.

It will be increasingly important to have sharp insight into applications running in production. Without it, you will miss quality goals, have higher maintenance costs, and lower customer satisfaction. With it, you can prioritize work based on actual usage patterns, identify, triage and resolve problems before your customers are seriously impacted. You can also test changes to see how they affect user behavior and intended outcome, and drive both hard and soft costs to a minimum.

Collecting, analyzing and acting on application runtime data poses unique challenges both in terms of the types of data that need to be gathered and the metrics that measure success. Effective application analytics implementations must accommodate the diversity of today’s applications and the emergence of cloud, mobile and distributed computing platforms. Narrower analytics technologies such as standard reports provided in a cloud service will never fully satisfy development and management objectives for corporations.

Existing analytic solutions have almost exclusively resided in the cloud. This makes perfect sense from a technological implementation standpoint for the analytics vendor. However, for companies with sensitive data or that are constrained by government regulation, storing your data “in the cloud” is simply not an option. The only appropriate application analytic solution is one where data can be surfaced on a variety of endpoints (on premise and/or off premise) according to client-specific rules for compliance with relevant industry standards and regulations.

Comprehensive application analytics must support enterprise, B2B and B2C use cases including cloud, servers, web-based, traditional PC and mobile apps – and the data should stream within a private network or across public networks as well.

Our application analytics solutions achieve that objective. Let’s look at the pieces:

• PreEmptive Analytics for TFS is a “Client-premises” or on-premises incident response solution that connects production incidents to development and operations via automated, intelligent, rule-driven creation and management of work items to decrease the mean time to fix an application.
• PreEmptive Analytics Runtime Intelligence Service is a managed, multi-tenant service providing broad analytics and archival services – it’s a hassle-free, always up, analytics platform ideally suited to measure the most common metrics and KPIs.
• PreEmptive Application Analytics Workbench is an on-premises solution that provides critical insight into the adoption, usage, performance, and impact of production applications to facilitate feedback-driven development and enhance software quality, user experience, and decrease the mean time to improve an application.

At this point you might be wondering which of these tools might be most useful to you now. That is where the Data Hub shines brightly.

The PreEmptive Analytics Data Hub is a client-premises endpoint that can be installed internally, on a “DMZ” server, or in the cloud - and it serves as the “one endpoint” for all of your applications, across all of our services – even as you expand and adjust your analytics strategies and implementations. The Data Hub monitors runtime data and routes that data to any/all other PreEmptive Analytics software and services (including other Data Hubs). The Data Hub is an enterprise-scale runtime data management and distribution service providing resilience (caching, retry and commit) and flexibility across architectures and platforms.

So you can instrument your apps, send them to the one endpoint you need, the Data Hub, and then slide in one or more of the available analytics solutions (including 3rd party solutions) that best meet your requirements. If your analytics toolset changes, you can make any necessary adjustment without having to re-instrument or redistribute your applications. Applications that do not have privacy or regulatory concerns could have runtime data forwarded to the “cloud”. And, Analytics for Applications that touch more sensitive data can be kept internally only. Runtime data can be sent to more than one place, providing a set of checks and balances. Flexible, powerful, secure, actionable… You can have your cake and eat it too.

The Issue at Hand
One adoption/monetization “variable” that is rarely measured in a clean way is the impact/influence that an app’s marketplace can have on the success of the app itself. This is in large part a practical issue – it’s not easy to compare, for example, Apple’s App Store with Google Play because the apps themselves are often quite distinct from one another – and so isolating the marketplace influence from the apps themselves can be tricky. However, with Android, we publish identical apps through two very different marketplaces; Amazon’s Android App Store and Google’s Google Play marketplace. By focusing on apps that are identical in every way BUT the API calls to the respective marketplaces, we can start to drill into the direct and indirect consequences of marketplace selection.

Android makes up roughly 51% of TheMobileYogi downloads.

Android downloads combine both Amazon and Google Play adoption.

Android Downloads of Yoga-pedia
As of January 29, 2012, the total downloads of Yoga-pedia were:

Google Play downloads were 75% greater than from Amazon.

…But downloads only tell a very small part of the story. What are users doing AFTER they download the app? How often do they use the app, for how long, and what exactly are they doing when they are inside?

Yoga-pedia Sessions
Using PreEmptive Analytics Runtime Intelligence, we see that there are in fact striking differences between the Google Play user population and the Amazon user population.

One glaring difference is the total number of users in each community.

The total unique users of from Google Play is 208% higher than that of Amazon.

If we were to stop here, I think our conclusion would be obvious – Google play delivers more downloads and more unique users than Amazon – and that has to make it a clear winner right? (Note, there has been no difference in marketing, advertising, etc. between the two marketplaces – specifically, we have done none).

…but if we were to stop here, we would be making a very big mistake!

How much time is spent inside the app?
Another glaring difference that our analytics reveal is the difference between the average session length of our users – Amazon users tend to stay inside the app almost 3 times longer!

So – if we multiply the total number of sessions by the average session length, we can calculate how many hours were spent inside Yoga-pedia.

Total time spent inside the app distributed through the Amazon marketplace is 40% higher than from Google Play.

If I am trying to maximize ad impressions, establish a brand or hold my user’s attention toward some other objective, Amazon now looks significantly more attractive to me than Google Play.

User Behavior
Since Amazon users spend so much more time inside Yoga-pedia – how is their behavior different and how does that translate into measurable value?

Returning Users

Returning users (in orange) form the majority of the Amazon session activity – Google Play users are less likely to use the app multiple times – they are ‘tire kickers’ for the most part. Returning users are roughly equivalent across the two marketplaces even though there are many more Google Play users overall.

Returning users are loyal and a lasting “relationship” can be established – whether you’re selling something, hoping to influence their behavior, or tap their expertise – recurring users are always “premium.”

Ad Click Through Rate (CTR)

Moving to a more concrete metric – we can compare total impressions, Ad Click through Rates (CTR) as well as Ad Server Errors – for this analysis, we’re just looking at 30 days. Note: in both cases, the apps use AdMob.

Amazon CTR is 164% higher than the Google Play CTR

Google Play Ad Delivery Failure rate is (ADFR) 199% higher than the Amazon ADFR

Now, it’s not really possible to isolate WHY these differences exist – but we can make some educated guesses. For CTR percentages – are Amazon users simply more conditioned or likely to buy stuff as compared to the typical Google Play user?

For ADFR percentages, we’re using the same ad service API, so the ad service itself is not to blame. Are the devices being used by Google Play users (as a total population) of lower quality or are they connecting through networks that are not as reliable?

Regardless, that kind of conversion delta is nothing to ignore.

Upgrades

As I’ve already mentioned, in addition to pushing ads, Yoga-pedia is one half of a freemium model where we hope to get these users to upgrade to our commercial version, A Pose for That.

With PreEmptive Analytics, I’ve instrumented the app to track the feature that takes a user back to their respective marketplace (positioned on the app upgrade page). The ratio of unique users (not sessions) to upgrade clicks tells another important story; how likely is an Amazon user versus a Google Play user to upgrade to our paid app?

Amazon user conversion rate is 54% higher than the Google Play conversion rate.

User Behavior Within My App

Yoga-pedia offers its users two locations where a user can click to upgrade; in a “tell me more” about the premium app page and at the end of an “Intro” to the current Yoga-pedia app.

By looking at the split of where users are more likely to “convert,” we can learn something important about the app’s design in general AND the differences between user patterns across marketplaces in particular. As a proportion, Amazon users are more likely to convert from the Intro page than their Google Play counterparts. The Intro page is “deeper” in the app (harder to find) and so this difference in usage pattern may imply a more thorough reading of embedded pages by Amazon users (and this would be supported by the much longer session times).

Exceptions
Exceptions not only interrupt a user’s experience (with all of the bad things that flow from that), they are also a material expense (support, development, etc.). Given that we are talking about two virtually identical apps – would we expect one version to be more unstable (and therefore expensive) than the other?

Whether or not we expected it, the Google Play version of Yoga-pedia has an error rate per session that is 15% higher than its Amazon equivalent.

Again – the analytics at this level can’t tell us why – but we can still make an educated guess regarding the differences in phone type and network stability of the two populations.

Detail

Of course, if you want to drill down into the specific exceptions (and examine stack traces, device types, carriers, etc – all of that is available through analytics as well.

Here are exception details for the error rates described above. Anyone want to help me debug these?

Do Marketplaces Matter? Of Course They Do.
Of course, different apps will yield different results – but I don’t think that there can be any question that each marketplace comes with its own unique bundle of user experience, service level, and general appeal – and that, taken together, these attract their own distinct constituencies (communities) with their own behaviors, likes, dislikes and demographics.

App developers who chose to ignore the market, commerce and security characteristics that come with each marketplace will do so at their peril – the differences are real, they should influence your design and marketing requirements, and they will undoubtedly impact your bottom line and your chances of delivering a truly successful app.

Today, in a strong move to protect American privacy, the Federal Trade Commission published the report Mobile Privacy Disclosures: Building Trust Through Transparency (PDF). For those that don’t want to read the entire report, checkout the coverage in the NY Times: F.T.C. Suggests Privacy Guidelines for Mobile Apps for a nice overview (not sure how long that link will be live though).

The take away from my perspective is this – while app marketplaces like Apple and Google and advertising services like Flurry continue to fall under increasing scrutiny, the app developer is no longer flying under the radar or going to be given a pass for not understanding the rapidly emerging policies, recommended practices and general principles.

From the referenced NY Times article above…

“We‘ve been looking at privacy issues for decades,” said Jon Leibowitz, the F.T.C. chairman. “But this is necessary because so much commerce is moving to mobile, and many of the rules and practices in the mobile space are sort of like the Wild West.”

and…

The F.T.C. also has its sights on thousands of small businesses that create apps that smartphone users can download for a specific service. The introduction of the iPhone created a sort of gold rush among start-ups to create apps featuring games, music, maps and consumer services like shopping and social networking.

“This says if you’re outside the recommended behavior, you’re at a higher risk of enforcement action,” said Mary Ellen Callahan, a partner at Jenner & Block and former chief privacy officer for the Department of Homeland Security.

Even before this report, “the F.T.C. has not been meek,” said Lisa J. Sotto, managing partner of Hunton & Williams in New York. “They have brought a number of enforcement actions,” she said. “Those in the mobile ecosystem know they’re in the regulators’ sights.”

…but do app developers really know?

In an earlier post of mine, COPPAesthetics: Form Follows Function Yet Again, I lay out in more detail both the privacy concepts that the FTC is developing and the technical and functional capabilities (and business models) that distinguish application analytics from the other analytics categories out there. These features include opt-in policy enforcement (for both regular usage and exception handling), encryption on the wire, greater control of data collection and more…

COPPA is a much more formal set of requirements to protect children with severe sentencing guidelines and a growing set of precedents where app developers are being fined with increasing regularity – BUT there is little doubt that the FTC is not limiting itself to children’s rights – in its latest report, the FTC recommends that:

“App developers should provide just-in-time disclosures and obtain affirmative express consent when collecting sensitive information outside the platform’s API, such as financial, health, or children’s data or sharing sensitive data with third parties.” (Page 29 of the report)

If you’re building mobile apps or services that support mobile apps and have been “getting by” using marketplace and marketing analytics services to get user and app usage feedback – be very careful – expect these services to become more and more restrictive – (even dropping apps that appear to be too risky). They will (rightly so) limit their data collection to fall within (and probably well within) regulatory constraints leaving developers to operate their apps “in the dark.” (or assume the risk of non-compliance)

Again from the NY Times article: “Morgan Reed, executive director of the Association for Competitive Technology, a trade group representing app developers, said that the organization generally supported the commission’s report but that it had some concerns about what he called “unintended consequences.” If app stores are worried about their own liability over whether they have adequately checked the privacy protections of a mobile app they sell, they might err on the side of caution and not screen for privacy at all, he said.”

App developers are welcome to collect runtime data necessary to operate (and improve) their applications (see my COPPA post for more clarity here) – collecting data usually only becomes an issue when that data is shared or used for other purposes or by other parties – and that is at the heart of application analytics and what distinguishes it from its peers.

Application analytics is all about improving application quality, ensuring operational excellence and delivering a superlative user experience – there is no ulterior motive or agenda.

On a separate track, I have long been calling out how the design intent (functional requirements) behind the various analytics technologies out there lead to profound (material) differences in architecture and feature sets (form). For a more technical treatment of this topic, see Application Analytics: what every developer should know.

Debuggers, profilers, web/mobile analytics, and last (but not least ;) application analytics are distinct from one another – and this latest update to COPPA serves as one more stark reminder as to why these distinctions really matter.

Information that is exempt from COPPA obligations include data that is used exclusively to “Support internal operations” including “maintaining or analyzing the functioning of app or service.” So, right away we can assume that debuggers and system monitoring software are still in the clear – but these tools do nothing to improve usability, measure adoption, identify user preferences, and – in most cases – cannot even reach out onto consumer devices like mobile phones and tablets.

Today, app/service providers (let’s call them “Owners” for now) have had two options for analytics inside their apps and online services;

Functionally, these two technologies are quite different and are complimentary (see the article listed above) – and another stark difference between these two categories is the preferred business model of their respective vendors.

All of this has potentially serious implications for both app/online service providers AND advertising networks.

Advertising network service providers are likely to simply ban (drop) app owners that may send them COPPA-governed data to avoid risk – and we see this happening already; checkout Flurry’s privacy policy here where they write (in part)

“Our Customers may not use the Flurry Services in connection with any application labeled or described as a “Kids” or “Children” application and may not use the Flurry Services a) in connection with any application, advertisement or service directed towards children or b) to collect any personal information from children.”

Application analytics service providers find themselves in a very different position. I need to stress that “application analytics” and “mobile analytics” are not equivalent – the latter will not promote an app or rank an app against competitors.

Yet, to the extent that measuring user experience, preferences and behavior leads to improved adoption and sales and to the extent that reducing mean-time-to-repair improves operations and increases satisfaction and application value – application analytics will, by design, continue to improve application quality and value in a safe and secure manner - even under increasing regulatory oversight and scrutiny.

Dotfuscator for Windows Phone 7: PreEmptive Solutions will be extending complimentary access to Dotfuscator for Windows Phone 7 through April, 2013. This means that existing users will continue to have access to world-class obfuscation for their WP7 apps; also, new users will continue to have complimentary access as well.

Dotfuscator Professional: Dotfuscator Professional currently supports obfuscation for Windows Phone 7 (WP7), Windows Phone 8 (WP8) and WinRT. The professional SKU will soon include the injection of application instrumentation for WP8 and WinRT as well (WP7 is already supported). This professional SKU also supports all flavors and versions of the .NET Framework beyond mobile including Windows 8.

Announcement

Dotfuscator for Windows Phone and WinRT: This new version of Dotfuscator will be available in early 2013 and will support all modern surfaces including WP7, WP8 and WinRT apps.

Special pricing: This WP/WinRT version of Dotfuscator, let’s call it Dotfuscator WW for now, will be offered in two ways; one for the individual hobbyist or “moonlighter” and another for development teams requiring higher levels of support and more flexible deployment options. In both cases, pricing will be set at a significantly lower price point as compared to our standard Professional SKU (due to the unique nature of this “consumer-focused” market as well as the limited functionality of the Dotfuscator WW SKU, e.g. it only operates on WinRT and Windows Phone apps).

Generally speaking, pricing will “net out” to $500 per developer per year. One thing that we believe hobbyists will appreciate is the option of subscribing for as short as a month at a time (at $50 per dev per month). This means that a hobbyist app developer can license Dotfuscator WW for just those months where obfuscation is required for release. If you release your app twice a year, that means you will be able to obfuscate your code for 2 X $50 or $100 per year – you will not have to pay the $500 per year annual fee (of course, if you need Dotfuscator for > 10 months per year, it will be cheaper to do the annual subscription).

Conclusion

For those developers targeting Windows Phone 7 - you are all set through April 2013.

Look for Dotfuscator for WinRT and Windows Phone in 2013 that offers IP protection across all of Microsoft’s modern surfaces priced to accommodate the “modern developer.”

Can’t wait to protect your WP8 or WinRT code? Contact us at sales@preemptive.com and I’m sure we will be able to work something out.

FAQ:

Will my Dotfuscator for WP7 expire at the end of 2012?
No – it will expire on 4/30/2013.

If I am targeting WP7 and obfuscating with Dotfuscator WP7, will my app run on a WP8 phone?
Yes!

When will Dotfuscator for WP and WINRT be available?
We are not committing to a specific date at this time, but the date will be well before the expiry of Dotfuscator for WP7 – developers will have plenty of time to make the transition if they so desire.

Is there any support for Windows Phone 8 or WinRT today?
Yes, Dotfuscator Professional supports these and all other .NET flavors. For pricing and availability, you can contact sales@preemptive.com.

Will there be a free version of Dotfuscator for Windows Phone after 4/30/2013?
Only Dotfuscator Community Edition (CE) included with Visual Studio will be free after 4/30/2013. Dotfuscator CE 2012 (the version embedded inside Visual Studio 2012) can obfuscate Windows Phone assemblies, but it cannot automatically consume XAP files and so there are additional steps required to make this work. Dotfuscator CE 2012 cannot obfuscate WinRT apps at all.

What about instrumentation and exception monitoring?
There has been no complimentary access to PreEmptive’s Runtime Intelligence Service since Microsoft discontinued its funding for the Windows Phone development community.

However, if a developer injects exception monitoring specifically (versus feature tracking) via Dotfuscator, developers can now take advantage of PreEmptive Analytics for TFS that is included as a part of Visual Studio and TFS 2012.

If you don’t already use TFS, check with Microsoft as they have a number of very attractive campaigns to make TFS available at no cost to small development teams. Of course, for a subscription fee, PreEmptive Solutions supports session, feature, user and exception monitoring and analytics across all .NET runtimes and devices as well as JavaScript, C++, Java and Android.

For more information on PreEmptive Analytics, visit www.preemptive.com/pa

Will Dotfuscator for WinRT and Windows Phone be able to obfuscate or instrument other .NET applications?
No. The full Dotfuscator Professional will be required.

What is the difference between a development team license and a hobbyist license?
The final details have not been published yet, but the distinction should be fairly intuitive; if you are developing an app on your own or even with a few buddies to see if you can make a mark in this new and exciting world of modern apps – and you’re doing this outside of your “day job” – you are most definitely a hobbyist – no matter how successful your work may be – and we hope it is wildly successful. If you’re building an app for your employer – then you’re going to require a development team option (and you’re going to want it too for your own requirements). We will of course be crystal clear on all of this as we officially release Dotfuscator WW and, as always, if you have any immediate questions on this, email sales@preemptive.com and someone will get back to you promptly.

Early last week, Microsoft posted the first release from their new quarterly release cadence for Visual Studio and Team Foundation Server. This new cadence means great things for users and partners – faster bug fixes, earlier delivery of new features, and a chance to be more engaged with Microsoft’s product development cycles.

But this creates a special challenge for us because we actually ship inside Visual Studio and Team Foundation Server. With this new cadence, we will often only have a few weeks between the time when we discover a breaking change and the time when it is available to users, i.e. every developer with a copy of Visual Studio 2012 or Team Foundation Server 2012.

This isn’t an insurmountable problem – we already use agile practices (Scrum + dev practices) and have fairly lightweight release processes – but it does mean that we have to tighten those practices and modify how we communicate and distribute our updates.

And there’s nothing like a real-world example to test out a theory! We recently discovered that our web components (within PreEmptive Analytics for TFS) require an update under TFS 2012 Update 1 – both for our Professional SKU and for our Community Edition (the version inside Visual Studio).

The good news is that it only took us a few weeks to identify the issue, determine the fix, schedule the work, and create a patch. (Great job, team!) Now we need to get this patch out to our users, and that’s where this is new ground – we don’t have a direct way to reach all our Community Edition users of PreEmptive Analytics for TFS. So our plan is to post a notice on the changelog (CE and Pro), write this blog post, and ask Microsoft to mention it via their development blogs. If you have suggestions for how else we can spread the word, please let us know!

To download the patch, you’ll need to be registered on our website, which is easy to do. If you’re a Professional user, you are probably already registered. If you’re using Community Edition, it was already a good idea to register so we can alert you to updates like this one, and provide free resources, etc. Once you’re registered, you can download the patch and follow the instructions to install it. The same patch will work for Professional and Community Edition users, and we posted localized versions for our Community Edition users.

Of course, new users of our Professional SKU won’t have to install this patch, but Community Edition users will have to apply patches like this one for the foreseeable future.

Coming back to our new release processes – we think the new quarterly release cadence is a good thing for us and our users, but it means we have to adjust our quality, communication, and operational processes in order to maintain our standards while meeting these new goals. Now that we’ve done that, we’d love to hear your feedback about how it went, or how you think it should go the next time around. Please post comments on this post if you have any feedback or ideas. (And thanks in advance!)

Mean Time To Repair (MTTR) is the interval from an unwanted event in production to the root cause being fixed and the service (or application) redeployed and in use by the customer.

A goal for IT is to decrease the MTTR for a service or application while possibly at the same time embracing the concepts of frequent delivery and deployment of software. A rapid diagnose-fix-deploy loop requires adoption of new tools and processes to be successful. Since 80% of MTTR may be spent in the diagnosis phase, it can be useful to focus on that first “diagnose” step in the loop.

Tools like PreEmptive Analytics can help development teams identify the most important and widespread issues in deployed applications even before users report them, and set the issues up to be quickly resolved.

PreEmptive Analytics collects and analyzes production incidents and automatically creates (and updates) work items within ALM tools such as Team Foundation Server. When development teams have easy and complete access to production incidents they can make great decisions about how to triage and repair problems. That saves time, money and makes for delighted users. Isn’t that what every development shop wants?

Companies recognize that they need to provide continuous quality and eliminate waste in their software development process. This is especially important as the focus shifts from long QA/user acceptance testing cycles to rapid identification and resolution of issues in production, and deployment of the fixed application back into production. This rapid identify-fix-deploy loop requires adoption of new tools and processes to be successful.

There are complimentary “devops” tools shipping in Visual Studio and TFS 2012 that help reduce cycle times and continuously improve quality. Let’s look at them:

SCOM Application Performance Monitoring

Use SCOM Application Performance Monitoring if you own Microsoft’s System Center Operations Manager 2012 and you have internal applications you wish to monitor such as Internet Information Services (IIS)-hosted .NET applications. The .NET Application Performance Monitoring (APM) template in System Center Operations Manager 2012 allows you to get details about application performance and reliability that can help you pinpoint root causes of incidents. You specify settings, the types of events to collect, the performance goals to measure, and servers to monitor and .NET Application Performance Monitoring reveals how applications are running. You can see how frequently a problem is occurring, how a server was performing when a problem occurred, and the chain of events related to the slow request or method that is raising exceptions.

PreEmptive Analytics

Use PreEmptive Analytics for Team Foundation Server (PA/TFS) if you have a broader set of applications you wish to monitor that are:

• Enterprise, B2B, or B2C: In and across enterprise, partner, & client domains.
• Heterogeneous: .NET, Java, JavaScript, Windows Native including WP, Android, and WinRT.
• Distributed: Client/Server, Mobile, Azure, hosted inside and across firewalls and domains.

PreEmptive Analytics allows you to see, as they happen, the errors your users are experiencing. By aggregating and analyzing exceptions reported by your production applications and automatically creating TFS work items, PA/TFS gives you access to incident data right where you need it – in Visual Studio, TFS Web, and Reporting Services. The work items include the exception type, message, full stack trace, the contents of any inner exceptions, and the list of all assemblies loaded at the time of the exception, as well as the number of such exceptions received. With a small amount of additional configuration, you will also be able to see unique instance identifiers (such as the serial number of each application instance that experienced the exception) as well as comments about the error and contact information, if provided by the end user. PreEmptive Analytics is always on (like Radar) and supports a broad set of “real world” scenarios efficiently, securely and reliably.

Intellitrace

Intellitrace is more like a CAT scan. It is able to provide “deep dives” into complex systems and is well suited when a problem is already known but not fully understood. As an example, if you are alerted to an issue in production with PreEmptive Analytics and you don’t have enough information yet to fully solve the problem, you could use Intellitrace to further dig in. To do this, start IntelliTrace data collection, reproduce the problem, and then stop collection. You can take the resulting log that IntelliTrace creates, open it in Visual Studio 2012 Ultimate, and analyze it. Intellitrace is great for debugging errors that are hard to reproduce or that happen outside Visual Studio, for example, in production or other environments.

DevOps tools and services improve software quality and reduce time to resolution by generating actionable bugs from deployed software. Visual Studio and TFS 2012 include three complimentary tools that help you dramatically reduce time for solving production errors. Take advantage of these and start building better software, faster.

Most everyone agrees that analytics are powerful - it’s why they’re powerful that gets interesting. 77% of development and their management identified “insight into production application usage” as influential, important or essential to their work, and 71% identified “near real-time notification of unhandled, caught, and/or thrown exceptions” in the same way (other choices were “moderately important” and “no importance”).

…but where specifically do application analytics have the greatest impact?

Usage, behavior and patterns

Figure 1: Where does insight into production application usage matter? (click to expand)

Developers need to know where and how to prioritize the work that’s right in front of them and nothing makes supporting users more straightforward than having direct insight into what they’ve been doing in production.  

While third in the cumulative vote count, Product planning was ranked 1^st in the “essential” categorization. If you don’t know what’s happening around you, there’s no way you can confidentially plan for the future.

Unhandled, thrown and caught exceptions

Figure 2: Where does insight into production incidents (all manner of exception) matter? (click to expand)

Not surprisingly, everyone can agree that insight into exceptions and failures in production provide critical insight into how future iterations of an application should be tested. The fact that 22% of respondents did NOT see exception analytics as being at least influential in customer support is somewhat surprising and will be the subject of future analysis – however, one potential explanation may lie in the obstacles development organizations face (or perceive) in actually implementing true feedback-driven customer support and development processes.

What’s getting in the way?

When comparing usage versus exception monitoring, respondents are mostly consistent in their ranking of obstacles – in fact, the consistency is striking when you consider the divergence in ranking of use cases across these two categories (usage versus exception monitoring).

Figure 3: What are the obstacles preventing development organizations from implementing effective application analytics solutions today? (click to expand)

While specific numbers vary somewhat, development, product owners and management focus first on security and privacy concerns (see my last post) – followed closely by performance and stability – let’s call that Quality with a capital “q” and “Lack of Best Practices,” which is understandable as application analytics is only now emerging alongside new platforms, tools and methodologies.

PreEmptive Solutions and Application Analytics

What the respondents’ agreement in “obstacles” also indicates is that it should be possible for a single technology solution combined with appropriate processes and patterns designed to address these obstacles to meet the user and organizational requirements across all of these use cases and scenarios.  …and, coincidentally that is exactly what PreEmptive Analytics has been built to accomplish.

For more information on PreEmptive Analytics, visit www.preemptive.com/pa

For an article I wrote for MSDN and the launch of Visual Studio 2012, checkout Application Analytics, what every developer should know.

In a recent survey that includes responses from 100’s of development organizations, two thirds identified application analytics as either essential or important in one or more of the following categories: Product planning, Development prioritization, Test plan definition, Customer support, and/or Development ROI calculation.

Among this group where application analytics has the greatest impact, the following were identified as the most serious obstacles to implementation. (click to enlarge graphic)

Half of all respondents identified security and privacy a 20% higher response rate than the next two closest obstacles e.g. lack of expertise and general quality concerns).

The emphasis on security and privacy is even more pronounced inside larger development teams. Nearly 3 out of every 4 development organization with greater than 50 identified privacy and security as an impediment – 50% more likely than development teams between 5 and 15.

In fact, an organization’s size appears to have a significant influence on virtually every perceived obstacle; larger organizations appear to be more concerned with performance, quality and connectivity while smaller organizations struggle with awareness of analytics solutions, development best practices, and the required integration of their development and operations processes.

One might make the generalization that, due to the complexities that come with size, larger organizations have had to move to more tightly integrated platforms and practices – putting them in a better position to implement application analytics (and so they focus on potential risks stemming from an implementation) whereas smaller teams may not have as an entrenched “feedback-driven” integrated approach to development. As such, they are more likely to struggle with how to move forward (keep in mind that all respondents identified application analytics as either essential or important).

Privacy and Security and PreEmptive Analytics

Regardless of development team size, privacy and security is the number one perceived obstacle – and PreEmptive Analytics is unique in its approach to this critical requirement. PreEmptive Analytics includes the following:

• Development teams own their own data. PreEmptive asks for no rights to aggregate, inspect or resell your data.
• A two-level opt-in switch is included ensuring user opt-in to transmit runtime data from both regular usage AND application exceptions. The logic itself can be injected post-build for .Net and Java and can always be defined by the development organization.
• All data is, by default, encrypted on the wire.
• Tamper-detection and defense can be used to detect and defend against any attempt to alter or redirect runtime data transmission.
• Obfuscation can be used to obscure inspection by third parties of what is being collected and transmitted.
• Unique keys identify both the organization and the application source for data.

For more information on how PreEmptive Analytics addresses the number one obstacle for implementing application analytics (as ranked by those that need it the most), visit www.preemptive.com/pa.