I’m personally focused on the professions of developers and architects, so that’s what I’ll talk about today. We all rely on abstractions, but making the right decisions about the level of abstractions we use is what differentiates a great developer or architect from a mediocre one. This isn't a new discussion, and comes up often with both new and older technologies. I often argue that developers and architects should be on the more conservative side here. Why is this?

I think it's important to take a couple of things into account when making these recommendations.

1. What is the potential downside of implementing without full understanding?

2. How much does implementation facilitate learning?

These two considerations weigh against each other when trying to decide if it makes sense to implement without deep understanding. If implementation will help us learn about and avoid the tradeoffs and downsides of a technology, then this is an argument for learning by doing. If potential downsides are very large, then this is an argument against. Remember, these attributes weigh against each other, so even if potential downside is small, if the learning upside is also small, it may still make sense to avoid an abstraction.

All of this must, of course, be weighed against other factors such as overall value (which includes downside). So this consideration is just one of many that needs to be taken into account.

So what of generative AI? Unfortunately here the potential downsides are fairly high and the ability to learn through abstraction is quite low. The prompt itself is an abstraction that seems to work against learning what is really going on in generative AI models such as LLMs, and which itself spawns troublesome secondary abstractions such as anthropomorphizing LLMs and certain types of magical thinking about what LLMs are really capable of.

I believe the reason for this lack of learning is that the primary failure modes of LLMs are usually invisible to non-experts. Hallucinations can only be spotted if the user already has good knowledge of the domain, is able to think critically about it, and is willing to make the effort to do their own research. Similarly, bias in all it's forms, including racism and sexism, is invisible at least as often as it is visible, and both filter models and fine-tuning have been deployed on LLMs to make bias even more invisible than it would otherwise be.

For these reasons, I strongly urge developers and architects intending to deploy these models in products and infrastructure to first gain a good understanding of how these models really work. This is not as difficult as it may seem. Bea Stollnitz's blogs on these topics are accessible, detailed, and link to the original papers on these topics, which themselves are usually written in a largely accessible manner for those with some technical background. Other resources and explainers have also become available if you prefer a different style or want to know about a different model (Stollnitz mostly focuses on GPT LLMs).

It’s important to remember that this recommendation is for developers and architects looking to deploy these models in products. For developers and architects trying to use these models as day-to-day assistants to augment their own capabilities, the calculus is quite different! Developers are domain experts in their own domain, so they are in a much better position to spot and address hallucinations. Additionally, racism and sexism biases in these models are unlikely to express themselves in a damaging way when we are trying to program algorithms or get feedback on coding practices and style from an LLM like Github CoPilot. Because the downsides are mitigated and smaller respectively, it is much more likely for a developer to be able to responsibly and productively use tools like Github CoPilot or ChatGPT as assistants in their professional work. The same is true for most domain experts.

Yesterday I tweeted about web scraping, mentioning that it's always a bad idea. What I mean by "web scraping" is the practice of using automated tools to harvest data from a website. Usually this is done in order to try to gather data for research or for use in a product.

I’ve done some web scrapping but I’m more and more convinced it is always a bad idea.

— Ethan Jewett (@esjewett) July 23, 2016

That was, I'm sure a bit too strong. After all, universals are always incorrect. But I think it's basically the right approach and I'd like to explain a little bit why that is.

The problems with scraping

Web scraping is problematic for several reasons, among them:

• Legality, in that scraping is usually prohibited by terms of service, and the use of scraped content is usually a copyright violation baring fair use exceptions. So in scraping you are risking running afoul of wonderful laws like (in the US) the CFAA and DMCA. If you've heard of Aaron Schwartz, you've heard of someone who got slammed with the CFAA for web scraping.
• Ethics, in that regardless of law it can be unethical to use content that doesn't belong to you, and in scraping you risk bypassing restrictions on content use that may have been put in place intentionally and which you may not even be aware of. Remember, the fact that information is technically accessible, or even public, doesn't mean that it is not sensitive. Example: Your whereabouts in a public street is technically public information, at least in the US. Your neighbor waving at you when you leave your house is great. Someone sitting outside your house 24/7 and tweeting whenever you come and go is grounds for a restraining order.
• Technical and financial concerns such as denial of service. You don't know what kind of capacity the website you have has available, nor how they are paying for it. It could well be that every request you send costs the website incremental money, or that their server will simply crash due to your scraping. You can't know this ahead of time if you haven't consulted with the person or organization running the website. You can be careful, and you can even respect technical indicators like robots.txt files, but you still won't really know unless you coordinate.

Instructors: Keep in mind that when you indicate to your students that it is OK to web scrape to gather data for their projects, you are opening them up to real liability. Encouraging your students to violate the CFAA is no small thing, no matter how bad a law it is.

What are the alternatives?

First, I'd like to remind us all to keep in mind that we are not generally entitled to use resources we don't own in the service of our own projects. Now, there are certainly situations in which using resources without the owner's permission can be justified, and we'll get to that down around step 6 or 7, but all other things being equal, if there is a better option, you should use it.

My recommendation when looking to acquire bulk data for research purposes is to do the following in this order:

1. Look for bulk download options provided by the content owner. It is very common to see people talk about scraping sites like Wikipedia, IMDB, Github, or StackOverflow. Guess what! No need, as all of these sites provide bulk data download options.
2. If no bulk download is available, see if there is an API that might provide what you need within the API terms of use.
3. Check the terms of service or user agreement to see if by some chance it actually explicitly allows automated scraping in some form. If it does, then go ahead, but this is a rare case.
4. Email the organization or person running the site and ask if it would be possible to arrange access to the data you are interested in.
   1. You may want to do this well ahead of the time you'll need the data so that there is time for negotiation, legal agreements, etc.
   2. Simultaneously, you may want to ask around among your network and advisors as to whether there is a version of the bulk data available already that you may have missed.
5. If the organization is not willing to allow use of their data, see if there is an alternative data set that could serve the same purpose. Note that this may require changing your research plan. That's fine! Part of building a good research plan is being able to execute it ethically and legally. It's a good skill to develop.
6. Consult a lawyer. And an ethicist if there is anything conceivably sensitive about the data you are accessing.
7. If you've determined that the merits of your need outweigh the legal and ethical issues around ownership and permission, and you are willing to accept the potential consequences of being wrong, scrape.

But...

... some people won't agree with the ethical points above, and keep in mind that I'm not a lawyer, nor do I really know anything about the law in this area. I'm laying out a fairly conservative position. If you disagree and your eyes are open to the possible consequences of being wrong, ranging from a mild slap on the wrist to career suicide and possible jail time, then who am I to stop you?

But if you are an instructor teaching a class, a lead researcher advising your subordinates, or an employee or contractor doing a project for a company, there is more at stake than just yourself. You could well get your students in trouble or subject your company to a damaging lawsuit without realizing that what you are doing is problematic. So think twice and look for alternatives.

I've found Crossfilter to be a very powerful tool for building applications that support data interactions that feel 'real-time'. The idea is that <100ms interactions are fast enough to allow us to identify patterns and correlations as the visualization changes in exact coordination with our filtering. I use Crossfilter in several places, but the biggest project is easily Palladio, which is a testbed platform for research activities and a powerful example of the possibilities of browser-based data exploration tools.

Crossfilter consistently trips up people trying to use it for the first time because, I believe, its programming model for aggregations is inconsistent with the model we usually expect. It's for this reason that I started Reductio, a library that helps build up aggregation functions that work correctly and efficiently with Crossfilter's model.

I'm not going to get into all of the details here, but defining aggregations in Crossfilter requires defining functions that incrementally update the aggregation based on the addition or removal of records. This is at odds with the standard way of computing aggregations by building them up from scratch that we see in SQL or more standard map-reduce models.*

// A function that can be passed to Array.reduce to sum the array
function(a, b) {
  return a + b;
}

// The equivalent operation in Crossfilter requires 3 functions
function reduceAdd(p, v) {
  return p + v.total;
}

function reduceRemove(p, v) {
  return p - v.total;
}

function reduceInitial() {
  return 0;
}

It's these aggregation functions that consistently trip people up. The summation case is simple enough because summation is an operation that is computationally equivalent given the running total and the value to be added or removed. But what about operations that are not reversible? Take the computation of a maximum. When adding a new record to a maximum aggregation, we just need to check if the value of the record is larger than the current largest value we've seen (the current maximum).

// A function that can be passed to Array.reduce to return the max
function (a, b) {
  return Math.max(a, b);
}

But if we remove a record with a value equal to the current maximum, we need to know the next smallest value. And if we remove the record with that value we need to know the next smallest, and so on. We have to keep track of all values seen in order to avoid needing to rebuild the entire aggregation from scratch when a record is removed. And yes, this is significantly faster than rebuilding from scratch!

// The equivalent operation in Crossfilter (optimized, with
// error guards). Adapted from Reductio
var bisect = crossfilter.bisect.by(function(d) { return d; }).left;

function reduceAdd(p, v) {
    i = bisect(p.values, v.number, 0, p.values.length);
    p.values.splice(i, 0, v.number);
    p.max = p.values[p.values.length - 1];
    return p;
}

function reduceRemove(p, v) {
    i = bisect(p.values, v.number, 0, p.values.length);
    p.values.splice(i, 1);
    
    // Check for undefined.
    if(p.values.length === 0) {
        p.max = undefined;
        return p;
    }
 
    p.max = p.values[p.values.length - 1];
    return p;
}

function reduceInitial() {
  return { values: [], max: 0 };
}

It's a lot of work to come up with these aggregation functions, make them efficient, and test them thoroughly, and Crossfilter beginners (myself included) consistently struggle with this. Believe it or not, things get even nastier when dealing with thing like pivot- or exception-aggregations, when building aggregations where a record counts in more than one aggregation group (like moving averages, or tags), or when trying to compose multiple aggregations (commonly required for visualizations as simple as a scatter-plot). Reductio supports all of these aggregations and a few more, and I hope it will allow a quicker, smoother start when using Crossfilter.

* This difference in model is to allow very fast interactions with data and aggregations. If you aren't building an application that requires real-time interactions and are thinking about using Crossfilter simply to do one-time calculations or aggregations on your data, think again about using Crossfilter as it is probably not the right tool for the job.

Permalink

]]>Ethan JewettWed, 04 Feb 2015 22:07:45 +0000http://www.esjewett.com/blog/skating-to-where-the-puck-is500c2012e4b0d820d6a397d2:500c2137e4b0d19362bd3c28:54d29838e4b0771432488abdBeing an SAP Mentor is an interesting situation. The relationship with SAP usually feels amazingly collaborative, but occasionally uncomfortably adversarial. ThIs is probably a healthy place for a relationship with a massive organization to be. Every organization includes many great people and interactions with those people are always delightful. Every organization is also, as an organization, predisposed to get the answers to certain questions consistently right and the answers to other sorts of questions consistently wrong.

One such question that SAP has consistently gotten wrong in the past is how to push technology adoption among a massive install base. Historically, it would appear that SAP has tended to try to monetize upgrades to underlying technology in order to recoup development costs or profit from its technology offerings. Netweaver, BW, Visual Composer, add-ons like SSO and Gateway, Personas, Business Client, and HANA* are all technologies in which SAP has either underinvested due to lack of direct revenue potential (BW especially) or has tried to monetize. The result is that widely deployed "free" technology from SAP often stagnates while advanced technology with an additional cost often sees limited adoption in SAP's install base.

This outcome is terrible for SAP's business. It makes it very difficult for SAP to keep its application offerings competitive in the marketplace. The reason is that basic application functionality to be competitive is often dependent on improvements in underlying technology. But if technology is either widely deployed and under-featured or decently featured but not widely deployed, then applications need to use only the under-featured technology stack in order to have a large enough potential install base to justify development costs. In other words, SAP often forces itself to build applications on sub-par technology.

We see this dynamic constantly with SAP. One example is UI technology. Clearly a long-standing weakness of SAP's applications has been user-interface and user-experience. The SAP GUI is outdated and SAP's attempts to improve on it like the Business Client have, in my opinion, been torpedoed by SAP's monetization reflex. They haven't initially been better enough to see widespread adoption under a monetization scheme, and with lack of adoption, investment has faded. A similar dynamic played out with SAP's support for Flash and Silverlight as UI technologies for years after it had become clear they were destined for the trash-heap of web-UI delivery technologies.

And yet, over the last 4 years, SAP has been able to overcome this tendency in one area that may be incredibly important to its long-term business prospects around the newly announced S/4HANA. In the case of a key succession of technologies, SAP has been able to do something different, with impressive results. Those technologies: Gateway, SAP UI5, and Fiori**.

Initially, all seemed to be going well. SAP developed Gateway in part due to prodding from influencers like SAP Mentors around the need for a modern, RESTful API layer in SAP applications to allow the development of custom user interfaces and add-ons in a sustainable manner. DJ Adams and others showed the way with projects like the Alternative Dispatch Layer (in 2009), to make development of these APIs easier. Uwe Fetzer taught the ABAP stack to speak JSON. And suddenly one could fairly easily create modern-looking APIs on SAP systems. SAP folded a lot of those learnings into Gateway, which was a great way to push the tools for building modern APIs into SAP's applications and out to the install-base.

Well, it would have been, but SAP made its usual mistake: it attempted to monetize Gateway.

The result of the monetization attempt could be predicted well enough. It would make roll-out of Gateway to the SAP install base slow, at best. Fiori would be delayed because it would need to build out its own API layer rather than relying on Gateway technology. Applications like Simple Finance or S/4HANA that are dependent on Fiori would subsequently be delayed, if they were created at all. Perhaps Fiori's roll-out is delayed by a year, Simple Finance by 2 years, and S/4HANA isn't announced until 2018.

But S/4HANA was announced in January 2015. So what went differently this time?

Fortunately, back in 2011, shortly after the initial Gateway monetization strategy was announced, SAP Mentors like Graham Robinson and other community members stepped in to explain why this was a mistake and push back against the strategy, both publicly and privately. While clearly not the only reason for SAP's change of heart, this feedback from the community was powerful, and ultimately SAP revised Gateway licensing in 2012 so that it made sense for SAP, its partners, and customers to build new applications using Gateway.

This revision set us on the path to relatively quick uptake of SAP UI5 (a modern browser-based front-end framework which leverages the Gateway APIs) and later Fiori and the Fiori apps (most of which are based on UI5 and Gateway APIs). With Fiori, SAP again thought short-term and attempted to monetize Fiori, only reversing course and including Fiori with existing Business Suite licenses after similar pressure from another group of SAP Mentors who had experienced customers' frustration with SAP's stagnant user-experience. These Mentors, community members, and analysts were able to communicate the role Fiori needed to play in guarding against SAP customers migrating to cloud vendors offering a more compelling user-experience story.

Fiori, meanwhile, is a hit and serves as the basis for Simple Finance and now S/4HANA - SAP's recently announced blockbuster refresh of the entire Business Suite offering, which SAP clearly hopes will drive its business for the next 10 years. Would that be happening now if SAP had remained on its standard course of attempting to monetize Gateway? I don't think so. The interaction with SAP on these topics often left some feathers a bit ruffled, but it sure must be nice for those like Graham and DJ to see some of the fruits of those discussions in major products like S/4HANA.

*A note on HANA: I think that HANA may be the exception in this story. Unlikely most of SAP's other technology offerings, HANA is good enough to be competitive on its own, and not simply as a platform for SAP's applications. The results are not yet in on the HANA monetization strategy, but things are looking OK, if not great. Of course, Graham has something to say about that, and what he says is always worth reading.

** Fiori is actually a design language focused on user-experience and a line of mini-applications implementing the Fiori design language to improve SAP Business Suite user-experience for specific business processes. However in the context of S/4HANA we can think of Fiori as a prerequisite and enabling component, like a technology prerequisite.

the initial worry - wasted effort

The idea of a data lake is that we establish a single huge repository to store all data in the enterprise in its original format. The idea is that the processes of data transformation, cleansing, and normalization result in loss of information that may be useful at a later date. To avoid this loss of information, we should store the original data rather than transforming it before storing it.

This is a relatively noble goal, but it overlooks the fact that, done correctly, these processes also introduce clarity and reduce false signals that are latent in the raw data. If every data analysis project needs to recreate the transformation and cleansing process before doing their analysis, this will result in duplicated work and errors. Long live the datawarehouse.

The deeper problem - adding information

In my reading, this sums up the usual justification for data lakes as well as the source of frustration usually expressed by data professionals. But I think there is another issue with data lakes that is often overlooked: that the data lake approach implies that transformation is an information-negative operation. That transforming necessarily discards data, and therefore information, from the original data set. It is a response to a common frustration with datawarehousing - that the data in the warehouse doesn't quite answer the question we are trying to ask and if only we could get at the source data we could reconstruct the dataset so as to be able to answer the problem. Sometimes true.

Usually, however, there are real issues with this approach. The transformation from raw data to datawarehouse (or other intermediate representation)  may remove information, but it adds other significant information to the source data. Specifically, it adds or acts on information about what data can go together and how it can be combined, how the data collection method may have influenced the data, sometimes weights data according to external information about its quality, might add interpolated data, etc. Usually this additional information is not stored along with the source data set. It's rare that it's stored in an accessible way along with the resulting data in a datawarehouse. I've seen almost no mention of maintaining this type of information in the context of a data lake. It is simply lost, or at best difficult to access.

Existential questions - If a tree falls...

@esjewett @BoobBoo @oswaldxxl what is data and what is noise is an existential question :)

— Vijay Vijayasankar (@vijayasankarv) January 31, 2015

Which brings me to the big lie around data lakes: storing everything. If a measurement is never taken, is it data? How many measurements are not taken for each measurement that is? Can a data lake store these un-measured measures?

The issue in data analysis, I find, is not so much that data to answer our question was collected but discarded. Rather, the problem is that the data was never collected, the measurements never taken, or taken improperly for use in addressing the question now at hand. Like the traveller in Frost's poem, we have limited resources. Collecting one measure is a road that precludes collecting other measures due to these limits. Storing everything, or even most things, is not possible.

Later, we often wonder if taking the other road would have allowed us to answer our current questions, but we can't go back and make different decisions about measurement. Data lakes don't change that. Maybe data lake vendors should switch to selling time machines.

Palladio is a platform for data visualization and exploration designed for use by humanities researchers. We're in early beta at the moment and will be doing a series of releases throughout 2014. You can read about it and try it out here: http://palladio.designhumanities.org/

I think the website does a pretty good job of explaining the capabilities of the platform, so I'll leave that for the moment. I encourage you to go check it out before reading on, because it will be worth understanding how the platform works to give some context to the discussion below.

So, why am I super-excited about this? Mostly because of the great team, which has the vision, technical skills, theoretical and domain knowledge, and information design chops to pull off this type of project. I consider myself lucky to be able to work with this group.

It's also great to be working on a project like this for a field that is simultaneously very strong on information theory and a bit underserved in terms of some types of tools. This is in stark contrast to my usual enterprise data management and visualization work where the theory tends to be weak but a plethora of tools exist.

In addition to trying to build a tool that incorporates important and underserved aspects of humanistic inquiry, I am excited to work with a team that buys into introducing state-of-the-art concepts around data exploration tools in general. Many of the concepts we are working to implement in Palladio are directly applicable to the types of data exploration problems we find in the enterprise and are concepts rarely expressed in existing tools. Palladio is a great example (one of many great examples) of how the process of humanistic inquiry can motivate the development of methods that are both technically and conceptually applicable in wildly different disciplines.

Interaction

The thing that initially most impresses people about Palladio is the way that filtering and movement are integral to the visualization. Specifically, the visualizations update and move in real-time as you filter. This is not a new concept, but I don't think I've ever seen it fully implemented in a general-purpose tool. Getting the level of movement right is a design challenge that the team is tackling as a work in progress, but in my opinion this characteristic of real-time updates and movement is a key feature for a data exploration tool, and few if any tools implement it.

I'll try not to get too squishy here, but this behavior of the tool allows a person to interact with the data in a very direct way, giving a feel for the data that would not otherwise exist. When you can see the results of your interactions with the data in real time, it is a lot easier to conceptually link step-changes and interesting events with the actions that caused them. For example, dragging a filter along the timeline component allows you to play back history at your own speed, speeding up or slowing down as suits you. My theory-foo is weak, but when you see it, I think you'll understand. Try it out with the sample data.

Browser

Techy alert: Palladio is a purely client-side, browser-based application. The only involvement of a server is to deliver the HTML, Javascript, and CSS files that comprise Palladio. We arrived at this design through a few iterations, but the motivation was that we wanted to be cross-platform, super-easy to install, and still support pretty big data sets and fluid interactions. 10 years ago, this would have been nearly impossible, but today we have web browsers that, amazingly, are capable of supporting these types of applications. Yes, browsers can now support dynamically filtering, aggregating, and displaying 10s and 100s of thousands of rows of data and displaying hundreds of data points simultaneously in SVG; thousands of data points if you use canvas instead.

The time for client-side data visualization in the browser has come and we are taking advantage of that in a big way. A great strength of browser-rendered visualizations is that they allow true interaction with the visualization. Just using SVG or Canvas as a nicer replacement for static images is fine, but it isn't fully exploiting the medium. Add to this that the type of interactivity we are providing with Palladio is technically impossible in a client-server setup. Even if the server responds to queries instantaneously, the round-trip time the client-server communication introduces means that interactions won't be as closely linked as they are in Palladio, severely degrading the quality of the interactive experience.

Admittedly, we have work to do on performance and our cross-browser support could be better. Additionally, the problem of data that simply doesn't fit in the browser's memory remains unaddressed, though we have some ideas for mitigating the problem. But I think this is an application design approach that could be exploited for the vast majority of data sets out there, either because the data itself is relatively small, or through judicial use of pre-aggregation to avoid performance and size issues.

Design

Lastly, user experience and information design have been integral components of this project from the start. The design has been overhauled several times along the way, and I wouldn't be at all surprised if it happened again. To be clear, I'm a complete design newb, but we have a great designer working on the team. One thing that has become clear to me through this process is that designing a general purpose interactive visualization tool is hard. There are more corner-cases than I previously imagined possible, but we are trying to get the big pieces right and I think we're on the road to success.

Obviously the organizational dynamics on a small team like ours are very different than those in a big development organization, but it seems like information design on most of the enterprise data exploration tools from larger vendors either started out problematic and stayed that way, or started out pretty well and started slipping as the tool took off. I'm not sure if there is an answer to this, but it's clear that when building a tool in this space, having at least one information designer with a strong voice on the team is indispensable.

Let me sum up

So, that's the Palladio project, as well as a few takeaways that I feel can be applied back to pretty much any data exploration project. In closing, I'll just mention that none of this would be possible without some great open source projects that we use and, most importantly, without the great team working on this as well as the feedback and patience of our dedicated beta participants. The central Javascript libraries we used to pull this off are d3.js, Crossfilter, and Angular.js. The members of the core team are Nicole Coleman (our fearless leader), Giorgio Caviglia (design and visualization), Mark Braude (documentation, testing, working with our beta users, and project management), and myself doing technical implementation work. Dan Edelstein is the principal investigator.

It's been a great ride so far and we've got some exciting things planned for the rest of the year. This is definitely a work in progress, and feedback is very welcome. Follow the Humanities + Design lab on Twitter for updates.

This discussion, and the fact that I'm back doing some serious ABAP development after a lot of non-ABAP work, got me thinking about the value SAP's application ecosystem contains that developers would love to add to, to SAP's gain. But it's very difficult to add value in this ecosystem as a developer because of (among other things) the weaknesses of the ABAP development environment. What, I asked myself, would a truly strong ABAP development environment would look like. Here are a few thoughts:

• ABAP system as a decentralized version control system (DVCS) node - An ABAP system should act as a true DVCS node. This is a good explanation of the nirvana you too could experience with a DVCS, with pictures of a cute dog. In short, an ABAP system should, at the package level or at the full system level, be able to commit changes, push those changes to a remote repository, pull changes from a remote repository, maintain branches, and merge (with user input if necessary) those branches. These descriptions use Git parlance, because that is what I know.
• Branch and merge - Related to behaving as a DVCS, the ABAP system should be able to support full branch & merge semantics. I'm not sure how well I can explain this. I'll just tell you, it's a great feeling to have the ability to create a branch of a complex project, rip out an integral component, completely rewrite it, merge those changes back into the main development branch, and have none of the other 3 people working on closely related parts of the project notice any change expect that things just work better. I know. I did it last week. Oh, and I simultaneously made fixes as they were requested on other branches of the project and pushed those out to my co-workers while I was working on that rip-and-replace. Good luck pulling that off on an ABAP development system.
• Namespaces - SAP should make namespaces available to open source projects for free, with minimal bureaucracy. And while you're at it, just do the same for everyone. What do I mean by "minimal bureaucracy"? A web form and account system that saves the namespace to a database after checking that no one else has already registered the same namespace. Why are namespaces important? They address the problem of naming collisions between development projects. Right now, the open source ABAP community mostly develops in the customer namespace. This is a recipe for naming-collision hell. But what other options do people have? SAP should to fix this, and fast. And while you are at it, make the namespaces versioned (see below).
• Improve SAPlink & ZAKE, and include them in the standard distribution - SAPLink and ZAKE are the defacto standard ways to develop and distribute open source ABAP code. They are massive improvements over the status quo, but they aren't very widely installed, and they are not perfect. If SAP wants a healthy ABAP developer community, I'd recommend that SAP join these projects, contribute to them, and make them awesome. Then include (up-to-date) versions of them in the standard Netweaver ABAP distribution.
• Speaking of up-to-date, implement ABAP package and dependency management - Maven is kind of a nightmare, but it works. Let's say I want an mock/fake library like MockA (highly recommended) to use in my ABAP Unit tests. I go out and find it on Github, install SAPLink (which doesn't work properly so I have to go back and activate everything), download the .slnk file, install that using SAPLink (which doesn't properly install some interfaces, so I have to reinstall it), and finally I can use it in my project. But I find a bug! So I report that bug to the maintainer, who is awesome and fixes it in less than a day. My reward? I get to do the whole download, install, install all over again. We should just be able to list library versions as dependencies of our packages, classes, and test classes, and the system should handle downloading and installing them. But this will result in organizational chaos, you say. No, it won't. Because everyone will be using namespaces, and those namespaces will be versioned, so we can have more than one version of a library installed and usable at a time.

Right now it's a bit frustrating to switch between ABAP and non-ABAP development. The tooling in the Javascript and Java ecosystems is so far superior and so much more concerned with developer productivity that it begins to be painful to return to ABAP. I can't imagine what it would be like to come to the ABAP ecosystem with no background.

This is a shame, because it's a nice language in a lot of ways, and the business applications that are accessible through ABAP are an absolute goldmine of information and business process execution logic that loads of developers would love to add value to, in the process creating a great deal of value for SAP. If they could only be productive on the platform. I'd love to see SAP take ownership of the developer experience and make the ABAP ecosystem flourish.

​What could this guy possibly be thinking? What are the financial reporters that give him a platform thinking? I'll admit, I couldn't help it. I don't know much about this stuff, but I dug in a little bit to try to figure it out.

Of course, the fact that David Trainer's PDF calculation​ pegs Apple's Net Operating Profit After Tax (NOPAT) for 2012 at under $11 billion is a little fishy since it was actually around $41 billion. Turns out, the analysis is based on a "what if" scenario assuming that Apple had a Return On Invested Capital (ROIC) for 2012 of 70% and for 2013 (to date) of 52%. These drive a calculated Economic Book Value per Share around $240, which is apparently Trainer's target.

​Of course, Apple didn't have these ROICs, by Trainer's own calculations, so it's a bit foolish to run a "what if" scenario on them. The actual ROIC for 2012, shown here, was 271%. 270% vs. 70%, but who's counting? God knows how Trainer is calculating the "Total Capital" denominator in the ROIC calculation. It looks pretty close to Working Capital. But for 2013 I'd expect the ROIC to drop somewhat versus the 2012 level even if Net Income remains constant. This is because dividends have gone up and Total Capital should as well, according to Horace Dediu's analysis on Asymco. But for ROIC to drop to 52% by Trainer's calculations would take a truly massive drop in income or a much larger spike in capital expenditure than is indicated by Apple's 10K.

But really, I don't know anything about this stuff. Don't take my word for it. Take Trainer's. Given Apple's actual ROICs, what does ​David Trainer's analysis indicate the Economic Book Value per Share is today? Well, he conveniently cuts off this PDF one line before he reveals his value, but if we do a little math based on the Price to Economic Book Value per Share ratio we get:

$443 / 0.56 = $791

(Don't worry, I've saved the PDFs in case Trainer decides he needs to cut off a little bit more of the unadjusted calculation.)​

​Now, of course, there is a decent chance that Apple's ROIC will decline to something a little closer to its rivals over the next several years, but it's exceedingly unlikely it will fall off a cliff in 2013 and hit 52%. And it's a bit more than disingenuous to retroactively assume that a company reported different financial performance than it actually did.

​

But I'm bothered by "gamification" - the trend of using game design methods to encourage specific behavior in non-game situations, like the work environment. Usually this involves the use of points, badges, levels, and other feedback mechanisms. Often there is an element of competition involved.

Since I enjoy both games and work, one might think I'd be a fan of bringing the two closer together. Actually, I find the idea fundamentally problematic and prone to misuse. If I worked like I play games, or cared about work like I care about games, I would have been fired many times over by now.

What are we missing when we advocate gamification systems focused on points and badges? It might be a misunderstanding of games, or perhaps a narrow focus on a particular type of game or on a specific type of mechanic employed in games. Sure, the collection of points and levels can be enjoyable in games, but points aren't the intrinsic reason people play games. Points, in games and in gamification, are extrinsic motivators that can provide a temporary motivation boost or a  framework for a more intrinsic motivation, but which don't in themselves drive long-term engagement.

For every good game, I can show you something besides points that is the real reason I'm interested in that game. These reasons will be things like beauty, risk, competition, empathy, cooperation, excitement, mastery, surprise, uncertainty, approbation, story, joy, exercise, fear, relaxation, zoning out, maybe even friendship.

Sometimes points will enable one of these intrinsic motivators, but intrinsic motivators more often manifest in other ways and points are either absent or irrelevant. Points can even be counterproductive when deep intrinsic motivators like beauty, joy, mastery, empathy, cooperation, and friendship are in play.

Let's take an example. A game I've never played (it's PS3-only), but which crystalized some of these thoughts for me: Journey.

​I'd like to draw your attention to just before 10:00 in this video, as the avatar is skiing down a slope and the short monologue by the caster after this. Then keep watching for another minute to see him meet up with another player. To paraphrase, this is a game that gets you to want to explore using story, beauty, and curiosity. There are no points. There is only "I wonder if ...", "What happens next?", and "Wow". The only thing approximating leveling is the fact that the character gains an enhanced ability to jump and "sing" as the game goes on, but in the context of this game it appears to be more of a plot device than a leveling system. And yet, the player wants to go on, wants to explore and find out more about the story, and (most surprisingly perhaps) loves to interact with others in the world and considers them to be friends.

Journey uses beauty, story, curiosity, serendipity, cooperation, and friendship to make the game interesting, re-playable, and emotional. And it obviously succeeds. For me it's surprisingly riveting to simply watch someone else play the game. (If you want to know more, the first video in the series is here, and here is a review that mostly relates the experience of watching others play the game including a touching anecdote.) You don't get this kind of experience and meaning using only points.

The best games, like Journey, are games that aspire to be more than games. These games embody stories, art, community, cooperation, competition, opportunities for mastery, or all of these things. Yet, even these exceptional games don't capture our imaginations for all that long.

So what do we take from this? Mostly a lot of questions. Points systems are certainly useful in some situations, especially when used to structure competition. But I wonder if the overwhelming place of points in the gamification discussion belies something fundamental missing from the work environments we are trying to fix using gamification techniques. When we use points, levels, and badges to motivate, does that ​indicate a lack of intrinsic motivation in the environment? Are gamification systems acting as amplifiers of the meaning of our work, or are they attempting to impose an artificial rewards system that is at odds with the meaning of our work? If work is meaningful, wouldn't it be best to directly communicate that meaning to workers rather than filtering it through a metaphor of points and levels? If work isn't meaningful, or if the meaning of our work is negative or morally questionable, then why do we glorify the tools used to trick people into working on these things?

Shouldn't gamification be about the question of how, in our work environments, we communicate the meaning of our work? How do we provide opportunities for mastery, joy, beauty, and positive social interaction in the workplace and in our lives? Those, and other positive intrinsic motivators, are the goals. Points are, at best, a means to these ends.

​

In future, hopefully shorter, episodes we are planning to get a BEx query set up with some calculated measures, pull in some additional employee data, and report on the whole thing using BI4 tools. Maybe we'll even take some initial steps away from a data-mart approach and towards a data warehouse approach.

Permalink

]]>Ethan JewettTue, 06 Nov 2012 22:55:00 +0000http://www.esjewett.com/blog/the-bwbobj-rift-followup500c2012e4b0d820d6a397d2:500c2137e4b0d19362bd3c28:50983e97e4b0c4855bdfecdcEric Vallo has written a quite wonderful and balanced piece starting to dig into the rift between the BW and BusinessObjects BI communities over on the EVTechnologies blog. As Eric mentions, his blog was the result of quite a bit of back-and-forth between the two of us, the other principals involved in the DS Layer site, and the wider SAP and BusinessObjects communities. Eric was kind enough to kick off the conversation, and I'll continue it here. Hopefully this will evolve into a longer public conversation.

So, let's get started!

Eric did a good job of laying out the different scenarios we tend to see out in the SAP & BusinessObjects customer ecosystem right now. While each of these scenarios is clear, I think there is still a huge amount of confusion within both the BW and BusinessObjects communities about what each set of tools does and what it should be used for. In other words, when do you go for each scenario or tool? In my opinion, this confusion stems from the reality that both toolsets do a lot of things, and there is a lot of overlap. You can build a full datawarehouse and BI reporting solution using only BusinessObjects tools. And you can do the same using only BW tools. Or you can mix-and-match and pull in 3rd-party tools as well. But each toolset certainly has its strengths and weaknesses.

My take on BW's strengths and weaknesses is, perhaps, a bit out of the mainstream. But I think it is justified and, importantly, I think it is in line with SAP's vision for it's data platform. Specifically, I see BW as a set of data warehousing, data management, and systems management services. I admit readily that I've shamelessly stolen this way of describing the concept from others, but it fits so well that I have to use it. So, BW provides a set of services like the following very incomplete list:

• An abstract, platform-independent data warehouse modeling environment
• A service for imposing a semantic model on top of the data warehouse modeling environment
• An analytic engine for executing queries on the data residing under the semantic model
• Security
• Data management services like near-line storage (the ability to pretty transparently persist data in a single semantic structure across two data-stores with remarkably different latency, structure, performance, and cost characteristics while still being able to query that data "live") or archiving.
• Visualization and reporting services

The implementation of this service concept is far from perfect. In fact, today, most of these "services" aren't really recognizable as such because BW is such an integrated intertwined monster. With BW on HANA, we are now starting to see some of these services exposed individually, and the result is rather impressive.

One advantage of seeing BW as a collection of services for the purposes of this discussion is that it becomes much less important whether BW continues to exist. When viewed this way, the whole discussion about whether or not SAP HANA is going to "kill" BW becomes moot. The important question is whether the services continue to exist, either as part of BW, part of the HANA platform, or in another way. It is the services that provide the value, not BW per se.

The same actually holds true for BusinessObjects. Here we have services like the following (again, incomplete):

• A service for imposing a semantic model on existing data models
• An analytical engine for executing queries on the data residing under the semantic model
• Security
• Services for moving data from one place to another and transforming that data
• Services for tracking and analyzing data lineage and quality across multiple platforms
• Visualization and reporting services

BusinessObjects took a much more service-oriented development approach from the beginning, so many of these services are much more obvious as individual BusinessObjects products or product components. You might recognize direct descriptions of Universes, Data Services, or Information Steward, or the host of BusinessObjects reporting and visualization tools whereas in the case of the BW services there is no individual tool we can install to provide the service.

You probably also notice some overlap with the list of services in BW. Hence, the confusion.

The really interesting part of this overlap is that some of these services are provided in pretty similar ways between the two platforms (the semantic layer) while others take a very different approach (visualization and reporting services) or simply don't exist on one platform or another (abstract data warehouse modeling environment, or tracking and analyzing data lineage). It's these different approaches or unique services that differentiate the platforms.

Until SAP matures its data platform strategy and makes some decisions about which overlapping services live, die, or merge on each platform, it's going to be up to customers to do the hard thinking necessary to make the best implementation decisions for them. It is, of course, important to know what services each platform offers, what technique the platform uses to offer those services, and what kind of performance and functional characteristics similar services have on the different platforms. I hope that the podcast series Josh Fletcher and I are working on will help with that discussion across customers. But in addition to these feature considerations, there are also cost, requirements, tooling, and expertise considerations that are going to be unique to each customer.

Personally, I think BW and BusinessObjects tools both have a lot to offer and we shouldn't discount one or the other when we're talking about addressing data warehousing or BI problems. As consultants, we'll have our personal preferences, but we should try not to let that influence our customer advisory roles when addressing early design and tool selection questions.

In the data warehousing space, my strong feeling is that BW's approach (if not BW itself) is a powerful approach to making datawarehousing more accessible. It doesn't solve the basic problems of data modeling and data governance, and it can introduce its own set of problems. But BW does help abstract some really knotty technical and organizational problems that like data consistency, technical governance, platform-specific optimization, and technical change management that must be addressed when building a data warehouse.

Meanwhile, the BusinessObjects approach of a really open, modern, and platform agnostic BI and EIM toolset is a great one. Especially in the ETL and BI space, many of the services that BusinessObjects offers are years, even decades ahead of where BW is with its more integrated and older transformation and reporting engines.

I'm hoping that the two communities can learn a lot from each other by interacting more both technically and socially. In other words, like Eric, I am really looking forward to feedback and teaching through other blogs or Twitter.

And now, back to you, Eric.

In this episode we discuss what BW is and what it can offer, and in future episodes we're going to piece together a Data Services extraction, BW-based data mart, and BusinessObjects-based reporting live. Tune in to watch us crash and burn. ;-)

Permalink

]]>Ethan JewettWed, 10 Oct 2012 21:12:25 +0000http://scn.sap.com/community/developer-center/hana/blog/2012/10/10/sap-developer-engagement--a-pre-teched-report-card-and-video-blog500c2012e4b0d820d6a397d2:500c2137e4b0d19362bd3c28:5075e4b5e4b0028f3161f475Jon Reed was kind enough to organize a discussion about SAP's developer engagement initiatives between Graham Robinson, Jon, and me. He's written it up on SCN and there is a video of the discussion available. I highly recommend reading the blog or watching the video if you are interested in the topic of development on top of SAP software, whether you are a development consultant, are making 3rd party add-ons or mobile apps, or are thinking about writing software that uses SAP's HANA database. If you want to chat about the topic, you can track one of us down at TechEd or on Twitter.

Permalink

]]>Ethan JewettMon, 01 Oct 2012 18:29:58 +0000http://www.esjewett.com/blog/it-matters-what-you-measure500c2012e4b0d820d6a397d2:500c2137e4b0d19362bd3c28:5069e126e4b0a374292f1716We see this a lot in the data analysis space, but it's worth remembering that it really matters what you measure.

Rarely have I seen such a clear example of that as the claim today from the app developer Snappli that the percentage of people using Apple Maps on iOS6 has dropped from 35% to 4%. The Guardian nails the story, noting that the claim may well be without merit.

What happened?

Well, it turns out that Snappli was measuring not the percentage of people using the maps application each day, but the percentage of people using maps data every day. In iOS5, these percentages were effectively the same because the Maps application almost always had to download new map tiles every time it was opened.

So Snappli happily thought they had a good proxy for maps usage and over time they may have forgotten that their measurement was a proxy. They may have begun to assume that they were measuring actual maps usage. This happens all the time and it's really easy to do. But it's pretty embarrassing when something changes, causing our assumption to become invalid, and we don't realize it. That's when we start making false claims and looking pretty foolish.

That is what happened to Snappli. Because (surprise!) the Maps app in iOS6 doesn't download data very often. I just now threw my phone into airplane mode and zoomed into the last three cities I'd looked at over the last week and a half: Madison, WI; Minneapolis, MN; and Chicago, IL. In each of these cities I could see streets down to the lowest level of detail, including shops and points of interest. No data use.

So it seems that what Snappli scored as a strike against the new Maps app should actually be counted as a point in it's favor. And this whole saga can serve as a good reminder to us in the data business that we need to keep the assumptions behind our metrics and measurements in plain view as much as possible.

P.S. I realize there are some major problems with the data behind the maps app on iOS6. However, I also think it's important to focus on actual problems and not made up problems.

It sure looked to me like SAP's Björn Goerke was strongly endorsing all new browser-based UI development happening in HTML, Javascript, and CSS (colloquially known as HTML5) rather than alternatives like Silverlight and Flash.

This is definitely the right move. I've long held the opinion that any UIs that can effectively be developed using HTML5 should be developed using HTML5. So it pained me, 3+ years ago, to see SAP developing new web-based UIs for their Business ByDesign product in Silverlight.

Of course, there have been reasons to doubt that HTML5 would win out as the web UI technology of choice. But it's now a no-brainer to turn to Flash and Silverlight only as a last resort for browser-based UIs. It has been ever since it became clear that the iPhone was going to have a sizable chunk of the phone market and that the iPad was going to dominate the tablet market for a good length of time: almost 2 years now. So it perplexed me a year ago to see the new UI for SAP's universal storefront released using Silverlight.

(The storefront is currently being rewritten in HTML5, and already displays as HTML5 on devices that don't support Silverlight, as well as in some browsers.)

With Goerke's statements, I had some hope that SAP was finally on target with its browser-based UI strategies. It would only build new browser-based UIs in HTML5, and in doing so it would get good at HTML5. Sometimes, to make a change, you've got to force yourself to make the change. It looked like SAP was going to bite the bullet and break free of Silverlight and Flash - two technologies that were holding it back on mobile, at the least. That made me happy.

Then I saw that SAP is planning to release a new tool in Q4 with (you guessed it) a browser-based Silverlight UI.

Back to waiting.

iOS seems to do an excellent job demonstrating that these vestigial technologies are no longer necessary in modern browsers. At this point, for most people, all Java provides is a less secure, less performant browsing experience.

Permalink

]]>Ethan JewettThu, 09 Aug 2012 15:55:34 +0000http://www.esjewett.com/blog/the-costs-of-getting-an-ac-in-line500c2012e4b0d820d6a397d2:500c2137e4b0d19362bd3c28:5023dd6ac4aac47b982fed1fDennis Howlett theorizes about possible controls issues and what is going on with SuccessFactors revenue vis a vis SAP's reported Q2 results. When I looked at SAP's reported 2nd quarter results, it seemed that something was fishy. SAP reported strong growth for SuccessFactors on non-standard measures like "12-month subscription billings from new business" but when it came to revenue SAP only compared total SAP Q2/2012 revenue (69€ million) with SAP Q2/2011 revenue (4€ million pre-SuccessFactors). Year-over-year comparisons of SuccessFactors revenue were avoided.

Given growth of SAP's homegrown Business ByDesign solution, there was some reason to believe that SuccessFactors reported revenue may even have been down year-over-year. Howlett is much more qualified than I when it comes to recognizing and investigating these types of issues, and he shines a spotlight on some behind the scenes reasons that these numbers may have fallen out the way they did.

This time via a tweet from Andrew Fox: The worst implementation of a pie chart I have ever seen, courtesy of SAP BusinessObjects Visual Intelligence.

Is there something between Jackets and Sweat-T-Shirts? And what is between Accessories and Dresses? What happened to Overcoats, City Trousers, and Outerwear? Fell through the cracks, I guess.

SAP BusinessObjects Visual Intelligence is supposed to be SAP's answer to industry leading data visualization products like Tableau and Qlikview. Seeing this, it looks more like an advertisement for these competing (I use the term loosely) products.

The full demo video is here.

​The announcement of more developer access to SAP's mobile platform allows developers to use SAP-provided Amazon EC2 instances with SUP installed on them. This mirrors similar announcements for SAP's HANA platform earlier this year. SAP also announced a new partner framework which makes the partnership process much easier to understand. Both announcements are major improvements for SAP's mobile efforts.

​That said, SAP still lags its competition significantly from a developer engagement perspective and these announcements underscore the fact that SAP has been unable to make the quantum leap necessary to become a player in the minds of non-SAP developers. There are two reasons for this: the immediate problem that keeps changing as SAP addresses symptoms, and the deeper reason that is keeping SAP from achieving the change required.

​

The immediate problem SAP seems unable to overcome is that the basis of any successful developer push is software that can be used for development. The AWS-only form-factor that SAP has chosen for SUP and HANA is sub-standard for development purposes. In some ways it may reflect a view of software development from the days before standard use of source control and local build/test environments. The AWS-only approach precludes setting up a local development environment, which is standard practice for most developers today (and for good reason). It also incurs a relatively small but unavoidable cost on developers, forcing them to pay for EC2 instances they may not otherwise need.

​Oracle, IBM, and Microsoft all provide easily accessible, downloadable versions of their database software for developers to use. SAP does not. Appcelerator, PhoneGap, and Sencha all provide easily accessible, downloadable versions of their mobile development platforms. SAP does not.

SAP claims that it is trying to appeal to the larger non-SAP developer community.​ But developers in those communities will look at what is on offer and see a company they consider a dinosaur asking them to give up their modern development practices and pay for the privilege. I think most developers will just say, "No thanks".

​

​The deeper problem is that SAP still hasn't come to terms with the fact that developer engagement is not something you can tack on to a product at the end of the product cycle. The product must be designed for developers. It must be launched along with developer programs including downloads available to developers. And the product team needs to engage with developers on an ongoing basis after launch. I'm no developer engagement professional, but from what I've seen of the industry, these are the absolute minimum requirements.

And yet, from a cultural perspective it is clear that SAP has not embraced developer engagement. Sybase launched SUP 2.0 over a year ago, and only now has it launched an arguably usable (but certainly not ideal) developer environment. HANA is one of the products where SAP is attempting most aggressively to appeal to developers, and the AWS-environment for HANA became available only a couple of months before the 1-year anniversary of HANA's public launch.

SAP needs to achieve a quantum leap in their developer engagement ​culture. Otherwise it's unlikely that any of its foundational technology products become breakout successes. Thanks to the hard work of many within SAP, it's possible that SAP could succeed in spite of itself. But it's certainly making it harder than it needs to be.