UML has been described by some as “the lingua franca of software engineering”. Evidence from industry does not necessarily support such endorsements. How exactly is UML being used in industry — if it is? This paper presents a corpus of interviews with 50 professional software engineers in 50 companies and identifies 5 patterns of UML use.

The abstract for this distinguished paper doesn’t do it justice. Over two years, the author interviewed over 50 developers from a broad cross-section of industries and countries. She found their use fell into five broad categories:

Among the reasons given for not using it were:

• Lack of context: UML deals with architecture, rather than with the whole system.
• The overheads of understanding the notation.
• Issues of synchronization and consistency.

Perhaps the most interesting category is the second: those people who selectively use some elements of UML, but not the whole notation. Some of the partial uses identified were:

• UML as a ‘thought tool’
• communicating with stakeholders
• collaborative dialogs
• adaptation (i.e., using a homegrown variant of the “real” notation), and
• selective traction (i.e., using it just as long as is useful, then moving on)

while the parts used were:

But there is much more in this paper than merely statistics. One of Petre’s many insightful comments is:

Responses concerning UML use tend to be polarized, between design use and implementation use… Despite the notional accommodation of the whole process, informants tend to use UML either in early design, or in implementation, rarely both (even when informants’ roles include the whole process).

There are two ways to react to this work. The first is to read it as an indictment: after 20 years, UML is still mostly not used and not valued. The second, and more hopeful, is as a concrete step toward improving it. Parts of UML are used; the more we learn about which ones, where, why, and how, the better our chances of building something better.

Patch generation is an essential software maintenance task because most software systems inevitably have bugs that need to be fixed. Unfortunately, human resources are often insufficient to fix all reported and known bugs. To address this issue, several automated patch generation techniques have been proposed. In particular, a genetic-programming-based patch generation technique, GenProg, proposed by Weimer et al., has shown promising results. However, these techniques can generate nonsensical patches due to the randomness of their mutation operations.
To address this limitation, we propose a novel patch generation approach, Pattern-based Automatic program Repair (PAR), using fix patterns learned from existing human-written patches. We manually inspected more than 60,000 human-written patches and found there are several common fix patterns. Our approach leverages these fix patterns to generate program patches automatically. We experimentally evaluated PAR on 119 real bugs. In addition, a user study involving 89 students and 164 developers confirmed that patches generated by our approach are more acceptable than those generated by GenProg. PAR successfully generated patches for 27 out of 119 bugs, while GenProg was successful for only 16 bugs.

Software products are released with known and unknown bugs. For example, this paper by Kim et al. mentions that Windows 2000 was released with over 60K known bugs. Why? Bug repairs are for the most part done manually and there are not enough resources to repair all of them. Automatic patch generation methods have been devise to reduce this manual effort.

To add to this field of research, Kim et al. integrate the human element with machine learning. They do this by manually reviewing human written patches to create fix-pattern templates used to then generate “candidate” patches. The purpose in doing this was to solve the issue of what they consider to be “nonsensical patches” produced by GenProg (https://www.cs.virginia.edu/~weimer/p/weimer-icse2012-genprog-preprint.pdf). GenProg’s approach to automated patch generation is based on genetic programming used to search the vast space of possible program repairs.

Two interesting aspects of the Kim et al. paper are that the patches generated from one project are used successfully on other projects and the fix-pattern templates once created can be use repeatedly. However, one element not measured in this paper is the monetary cost of fixing each bug (measured in GenProg), considering the bounties offered for bug fixes by some companies, it would be interesting to see such a result included in future automatic patch generation methods.

While many bug prediction algorithms have been developed by academia, they’re often only tested and verified in the lab using automated means. We do not have a strong idea about whether such algorithms are useful to guide human developers. We deployed a bug prediction algorithm across Google, and found no identifiable change in developer behavior. Using our experience, we provide several characteristics that bug prediction algorithms need to meet in order to be accepted by human developers and truly change how developers evaluate their code.

This paper highlights the divide between the success of bug prediction algorithms in academia and the lack of their adoption in software engineering practice. Lewis et al. presented volunteer software developers at Google with the results of two state-of-the-art algorithms. The first, the award winning FIxCache which caches files that are predicted to be bug-prone (Lewis et al. used two versions of this algorithm, one biased to cache newer files and the other biased to cache older files) and the second is what they call the Rahman algorithm which uses “the number of closed bugs to rank files from most bug-prone to least bug-prone”.

The highlight of this paper is the list of three must-have characteristics for a bug prediction algorithm to be adopted as part of the software development process:

1. Actionable messages: The output of a bug prediction algorithm should be actionable.
2. Obvious reasoning: When a bug prediction algorithm flags a file as bug prone, developers would like to know why to allay any fears that the flag is a false positive.
3. Bias towards the new: Developers are more concerned with files that are currently causing problems.

These characteristics were born from conversations with Google developers which led Lewis et al. to create TWS (an optimized version of the Rahman algorithm). TWS addressed two of the three must-have characteristics (2 and 3 above). The results showed that developer behavior did not change significantly before and after TWS was deployed. Lewis et al. looked at this result as a failure of TWS which did not present developers with actionable messages.

This work opens a line of discussion as to the real success of bug prediction algorithms, who are their most likely users, and what they should offer beyond precision, recall and F-measures.

The MSR program is at http://2013.msrconf.org/ along with links, where possible, to the papers being discussed. Sadly, the IEEE is not in the information-sharing business, but many of the papers can be found linked from here. Prof. Gail Murphy opened the talk with a pretty straightforward question: what is developer productivity? Her slides are here, but they don’t really convey how good a talk it was.

The one thing that we as an industry were repeatedly asked for, and this came up a lot—perhaps unsurprisingly, given that this is a group of people whose job is mining software repositories—was this:

You should have instrumentation and logging around your processes.

This isn’t pure altruism on their part, of course, but it’s not pure self-interest either. Over and over again, the papers being presented drove home the points that there are a surprising number of tool-assisted efficiencies hidden in your software’s history, provided that history is around to be analyzed in the first place. Instrumentation grows up to become automation informed by logs that turn into history, and good tooling around a project’s history can play an active role in building its future.

Dr. Murphy’s talk got a lot more specific and a lot heavier, than that—read the slides, at the very least. But one point that I thought she went through a too fast was: what if it was simple? What if it was easy to drill into comprehensive lifecycle data, see trends, and reflect on my own place and role in those trends? I’ve got some theories about that, but I’ll get back to them shortly.

So, right from the opening, there were results being presented that jumped out at me as being relevant to our work at Mozilla, particularly given our very much inbox-driven development and communication approach. For example, the paper by Mukherjee and Garg about triaging email notifications proposes:

…a machine learning based approach to predict whether a notification will prompt any action from its recipient. Such a prediction can help to suitably mark up notifications and to decide whether a notification needs to be sent out immediately or be bundled in a message digest.

This certainly seems like something we could use, as do the other two papers in that timeslot, describing different approaches to automatically triaging bugs. Later on in the day it was shown that the #1 factor affecting how long it takes a bug to be fixed is fast, accurate triage, so a tool-guided bug triage process seems like a pretty big win, particularly one that learns over the long term.

Over the course of the day, as we moved into the MSR-Mobile and Mining Challenge parts of the conference, even things that weren’t directly on-point for Mozilla were still pretty compelling. Textual analysis of Stack Overflow and other help sites that highlights pain points for our community across our products and services is a great idea. Automated approaches to figuring out which parts of an API are brittle or hard to use, which parts of our documentation need help, what new features people are asking for…are we doing that? Not in depth, I don’t think, and not to drive decision-making. The Mozilla Developer Network (MDN) is making some moves in that direction, but I think there’s some opportunities there we’re not taking advantage of.

Another thing that came to light during the Stack Overflow data analysis talks was that the activity from people who are awarded “marathon” badges—500 edits, 500 comments, that sort of thing—drops way off after they’ve received the badge. I’d like to dig into that a bit more, to try and understand the causal relationship there: is it that people just get the achievement and then stop, is it that people suddenly realize how much time they’ve spent on Stack Overflow and decide to maybe go outside instead? Still, that discovery should inform some of our badging efforts as we start to ramp up the OpenBadges efforts on Mozillians.

The Analysis Of Bug Reports section of the talk…just read the whole thing. I’d be doing it a disservice to summarize, but certainly in light of the well-documented problems porting defect-prediction methods across software projects (a noble endeavor with a 96% failure rate, if you needed a windmill to tilt at this week) it was pretty great that so many of the papers presented here were built on Mozilla’s historical data. Go team! (Incidentally, this is the part of the day that “accurate triage is the most important thing” comes from, courtesy of Mani et al.)

There’s a lot more to say about this, but shortly after this the content of the talks went a little bit past my pay grade. I spent more than a few of them feeling like that one guy in GoodBurger: “Hmm, yeah… I know some of these words”.

What I want to talk about most is building out tools with an eye towards future machine learning and tool-driven analysis. I’ve been a sysadmin by trade for a long time, and one of the most important things I’ve learned there is that if you’re in a complex environment with a lot of moving parts, you absolutely must have comprehensive, centralized logging. It’s at the core of virtually everything you need to accomplish, whether it’s today’s sev-1 incident response or next year’s capacity planning, and without it you’re flying deaf and blind. But if you’ve got it, you’ve got notifications, advance warnings, statistical analysis, fault prediction—dozens and maybe hundreds of options and opportunities open up on timescales from preemptive to strategic.

I think we should try to apply the same approach to development tools and developer machines. Disk space isn’t super-expensive—not free, I know, but I’m talking about text here—so here’s my one-point plan: if you’re building a tool that reports back to stdout/stderr, whatever it is, build in an option to send that information (or a structured version of it) to syslog at the same time. If you’re part of a team using those tools, centralize those logs. It seems like an odd thing to do—certainly I don’t know of anyone else doing that—but while I don’t know for sure, I’m willing to bet after a while, the information we’ll be able to learn from those logs will surprise us. And even if we don’t know what to do with it, the people at MSR who are keen to collaborate with us will.

SCOPE is adopted by thousands of developers from tens of different product teams in Microsoft Bing for daily web-scale data processing, including index building, search ranking, and advertisement display. A SCOPE job is composed of declarative SQL-like queries and imperative C# user-defined functions (UDFs), which are executed in pipeline by thousands of machines. There are tens of thousands of SCOPE jobs executed on Microsoft clusters per day, while some of them fail after a long execution time and thus waste tremendous resources. Reducing SCOPE failures would save significant resources.

This paper presents a comprehensive characteristic study on 200 SCOPE failures/fixes and 50 SCOPE failures with debugging statistics from Microsoft Bing, investigating not only major failure types, failure sources, and fixes, but also current debugging practice. Our major findings include (1) most of the failures (84.5%) are caused by defects in data processing rather than defects in code logic; (2) table-level failures (22.5%) are mainly caused by programmers’ mistakes and frequent data-schema changes while row-level failures (62%) are mainly caused by exceptional data; (3) 93% of fixes do not change data processing logic; (4) there are 8% failures with root cause not at the failure-exposing stage, making current debugging practice insufficient in this case. Our study results provide valuable guidelines for future development of data-parallel programs. We believe that these guidelines are not limited to SCOPE, but can also be generalized to other similar data-parallel platforms.

This insightful little paper is a great example of how a small investment in studying how things actually work can beneficially impact where developers focus their effort. A similar fault analysis of (for example) the way novices do joins in SQL, or of how they get list comprehensions in Python wrong, would be very welcome. We look forward to seeing how the team at Microsoft incorporates these findings into the next version of SCOPE.

Peer Instruction (PI) is a teaching method that supports student- centric classrooms, where students construct their own understanding through a structured approach featuring questions with peer discussions. PI has been shown to increase learning in STEM disciplines such as physics and biology. In this report we look at another indicator of student success – the rate at which students pass the course or, conversely, the rate at which they fail. Evaluating 10 years of instruction of 4 different courses spanning 16 PI course instances, we find that adoption of the PI methodology in the classroom reduces fail rates by a per-course average of 61% (20% reduced to 7%) compared to Standard Instruction (SI). Moreover, we also find statistically significant improvements within-instructor. For the same instructor teaching the same course, we find PI decreases the fail rate, on average, by 67% (from 23% to 8%) compared to SI. As an in-situ study, we discuss the various threats to the validity of this work and consider implications of wide-spread adoption of PI in computing programs.

This paper, which has just been presented at SIGCSE 2013 in Denver, may be one of the most significant empirical results we’ve ever reported. As the abstract says, a specific teaching technique can cut the failure rate in introductory classes by more than half; it also increases self-reported learner satisfaction. To find out more, check out http://www.peerinstruction4cs.org/.

The impact and applicability of software metrics continues to be a subject of debate, especially since there are many metrics that measure similar properties, like cohesion. This raises the question of the extent to which these metrics agree or not.

The interesting idea that this paper proposes is to not only analyze the agreement and disagreement of metrics, but to also investigate how the metrics change on refactored versions of the same code. The authors do so by randomly applying automated refactorings to a code base and observing how these refactorings impact the metrics. By running these automated refactoring analysis, the authors want to distinguish between what they call volatile metrics, those that are easily impacted, and inert metrics that hardly change under refactoring. Furthermore, they want to know what metrics change in relation with one another, are the refactorings that cause one metric to increase, while another (supposedly measuring a similar property) decreases.

They applied their method to 300KLOC of Java code of 8 open source systems and investigated the following five metrics:

• Tight Class Cohesion(TCC)
• Lack of Cohesion between Methods (LCOM5)
• Class Cohesion (CC)
• Sensitive Class Cohesion (SCOM)
• Low-level Similarity Base Class Cohesion. (LSCC)

Their evaluation shows that LSCC, CC and LCOM5 are all highly volatile metrics: in 99% of the refactorings, these were either increased or decreased. The results, however, were different for the 8 systems under consideration. In one case, for example, all metrics turned out to be volatile. Even when normalizing for relative volatility, the variance remained high.

In a second evaluation, the relationship between two of the cohesion metrics, LSCC and TCC, is explored in more detail. Refatorings where one of those two metrics is lowered, while the other is increased are studied in more detail.

What makes this work so interesting, apart from the cool originality of applying automated refactoring in the context of metrics, is the fact that it changes our perception of metrics. Where we previously assumed that different metrics for cohesion were mainly a matter of taste (and hence debate), this papers finds that metrics can not only differ, but that they can be conflicting in many cases.

Co-located with the 35th ACM/IEEE International Conference on Software Engineering (ICSE 2013)
Sponsored by IEEE TCSE and ACM SIGSOFT

NEW IN 2013!

• Data papers for describing data sets curated by their authors and making them available to the research community
• Practice papers for experiences of applying mining repository algorithms in an industry/open source organization context
• Microsoft Surface tablet with Windows RT as prize for the best Mining Challenge, sponsored by Microsoft Research.

IMPORTANT DATES

• Research/Practice abstracts: Feb 8, 2013
• Research/Practice papers: Feb 15, 2013
• Data papers: Feb 15, 2013
• Challenge papers: Mar 04, 2013
• Author notification: Mar 15, 2013
• Camera-ready copy: Mar 29, 2013
• Conference: May 18-19, 2013

All submission deadlines are 11:59 PM (Pago Pago, American Samoa) on the dates indicated.

CALL FOR PAPERS

Software repositories such as source control systems, archived communications between project personnel, and defect tracking systems are used to help manage the progress of software projects. Software practitioners and researchers are recognizing the benefits of mining this information to support the maintenance of software systems, improve software design/reuse, and empirically validate novel ideas and techniques. Research is now proceeding to uncover the ways in which mining these repositories can help to understand software development and software evolution, to support predictions about software development, and to exploit this knowledge concretely in planning future development. The goal of this two-day working conference is to advance the science and practice of software engineering via the analysis of data stored in software repositories.

This year, we will solicit three tracks of papers: Research, Practice, and Data. As in previous MSR editions, there will be a Mining Challenge on Stack Overflow data and a special issue of best MSR papers in the Empirical Software Engineering journal.

Research papers: Research papers can be short papers (4 pages) and full papers (10 pages). Short research papers should discuss controversial issues in the field, or describe interesting or thought provoking ideas that are not yet fully developed. Accepted short papers will present their ideas in a short lightning talk. Full research papers are expected to describe new research results, and have a higher degree of technical rigor than short papers.

Practice papers: (New!) Practice papers should report experiences of applying mining repository algorithms in an industry/open source organization context. Practice papers aim at reporting positive or negative experiences of applying known algorithms, but adapting existing algorithms or proposing new algorithms for practical use would be plus. Practice papers also can be short papers (4 pages) and full papers (10 pages).

Data papers: (New!) We want to encourage researchers to share their data. Data papers should describe data sets curated by their authors and made available to others. They are expected to be at most 4 pages long and should address the following: description of the data, including its source; methodology used to gather it; description of the schema used to store it, and any limitations and/or challenges of this data set. The data should be made available at the time of submission of the paper for review, but will be considered confidential until publication of the paper. Further details about data papers are available on the conference website.

Mining challenge: In the Mining Challenge, we invite researchers to demonstrate the usefulness of their mining tools on preselected software repositories and summarize their findings in a challenge report (4 pages). Please visit our Challenge Web Site for details about the Mining Challenge. This year, the challenge is on the Stack Overflow data. We provide the dump for the Stack Overflow web service and you should use your brain, tools, computational power, and magic to uncover interesting findings related to it.

EMSE SPECIAL ISSUE

A selection of the best research papers will be invited for consideration in a special issue of the journal, Empirical Software Engineering (EMSE), edited by Springer.

TOPICS

Papers may address issues along the general themes, including but not limited to the following:

• Analysis of software ecosystems and mining of repositories across multiple projects
• Models for social and development processes that occur in large software projects
• Prediction of future software qualities via analysis of software repositories
• Models of software project evolution based on historical repository data
• Characterization, classification, and prediction of software defects based on analysis of software repositories
• Techniques to model reliability and defect occurrences
• Search-driven software development, including search techniques to assist developers in finding suitable components and code fragments for reuse, and software search engines
• Analysis of change patterns and trends to assist in future development
• Visualization techniques and models of mined data
• Techniques and tools for capturing new forms of data for storage in software repositories, such as effort data, fine-grained changes, and refactoring
• Characterization of bias in mining and guidelines to ensure quality results
• Privacy and ethics in mining software repositories
• Meta-models, exchange formats, and infrastructure tools to facilitate the sharing of extracted data and to encourage reuse and repeatability
• Empirical studies on extracting data from repositories of large long-lived and/or industrial projects
• Methods of integrating mined data from various historical sources
• Approaches, applications, and tools for software repository mining
• Mining software licensing and copyrights
• Mining execution traces and logs
• Analysis of natural language artifacts in software repositories

SUBMISSION

All papers must conform at time of submission to the ICSE/MSR 2013 Formatting Instructions and must not exceed the page limits (research/practice papers: 10 pages; short papers: 4 pages; data papers: 4 pages; challenge reports: 4 pages), including all text, references, appendices and figures. All submissions must be in English and in PDF format.

Papers submitted for consideration should not have been published elsewhere and should not be under review or submitted for review elsewhere for the duration of consideration. ACM plagiarism policies and procedures shall be followed for cases of double submission.

Papers must be submitted electronically through EasyChair using the following URL: http://easychair.org/conferences/?conf=msr2013

Upon notification of acceptance, all authors of accepted papers will be asked to complete an IEEE Copyright form and will receive further instructions for preparing their camera ready versions. At least one author of each paper is expected to present the results at the MSR 2013 conference. All accepted contributions will be published in the conference electronic proceedings.

Quite regularly, I get questions about what empirical evidence supports my “belief” that models are good… Until now, I used to point to the (true, few) scientific empirical studies on the effectiveness of software modeling…but now I have an even anser to give you: “Empirical Evidence of the Value of Version Control”.
No, I haven’t lost my mind. The point of this link is to show you that there’s no proof that version control is better for software development, and yet, I don’t think any of you would argue against it.
Same for modeling and model-driven engineering. It would be great to have more proof but the absence of proof alone should not be used against it unless you want to start also abandoning other unproven things like version control.

He’s right: if we’re willing to accept that version control is valuable, without proof, then we can hardly require advocates of modeling to prove their case. Or advocates of functional programming, or literate programming, or Hungarian notation. Heck, if we don’t require proof for our claims, then we’re honor-bound to accept that Perl is “intuitive” because its grammar has as many special cases and contradictions as the grammars of natural languages, aren’t we? Or that learning Befunge makes you a better programmer (seriously, I’ve heard that claim too).

At some point, the statement, “If we don’t need to prove the value of version control, we don’t need to prove the value of X” becomes absurd. However, everyone’s threshold of absurdity is different. I personally don’t think that modeling adds value for most developers in most situations—I think that if it did, or if its benefits really were as significant as its advocates claim, more developers would have adopted it by now—but I don’t know. What I do know is, if we can’t demonstrate the value of something that most of us believe in, like version control, what chance do we have of telling whether other practices, like modeling and test-driven development, are worth adopting (or rather, when they’re worth adopting and by whom, since I doubt there’s a one-size-fits-all answer)?

So here are my requests:

1. Tell us what kind of study would convince you that using Befunge didn’t make programmers more productive.
2. Then tell us what kind of study would convince you that version control didn’t either.

If your answer to the second question is is, “Nothing ever could,” then version control is an article of faith for you, and there’s no point arguing further [2]. If your answer to the second is different from your answer to the first, please tell us why.

[1] Full disclosure: Jordi and I co-authored a study of web-based software project portals. And either way, we hope you have a happy and productive 2013.

[2] This request is inspired by Karl Popper’s notion of falsifiability: a claim is only scientific if there is some way to prove it wrong.