Born to be geek!

Intensive Metrics for Software Evolution

2013-05-17T00:00:00-07:00

In natural sciences, intensive metrics do not depend on the size of the system (or amount of material) being measured. For instance, the temperature of an object or the hardness of a material are intensive properties.

In the case of software projects, many of the metrics used to study them are extensive, that is, the metrics depend on the size of the system. Thus, it is hard to compare different software projects using these metrics.

For the case of the Apache Software Foundation, we have found some ratios that are intensive properties, and we show how to use them to study the evolution of different projects, comparing them regardless their size, intended audience, technology used or maturity state. The metrics are obtained from the ratios between communication and development activies in the project. Interested? See a tag cloud of the keywords of this paper:

created at TagCrowd.com

The paper is available in PDF, and it can be cited using this BibTeX code (also available as a text file for your convenience):

@InProceedings{gala2013:apache,
  author    =    {Santiago Gala-Perez
                  and Gregorio Robles
                  and Jesus M. Gonzalez-Barahona 
                  and Israel Herraiz},
  title     =    {Intensive Metrics for the Study of the Evolution of Open Source Projects},
  booktitle =    {10th IEEE Working Conference on Mining Software Repositories},
  year      =    {2013},
  address   =    {San Francisco, California, USA},
}

We are presenting this paper at the 10th IEEE Working Conference on Mining Software Repositories, with the following slides (see the slides at SlideShare.net):

Finally, this work is derived from Santiago Gala's master thesis, available in PDF format and including full details about the study of intensive metrics for software evolution. There is also a replication package that you can use to verify and extend our empirical study.

Don't do empirical software engineering with Excel

2012-11-23T00:00:00-08:00

Or any other statistical analysis. From time to time, I see papers published in Empirical Software Engineering conferences and journals that have used Excel for the statistical analyses (although I would not point here to any of those papers :). I have not liked this much, but it was mostly because of personal taste, and probably also a sort of prejudice.

But I found this paper today:

On the accuracy of statistical procedures in Microsoft Excel 2007
B.D. McCullough , David A. Heiser
Computational Statistics and Data Analysis 52 (2008) 4570–4578

The paper remarks some of the flaws of Excel 2007 in standard statistical methods. Two of the flaws are related to the least squares fitting of exponential models, and to the normality plot of a sample against the deciles of the Normal distribution. Apparently, this situation has repeated since years ago with every new release of Excel. Some of the problems have even been claimed to be fixed, while the truth is that they remain providing wrong results.

If you use Excel in your research, I think you would better consider switching to GNU R, and its superb IDE R Studio.

The impact of bias in bug-fix datasets for defects prediction

2012-04-15T00:00:00-07:00

Last week I gave a talk at UC Davis about the research work I will be doing during these months. It contains some preliminary results about the impact of bias in bug-fix datasets.

In projects with bug tracking systems and version control repositories, when a commit corresponds to a bug fix, it is usually marked accordingly (for instance, with a message like "Fixes bug #123"). This information can be used to recover the relation between commits and bugs, which is useful for defects prediction. The preliminary results I have obtained so far, show that the impact of bias is negligible for defects prediction if the model is based on a binary classifier (that is, only predicts whether an entity will contain or not defects, not how many defects it will contain). However, it is true that a non-biased dataset can provide a better accuracy, but just because, by definition, non-biased datasets contain more data. If we reduce the size of a non-biased dataset, by extracting a random sub-sample, it is as good as a biased dataset of the same size. Well, at least for the two cases I have studied so far.

More details in the slides. You can also see the slides at Slideshare.net, and get a PDF copy.

Visiting UC Davis

2012-04-02T00:00:00-07:00

Since a couple of days ago, I am in Davis, California, for a 4-months visit to UC Davis, hosted by Prof. Prem Devanbu. This visit is possible thanks to a "José Castillejo" grant awarded by the Spanish Ministry of Education and Science.

The main goal of this visit is to work on finding an automated method to evaluate the bias in bugs datasets. This bias is introduced when the bug-fix reports are linked with commits in the version control system. When a developer accepts and/or fix a bug report, she decides and accordingly marks the report with a severity level. In Bugzilla, one of the most used bug tracking systems, a developer can mark severity using a seven levels scale. In a previous paper (PDF available), I have shown that not all developers use the same criteria to select the severity, and it should be enough with only three levels. This difference in the developers criteria to mark and classify bug reports is one of the sources of bias in the bug-fix datasets (PDF of the paper available). Another source of bias is developer confidence; not all developers mark commits or bug reports with commit ids when they are starting in a project, because they are afraid of exposing themselves. However, those commits do correspond to bug fixes, and should be accounted for in a bug-fix dataset.

This bias disease affects the Eclipse Bug Data from Software Engineering Chair at Saarland University, which is one of the main data sources used for empirical software engineering. As an example, a paper studying the distribution of software bugs which was based on that Eclipse data has generated a response which has found other better distribution fits, and that does not reuse the same dataset but gathers the data directly from the original sources.

Clearly, reusing datasets for empirical software engineering is a good idea, which fosters reproducibility and verifiability, essential properties of any empirical research discipline. However, if we can not assure the quality of the reusable datasets, reusable datasets can cause more harm than benefits.

My goal with this visit is to apply statistical methods to evaluate the bias in a bug-fix dataset. The two papers about the distribution of bugs in Eclipse are an example of the kind of work I want to do. If we can be sure of the quality and lack of bias of a dataset, carefully built to act as a "canonical" dataset, we can compare other datasets against that canonical dataset, to find out if there is any bias. The two papers about Eclipse mentioned above show that the distribution of bugs can vary in the presence of bias. The first paper used a biased dataset, and the second paper repeated the data gathering process from scratch, avoiding the use of the biased dataset. Although it can also be due to methodological differences, they found different distributions for software bugs.

So my goal is to measure this difference in the distribution using a statistical technique, to detect the presence of bias, and develop a statistical test to find bias in reusable datasets. I am assuming here that the distribution cannot change due to other factors (and we already know that there are other sources of bias in bug reports), and that the shape of the distribution is unique. The second assumption is quite fair, but the first assumption is more complicated, and it will require to find more than one dataset that is known to be unbiased.

I hope this work will provide a tool to assess the quality of a bug-fix dataset, and to avoid the problems of bias, which are a threat to the validity of all the empirical studies using these bug-fix datasets.

Popularity bias in bug datasets

2011-11-01T00:00:00-07:00

In recent times, the replicability of Software Engineering empirical studies has become a main concern in the research community. One way to achieve replicability is by reusing datasets, so everybody base their results on the same data. However, if these datasets contain any kind of problem, they could cause more harm than benefits.

In the case of software defects, there are datasets that are known to contain bias, mainly when referencing a fix to a particular bug report.

We have studied a different kind of bias: popularity bias. A software project with less bugs is of higher quality. However, in open source software development, more bugs may mean more quality. Why? Because more found bugs imply more people looking for those bugs. This is, if you have no bugs it is because nobody is using your software and reporting them. If you have more bugs, it is because your software is popular; should your software be less popular, the number of bugs would be lower. We have studied this effect in the case of Debian, using the Ultimate Debian Database, and we indeed find that only very popular Debian packages will present a very high number of bugs, and that non-popular packages get very few bug reports.

If you want to know more, read our WCRE 2011 paper, entitled "Impact of Installation Counts on Perceived Quality: A Case Study on Debian". A tag cloud of the contents of the paper:

created at TagCrowd.com

To cite this paper, there is a BibTeX file available, or you can copy from below

@InProceedings{debian_wcre2011,
  author =       {Israel Herraiz and Emad Shihab and Thanh H.D. Nguyen and Ahmed E. Hassan},
  title =        {Impact of Installation Counts on Perceived Quality: A Case Study on {D}ebian},
  booktitle = {Proceedings of the 18th Working Conference on Reverse Engineering},
  year =         {2011},
  publisher = {IEEE Computer Society},
}