FLOSS for Science

SOFA Statistics: An Interview with developer Dr. Grant Paton-Simpson

edwood — Wed, 10 Oct 2012 10:00:43 +0000

Our interviews are unique and there is nothing like them in the web.
Take with you the first 30 Floss4Science Interviews

Dr. Grant Paton-Simpson

This time we bring you an interview with Dr. Grant Paton-Simpson the developer of SOFA (Statistics Open For All) a statistics and reporting application. Grant has a PhD in Sociology and has provided database and analysis services to the gambling treatment sector continuously since 1997. Have a good read!

F4S: Hi Grant. Please, give us a brief introduction about yourself.

I have been a data analyst and researcher since the mid 90s, mainly in the health sector. Was an SPSS user from about 1989. Discovered that most analysis in many sectors is very rudimentary (simple descriptive tables, maybe a barchart, possibly a t-test). Thought there would be a gap for a program that makes the 20% of functionality used 80% of the time very easy to use. I have four children but still find enough peace and quiet to think and program. Badminton makes a nice change from sitting in front of a keyboard.

F4S: What is SOFA Statistics?

SOFA (Statistics Open For All) is a user-friendly statistics, analysis, and reporting program. It is free and open source, of course, with an emphasis on ease of use, learn as you go, and beautiful output.

F4S: Why and when did SOFA come to be?

I’d had the idea of making an analysis application for many years. I used to refer to it as the “octopus” with the ability to put its tentacles into a variety of different databases and extract results. It was once I became an experienced Python developer that I decided to give it a go, and bit by bit it became SOFA. SOFA can import data from a range of different spreadsheets and can dynamically link to SQLite, MySQL, PostgreSQLL, CUBRID, MS Access, and MS SQL Server databases. There is nothing like being able to think of an improvement you’d like to see in a program you rely on – and being able to bring it into being as quickly as you are able.

F4S: In which language(s) and platform(s) is the project developed?

Python (with wxPython as the GI toolkit) with packages for Windows, Mac, Ubuntu, and other Linux distros.

F4S: Does SOFA have sponsors?

Not at present.

F4S: How many users you estimate SOFA Statistics have?

I really don’t know but on Sourceforge it has passed the 100,000 downloads milestone some time ago. On the one hand you have to exclude existing users downloading multiple times, and people using SOFA temporarily. On the other there are other ways of getting SOFA than Sourceforge e.g. from teachers, via heise online etc.

F4S: Do you know where SOFA Statistics is used?
I have some idea from the support emails I receive. Some examples to illustrate: the lead researcher in an Asian genetics lab, an agricultural statistician in New Zealand, a nutrition teacher in the French part of East India, a geotechnical researcher in Israel, a Freelance researcher in Sweden, a professor of children’s health in the US, and a student at a Brazilian university. Some rely on SOFA for everything they need; others use SOFA for some tasks e.g. pumping out lots of attractive exploratory tables.

F4S: How many team members does the project have?
Basically it is just me with help on a few things (gratefully received) from others. Often an individual will help me nail down a bug intensively for a while and then it is solved. There have also been a number of people providing translations, and J. David Eisenberg deserves special mention for the videos he has made.

F4S: In what areas of SOFA’s development do you currently need help?
I would like to add some new tests e.g. around regression, or multivariate ANOVAs but I need some reliable, trustworthy open source Python libraries I can include.

F4S: What features are in the roadmap?
My next priority is to add some more plug-in extensions around the open source core. I have already created export output functionality that makes it easy to export reports as high-quality images and or PDFs ready to put into documents or slideshows. And there is also a plug-in for exporting data tables to spreadsheet. In keeping with the SOFA ethos of statistics open for all, prices have been kept low, and there are student discounts to make sure no-one misses out. In case of hardship, I have invited people to email me directly for a free copy of the relevant plug-in.

F4S: Which projects, blogs or sites related to open source software for science can you recommend?
I have a list of links here: http://www.sofastatistics.com/help.php but here are two to give a taste:

I should also give http://www.statschat.org.nz/ a plug as well.

F4S: Why do you consider free/libre open source software important for the advancement of your field?

There is something about the spirit of enquiry which doesn’t sit well purely commercial imperatives. Far too often there are costly pay-walls around the tools and resources people need to explore and understand. I am a big fan of open source and it has been good to see it becoming more and more widespread.

F4S: Is there any other topic you would like our readers to know about?
I’ll make a quick plug for learning Python as a way of opening up access to a wide range of very powerful scientific tools.

F4S: Where people can contact you and learn more about SOFA Statistics?

Website: http://www.sofastatistics.com
Blog: http://www.sofastatistics.com/blog/
Email: grant@sofastatistics.com
Twitter: https://twitter.com/sofastatistics
identi.ca: http://identi.ca/sofastatistics/all
Facebook: http://www.facebook.com/pages/SOFA-Statistics/201087293255544
Google Group: https://groups.google.com/forum/?fromgroups#!forum/sofastatistics
YouTube: http://www.youtube.com/user/pykiwi?feature=guide

F4S: Thank you Grant and success to you and this project.

More SOFA videos at http://www.sofastatistics.com/videos.php

Note: If you liked this interview you can Flattr it at the top of the post!

How to get these interviews in your mobile library ...

Books that may interest you

Recent Books on Geospatial Free/Libre Open Source Software

edwood — Mon, 08 Oct 2012 10:00:05 +0000

These are recent geospatial books involving free/libre open source software (from 2010 to 2012): PostGIS, OpenStreetMap, OpenLayers, GDAL/OGR and QGIS among others.

Note: Books have affiliate links.

The Geospatial Desktop

This book provides a foundational level of knowledge for understanding GIS and the open source desktop mapping applications that are available for use, for free, today. Learn about vector and raster data, how to convert data, interacting with spatial databases, creating new map data, geoprocessing, scripting, and more. Special sections include focused learning on the Quantum GIS and GRASS GIS software platforms but other packages are also introduced. The Geospatial Desktop is written by the creator of Quantum GIS, so you can rest assured that you will be led by one of the most knowledgeable authors on the subject.

Programming GPS and OpenStreetMap Applications with Java: The RealObject Application Framework

Written by an expert in the development of GPS systems with digital maps and navigation, Programming GPS and OpenStreetMap Applications with Java: The RealObject Application Framework provides a concrete paradigm for object-oriented modeling and programming. It presents a thorough introduction to the use of available global positioning data for the development of applications involving digital maps.

Online Maps with APIs and WebServices (Lecture Notes in Geoinformation and Cartography)

The Internet has become the major form of map delivery. The current presentation of maps is based on the use of online services. This session examines developments related to online methods of map delivery, particularly Application Programmer Interfaces (APIs) and MapServices in general, including Google Maps API and similar services. Map mashups have had a major impact on how spatial information is presented. The advantage of using a major online mapping site is that the maps represent a common and recognizable representation of the world. Overlaying features on top of these maps provides a frame of reference for the map user. A particular advantage for thematic mapping is the ability to spatially reference thematic data.

Geospatial Free and Open Source Software in the 21st Century (Lecture Notes in Geoinformation and Cartography)

This book contains papers presented at the first Open Source Geospatial Research Symposium held in Nantes City, France, 8-10 July, 2009. It brings together insights and ideas in the fields of Geospatial Information and Geoinformatics. It demonstrates the scientific community dynamism related to open source and free software as well as in defining new concepts, standards or tools.

Springer Handbook of Geographic Information

Computer science provides a powerful tool that was virtually unknown three generations ago. Some of the classical fields of knowledge are geodesy (surveying), cartography, and geography. Electronics have revolutionized geodetic methods. Cartography has faced the dominance of the computer that results in simplified cartographic products. All three fields make use of basic components such as the Internet and databases. The Springer Handbook of Geographic Information is organized in three parts, Basics, Geographic Information and Applications. Some parts of the basics belong to the larger field of computer science. However, the reader gets a comprehensive view on geographic information because the topics selected from computer science have a close relation to geographic information. The Springer Handbook of Geographic Information is written for scientists at universities and industry as well as advanced and PhD students.

Open Source Based Web-GIS Technology for Voter ID Verification: A Case Study of K.K.Nagar, Tiruchirappalli, Tamil Nadu, India

Open source is an approach to design, development, and distribution offering practical accessibility to a product’s source (goods and knowledge). A Geographic Information System is a system for management, analysis, and display of geographic knowledge, which is represented using a series of information sets such as maps, geographic data sets, processing and work flow models, data models, and metadata. Electoral GIS is an important information system to find out the voters list with their residential map and their addresses to check the correct address with their location. In present time, open source initiative has gained momentum and projects are being developed using open source products. Hence, an attempt is made to develop a GIS based application tool for Voter ID address verification with their residential maps. With the help of this application, people and officials can verify the address and solve the address mismatches.

Participatory Geospatial Development Using Python (Volume 1)

The book contains discussion on raster and vector data processing using Python binding of GDAL/OGR library. Also, different approaches of representing spatial reference system are enumerated. There is a discussion on LIDAR data processing using Python binding of libLAS library. Apart from processing geospatial data, the book also covers plotting of geospatial data. The last chapter deals with freely available geospatial data, such as ASTER GDEM, SRTM data.

OpenLayers Cookbook

60 recipes to create GIS web applications with the open source JavaScript library

Understand the main concepts about maps, layers, controls, protocols, events etc
Learn about the important tile providers and WMS servers
Packed with code examples and screenshots for practical, easy learning

PostGIS in Action

PostGIS in Action is the first book devoted entirely to PostGIS. It will help both new and experienced users write spatial queries to solve real-world problems. For those with experience in more traditional relational databases, this book provides a background in vector-based GIS so you can quickly move to analyzing, viewing, and mapping data. Advanced users will learn how to optimize queries for maximum speed, simplify geometries for greater efficiency, and create custom functions suited specifically to their applications. It also discusses the new features available in PostgreSQL 9.0 /9.1 and PostGIS 2.0 and provides tutorials on using additional open source GIS tools in conjunction with PostGIS.

OpenStreetMap

This book introduces the OSM project, its aims and objectives, and its history, then guides you through the process of gathering, editing, and using OpenStreetMap data using a series of real-world examples. This book is the perfect aid for geographic-information professionals interested in using OpenStreetMap in their work and web designers and developers who want to include mapping in their sites, and want a distinctive style. It is for you if you have a need to use maps and geographic data for work or leisure, and want accurate, up-to-date maps showing the information you’re interested in, without details you don’t need. If you want to use maps for navigation and want more or less detail than traditional printed maps give this book is perfect for you.

Python Geospatial Development

The GIS Guide to Public Domain Data

The GIS Guide to Public Domain Data gives users of geographic information systems (GIS) relevant information about the sources and quality of available public domain spatial data. Readers will understand how to find, evaluate, and analyze data to solve location-based problems. This guide covers practical issues such as copyrights, cloud computing, online data portals, volunteered geographic information, and international data. Supplementary exercises are available online to help put the concepts into practice. Students, researchers, and professionals will find The GIS Guide to Public Domain Data a useful desk companion to help them navigate the world of spatial data in the public domain.

GNU Octave: An interview with John W. Eaton and Jordi Gutiérrez

edwood — Wed, 26 Sep 2012 10:00:39 +0000

Our interviews are unique and there is nothing like them in the web.
Take with you the first 30 Floss4Science Interviews

John W. Eaton

Jordi Gutiérrez

What a great interview we have for you this week! John W. Eaton, THE developer of GNU Octave, and Jordi Gutiérrez, one of their main developers and organizer of the OctConf 2012 at Montréal Canada (slides and report), took time from their busy schedules to share with our readers the story behind the venerable GNU Octave and details of this still very active project. Have a great read!

F4S: Hello guys. Please, give us a brief introduction about yourselves.

[John] I’m the original author of GNU Octave and have been it’s maintainer from the beginning, in 1992. When I first started working on Octave I was post-doctoral researcher and systems administrator at the University of Texas. Then from 1995 until 2008 I was a researcher at the University of Wisconsin. But most of my time from 1992 until 2008 was spent working on Octave. Now I have my own software support company focused on supporting Octave.

[Jordi] I call myself a mathematician, coder, and quixotic hacker-errant. I have a degree and a half in mathematics, and I started using Octave during that second half of the latter degree. After some time, I found myself in Octave development, and now I want to work on making it better for everyone. My current employer is in the medical imaging processing business, and they originally hired me for helping them to integrate Octave into their pipeline.

F4S: What is GNU Octave?

[Jordi] Octave is a free numerical environment mostly compatible with Matlab. The idea is to liberate all of the code written for Matlab and make it run in Octave with as little modification as possible; optimally, with no modification. This is a very difficult task, because Matlab is a moving target and its language keeps changing. We’re always going to be at least a little behind Matlab’s changes. In a few cases historically we have implemented a feature before Matlab, but then Matlab implemented it in a slightly different way from us. This makes it very difficult to innovate too much at the language level, so we are fairly conservative about it.

For development of packages, analogous to Matlab’s toolboxes, we have Octave-Forge, which has more lax standards of contribution. We are currently trying to bring the two projects together, because we have some organisational difficulties between the two.

F4S: Why and when did GNU Octave come to be?

[John] Octave was originally intended to be companion software for an undergraduate-level textbook on chemical reactor design being written by James B. Rawlings of the University of Wisconsin-Madison and John G. Ekerdt of the University of Texas. As I recall, we started talking about writing software for the textbook in the summer of 1988 or maybe 1989. But I didn’t start working on it until after I finished my Ph.D. thesis in the beginning of 1992.

Initially, we planned to write some Fortran subroutines and distribute them with the textbook as was common at the time. But prior to writing any code for the textbook, we also became convinced that a library of Fortran subroutines would not be very successful, in part because of experiences observing undergraduate students struggle with mechanics of computational assignments. We found that they were spending too much time trying to make a Fortran program compile and run and not enough time understanding the ideas that the computations were intended to illustrate.

It was clear that we needed to make it easier to perform the computations required to solve reaction engineering problems, but we also firmly believed that we should not make the software too simplistic. For example, we quickly rejected the idea of creating some graphical software that would not allow the students to do more than vary parameters for limited example problems as we had seen in other software distributed with textbooks at the time. We were certain that it is essential that students have some understanding of programming concepts, not just data entry. We also believed that it would be beneficial for students to be able to examine all parts of the computation to see how calculations really work behind the scenes. This last point, and the objective of allowing the software to be distributed and used with or without the textbook, led us to make Octave free software.

Eventually, we settled on the idea of writing a small language suited to the numerical computations needed to illustrate the concepts of chemical reactor design. Since a growing number of our colleagues were beginning to use Matlab as their primary computational tool, we choose to use a mostly Matlab-compatible syntax so that our colleagues could use Octave without having to learn a completely new language.

The initial goal was not to create a system completely compatible with Matlab, and Octave has introduced a number of features that were not present in Matlab. However, since Octave was first released there has been steady and increasing pressure from Octave users to make it more compatible with Matlab. Now instead of trying to correct any perceived misfeatures of Matlab, the Octave community is more likely to remove the differences, making an ever larger body of existing Matlab code (much of it also free software) accessible to the community. Ideally, it should be trivial to use Octave to run code written for Matlab.

F4S: In which language(s) and platform(s) is the project developed?

[Jordi] Mostly C++ and Octave’s own scripting language, together with a bit of Fortran. Indeed, Octave started out as a wrapper to Fortran libraries, and it still does that. As of today, it’s about 600,000 lines of C++, about 100,000 lines of m-files (Octave’s own language), and about 50,000 lines of Fortran.

F4S: Does GNU Octave have sponsors?

[Jordi] The FSF handles donations for us. In the past, there have been other sponsors, but we don’t have any right now.

F4S: How are the sponsors supporting the project?
[John] So far, we’ve used funds collected by the FSF to help cover travel expenses for a meeting of Octave developers held in Montreal in July 2012. That meeting was quite productive. Bringing many of the most active contributors together has definitely helped the project.

My advisor at the University of Texas and the University of Wisconsin, Jim Rawlings, secured most of the funding to pay for my initial work on Octave through a combination of government and industrial research grants. Since leaving the University of Wisconsin, I’ve been doing contract work to support my work on the project.

With more sponsors, we might be able to pay some developers to work on Octave full time. Currently, all paid work on Octave is done by contracts with individual developers, not through the Octave project itself.

F4S: How many users you estimate GNU Octave have?

[Jordi] Based on download numbers and popularity contest reports from Ubuntu and Debian, I estimate from 10,000 to 100,000 users. I can’t do much better than a loose order of magnitude.

[John] I think there were several hundred thousand downloads of Octave 3.2.4 for Windows. I don’t think the current numbers on source-forge are correct. Anyway, I think 100,000 may be low, but I don’t really have a good idea what the number is.

F4S: Do you know where GNU Octave is used?

[Jordi] We have a healthy mix of users from academia and industry. As I said above, my own workplace also uses Octave. We regularly ask that question, and these are the responses we get:

Who Uses Octave?

F4S: How many team members does the project have?
[Jordi] There are 13 people currently with write access to the main repository, and we have had 296 contributors over all time. In addition, our sister project Octave-Forge has 49 users with write access, 38 of which made a commit last year.

F4S: How can people get involved with the project?
[Jordi] We have a FAQ for this:

How can I get involved in Octave development?

F4S: What features are in the roadmap?
[Jordi] There are two very exciting developments: a native GUI, and JIT compiling. The GUI has taken a long time to develop properly, mostly because the core Octave developers have never really thought it was necessary. We’re finally listening to the waves of users who have always asked for a GUI, so it’s finally coming. JIT compiling is a feature that will greatly speed up non-vectorised code.

F4S: Which projects, blogs or sites related to open source software for science can you recommend?
[Jordi] I don’t know many, but I really like this blog post about how science is done right now and how it could be done:

http://marciovm.com/i-want-a-github-of-science/

It’s a little starry-eyed, but I’m a dreamer too.

F4S: Why do you consider free/libre open source software important for the advancement of your field?
[Jordi] It is absolutely essential for science to be *reproducible*. How often does one find results in published articles that allude to software that is never seen and instead they only give broad strokes of the algorithms used? I wish scientists treated source code with the same respect they treat publications and articles: as part of their scientific activity and something that requires careful work.

F4S: Where people can contact you and learn more about GNU Octave?

[Jordi] http://octave.org, http://planet.octave.org/, http://identi.ca/octave, and I think people are working on a Facebook and Twitter presence. We’re only starting to be more social online, so some of these are only starting.

F4S: Thank you very much John and Jordi for sharing with us more about you and GNU Octave.

GNU Octave in Action

Note: If you liked this interview you can Flattr it at the top of the post!

How to get these interviews in your mobile library ...

GNU Octave Manual Version 3

GNU Octave Beginner's Guide

No related posts.

WinPython scientific distribution

edwood — Thu, 20 Sep 2012 11:24:30 +0000

WinPython is a portable scientific Python distribution for Windows (32 and 64-bits) built on top of NumPy, SciPy, matplotlib and the scientific environment Spyder:

WinPython, a portable scientific Python distribution for Windows (32&64bit)

Beginning Python Visualization

Turn many types of data sources into useful visual data.

Pandas plotting examples

edwood — Thu, 20 Sep 2012 11:11:25 +0000

A blog published by Vytautas Jančauskas contains examples of various plotting techniques using pandas charting functions:

Plotting for Pandas GSoC2012

It appears to be worke done during Google Summer of Code 2012.

ggplot2: Elegant Graphics for Data Analysis (Use R)

This book will be useful to everyone who has struggled with displaying their data in an informative and attractive way. You will need some basic knowledge of R (i.e. you should be able to get your data into R), but ggplot2 is a mini-language specifically tailored for producing graphics, and you'll learn everything you need in the book. After reading this book you'll be able to produce graphics customized precisely for your problem.

Guidebook to R Graphics Using Microsoft Windows

Throughout the book, concise explanations of key concepts of R graphics assist readers in carrying out the presented procedures, and any coverage of functions is clearly written out and displayed in the text as demos. The discussed techniques are accompanied by a wealth of screenshots and graphics with related R code available on the book's FTP site, and numerous exercises allow readers to test their understanding of the presented material. Guidebook to R Graphics Using Microsoft® Windows is a valuable resource for researchers in the fields of statistics, public health, business, and the life and social sciences who use or would like to learn how to use R to create visual representations of data. The book can also be used as a supplement for courses on statistical analysis at the upper-undergraduate level.

Tutorial pandas DataFrames and SciPy

edwood — Wed, 19 Sep 2012 11:26:36 +0000

Randy Olson published a tutorial on how to use pandas DataFrames from SciPy including examples on how to perform the Mann-Whitney-Wilcoxon (MWW) RankSum test and Analysis of Variance (ANOVA):

Statistical analysis made easy in Python

Worth a bookmark!

The R Book

The R Book is packed with worked examples, providing an all inclusive guide to R, ideal for novice and more accomplished users alike. The book assumes no background in statistics or computing and introduces the advantages of the R environment, detailing its applications in a wide range of disciplines.

BioRuby 1.4.3 available for download

edwood — Tue, 18 Sep 2012 12:15:55 +0000

Version 1.4.3 of the BioRuby project has been released.

Download at: BioRuby.

BioRuby project provides an integrated environment in bioinformatics for the Ruby language. This project is supported by University of Tokyo (Human Genome Center), Kyoto University (Bioinformatics Center) and the Open Bio Foundation.

You can also browse the bookstore ...

BioRuby 1.4.2 download

OSGeo-Live 6.0 geospatial distribution available for download

edwood — Tue, 18 Sep 2012 12:09:48 +0000

OSGeo live DVD distibution version 6.0 is available. This version i based on Xubuntu 12.04 LTS (Long Term Support).

Download: OSGeo-Live 6.0.

OSGeo-Live is a self-contained bootable DVD, USB thumb drive or Virtual Machine that allows you to try a wide variety of open source geospatial software without installing anything.

Books on the topic

GeoTools 8.1 available for download

edwood — Tue, 18 Sep 2012 12:03:18 +0000

GeoTools is an open source Java library that provides tools for geospatial data. Version 8.1 of the library is available:

GeoTools: GeoTools 8.1 Released

Books on the topic

No related posts.

SciPy 2012 videos

edwood — Sat, 15 Sep 2012 11:19:58 +0000

The SciPy 2012 videos are available at YouTube and PyVideo.org (you should bookmark this!):

SciPy 2012 Videos

There are a total of 62 videos. Enjoy!