In my current consulting project, I’m working with a lot of SAS transports files (about 120 for a total of 30 GB). I need to get the data into a format that is usable by dnAnalytics and QuantPro. I usually use DBMS/Copy or StatTransfer to convert SAS file into CSV and then go from there. In this case, I thought I would just write a SAS transport file parser and add it to QuantPro. This would allow me to import the SAS data directly to QuantPro.

The XPORT file format is a really simple, but with some minor gotchas. First, the data is stored in a big-endian byte order. Second, all floating point numbers are stored in an old IBM mainframe format (not IEEE 754). This format is not supported by .NET but it was simple enough to create a converter (but I probably didn’t do it in most efficient manner).

The parser has been added into QuantPro.IO.Sas namespace. It can be used as followed to copy variables from a SAS XPORT file into a QuantPro data store:

var reader = new TransportFileReader("data.xpt");
foreach( var member in reader.Members ){
   using( var qpFile = new ZipFileStore(member.DatasetName) ){
     var variables = reader.GetVariables(member.DatasetName);
     foreach( var variable in variables ){
       qp.Add(variable);
    }
  }
}

The parser supports multiple data members but ignores format labels. Data in SAS transport files is stored as either a string or a floating point number. Format labels can then be used to convert the floating point number into integers or dates. The parser only returns a string or double variable. It is up to the user to do any conversion. I’ll be adding support for format labels sometime in the future.

A little aside about DBMS/Copy. It looks like the utility is no longer available. SAS bought out Conceptual (the company the produced DBMS/Copy), but I cannot find it on the SAS site. That is a shame since it was probably the best tool for converting data from one format to another. Maybe its time to create QuantPro Transfer, a free tool with similar functionality as DBMS/Copy and StatTransfer.

Tweet This Post  Plurk This Post  Delicious This Post 

I was curious about NDepend so I thought I would give it a spin. Wow – it could do much more than I thought. Three features I found very useful are project dependencies, build comparisons, and static analysis (NDepend has many more features). Project dependencies let you see how every detail of your project depends on every other detail – from assemblies to namespaces to types to methods to fields. Want to know which types use a particular a method, no problem. NDepend makes it easy to find where your code is tightly coupled.

Comparing builds allows you to see what was added, removed, or changed (code, names, comments, or items marked as obsolete). I compared the last two dnAnalytics releases and I see that we removed three classes and added six new namespaces, 92 classes, and 960 methods. We also modified 74 methods in 24 classes. This beats having to go through incomplete source control logs and support tickets to generate a change log. There is one glitch. It includes all compiler generated types and methods. I don’t care about them and it would be great if NDepend could filter them out.

My favorite feature is static analysis. NDepend includes a Code Query Language (CQL). CQL lets you query any detail about your code. NDepend has about 150 queries built it – all of which can be customized – and users can write their own. It includes such task as “Show me the methods with more the X lines” or “Fields that can be marked as readonly”. It even has tasks to find breaking changes between builds and detecting naming convention violations. You can integrate NDepend into your build process to flag problems. Quickly going through some of the built-in queries I see that dnAnalytics is using non-generic collections in a couple places and even several instances of throwing Exception (how did that get past a code review?). Queries also include compiler generated classes and there should be an option not to.

I had some minor issues with NDepend. The GUI is a little sluggish and sometime difficult to navigate. But they have predefined views and you can create your own. NDepend generates some default project metrics – two of which are Efferent coupling [1] and Instability [2]. A problem with these metrics is that they incorporate dependencies on the .NET framework base class libraries (BCL). For a single DLL project like dnAnalytics you get odd results. dnAnalytics instability metric is 1 – completely unstable. NDepend should have an option to not include the BCL when computing metrics.

Overall, I’m very impressed. It’s a complex piece of software and there is a bit of learning curve. I recommend that new users watch the short tutorials.

From NDepend’s website:
[1] Efferent coupling (Ce): The number of types inside this assembly that depends on types outside this assembly. High efferent coupling indicates that the concerned assembly is dependant. Notice that types declared in framework assemblies are taken into account.
[2] Instability (I): The ratio of efferent coupling (Ce) to total coupling. I = Ce / (Ce + Ca). This metric is an indicator of the package’s resilience to change. The range for this metric is 0 to 1, with I=0 indicating a completely stable package and I=1 indicating a completely instable package.

Tweet This Post  Plurk This Post  Delicious This Post 

[Update: The code below will perform roughly the same as a non-generic version on the x86 CLR, but is about 30% slower on the x64 CLR. This due to the fact that the x64 CLR doesn't inline methods with struct parameters]

I’ve spent last several days trying to create a generic matrix class, Matrix<T>. Since .NET doesn’t support generic arithmetic [1], it’s not an easy thing to do. My goal was to create a fast, clean implementation – but it seems that fast and clean are mutually exclusive in this case. All the clean designs I came up with were at least twice as slow as the non-generic case. So could I come up with a fast design wasn’t too repulsive? I think I might have.

We are not only going to make the matrix class generic, but also support multiple storage schemes. For this discussion, we are only going to have two – jagged array storage and 1-D array storage. To handle the generic arithmetic problem, we’ll use generic calculators similar to the ones discussed in this CodeProject article – but we are going to hide them from the end user. We are also going to restrict the possible numeric types to double, float, complex, and complex32.

Lets start with a simple matrix class that only supports addition (error checking as been removed to condense the code).

public class Matrix<T> where T : struct {
   private static readonly MatrixCalculator<T> Calculator = new MatrixCalculator<T>();
   private readonly MatrixStorage<T> _storage;

   public Matrix(int rows, int columns){
       _storage = new Dense<T>(rows, columns);
       StorageType = _storage.GetType();
   }

   public Matrix(MatrixStorage<T> storage) {
       _storage = storage;
       StorageType = _storage.GetType();
   }

   public int Rows { get { return _storage.Rows; } }

   public int Columns { get { return _storage.Columns; } }

   public T this[int row, int column] {
       get { return _storage[row, column]; }
       set { _storage[row, column] = value; }
   }

   internal MatrixStorage<T> Storage { get { return _storage; } }

   internal Type StorageType { get; private set; }

   public Matrix<T> Add(Matrix<T> other){
       return Calculator.Add(this, other);
   }

   public static Matrix<T> operator +(Matrix<T> left, Matrix<T> right) {
       return left.Add(right);
   }
}

The matrix class has two dependencies, MatrixStorage and MatrixCalculator.
MatrixStorage:

public abstract class MatrixStorage<T> where T : struct{
   static MatrixStorage(){
       var type = typeof (T);
       if( type != typeof(double) && type != typeof(float) && type != typeof(Complex) && type != typeof(Complex32)) {
           throw new NotSupportedException();
       }
   }

   protected MatrixStorage(int rows, int columns) {
       Rows = rows;
       Columns = columns;
   }

    public int Rows { get; private set; }
    public int Columns { get; private set; }

    public abstract T this[int row, int column] { get; set; }
}

public class Dense<T> : MatrixStorage<T> where T : struct {
   internal T[] _data;

   public Dense(int rows, int columns) : base(rows, columns) {
       _data = new T[rows*columns];
   }

   public override T this[int row, int column] {
       get { return _data[column*Rows + row]; }
       set { _data[column*Rows + row] = value; }
   }
}

public class JaggedDense<T> : MatrixStorage<T> where T : struct {
   internal T[][] _data;

   public JaggedDense(int rows, int columns) : base(rows, columns) {
       _data = new T[rows][];
       for (var row = 0; row < rows; row++) {
           _data[row] = new T[columns];
       }
   }
   public override T this[int row, int column] {
       get { return _data[row][column]; }
       set { _data[row][column] = value; }
   }
}

The MatrixStorage static constructor is where we reject unsupported types. This just shows 1-D and jagged arrays for dense storage, but we can easily add sparse, diagonal, band, identity, and triangular storage schemes.

MatrixCalculator:

internal class MatrixCalculator<T> where T : struct {
   public Matrix<T> Add(Matrix<T> left, Matrix<T> right) {
       return CalculationRunner<T, Matrix<T>>.Execute(typeof (MatrixCalculator<,>), "Add", left, right);
   }
}
internal partial class MatrixCalculator<T, K> where T : struct where K : ICalculator<T>, new() {
   private static readonly K Calculator = new K();
   public Matrix<T> Add(Matrix<T> left, Matrix<T> right) {
       if (left.StorageType == typeof (Dense<T>) && right.StorageType == typeof (Dense<T>)){
           return Add((Dense<T>) left.Storage, (Dense<T>) right.Storage);
       }
       if (left.StorageType == typeof (JaggedDense<T>) && right.StorageType == typeof (JaggedDense<T>)){
           return Add((JaggedDense<T>) left.Storage, (JaggedDense<T>) right.Storage);
       }
       return Add(left.Storage, right.Storage);
   }

    private static Matrix<T> Add(MatrixStorage<T> left, MatrixStorage<T> right){
       var matrix = new Matrix<T>(left.Rows, left.Columns);
       for (var row = 0; row < left.Rows; row++) {
          for (var column = 0; column < left.Columns; column++) {
               matrix[row, column] = Calculator.Add(left[row, column], right[row, column]);
           }
       }
       return matrix;
   }
}

MatrixCalculator<T, K> is a partial class and we’ll add optimized versions of the Add method for 1-D and jagged array storage in separate files – but this will give you the gist.

The MatrixCalculator is dependent on an ICalculator and CalculationRunner. The ICalculator solves the generic arithmetic problem with a negligible impact on performance.

internal interface ICalculator<T> where T : struct {
   T Zero { get; }
   T One { get; }
   T Add(T left, T right);
   T Subtract(T left, T right);
   T Divide(T left, T right);
   T Multiply(T left, T right);
   T Sqrt(T value);
   bool LessThan(T left, T right);
   bool GreaterThan(T left, T right);
   bool LessThanEqualTo(T left, T right);
   bool GreaterThanEqualTo(T left, T right);
   T Conjugate(T value);
}

internal struct DoubleCalculator : ICalculator<double> {
    public double Zero{ get { return 0.0; }  }
    public double One{get { return 1.0; } }

    public double Add(double left, double right)  { return left + right; }
    public double Subtract(double left, double right) { return left - right; }
    public double Divide(double left, double right){ return left/right; }
    public double Multiply(double left, double right){ return left*right; }
    public double Sqrt(double value) { return Math.Sqrt(value); }
    public bool LessThan(double left, double right) { return left < right; }
    public bool GreaterThan(double left, double right) { return left > right; }
    public bool LessThanEqualTo(double left, double right) { return left <= right; }
    public bool GreaterThanEqualTo(double left, double right) { return left >= right; }
    public double Conjugate(double value) { return value; }
 }
internal struct FloatCalculator : ICalculator<float> {...}
internal struct ComplexCalculator : ICalculator<Complex> {...}
internal struct Complex32Calculator : ICalculator<Complex32> {...}

The CalculationRunner is used to instantiate a MatrixCalculator based on the type of T and invoke the requested method. It is also used to create other types of calculation classes such as the matrix decomposition classes. The runner uses reflection to create the classes and caches them – there isn’t too much of a performance hit. The current caching code is a little sloppy, so I’ll just show an outline of the class.

internal static class CalculationRunner<T, K>  {
...
  public static K Execute(Type type, string methodName, params object[] parmeters) {
       MethodInfo method;
       object obj;

       var dataType = typeof (T);

       if (Cache.ContainsKey(type))  {
           var methodCache = Cache[type];
           if (methodCache.ContainsKey(dataType))   {
               var execInfo = methodCache[dataType];
               method = execInfo.Method;
               obj = execInfo.Object;
            }  else  {
                var calcType = GetCalcType(dataType);
                var fullType = type.MakeGenericType(new[] {dataType, calcType});
                method = fullType.GetMethod(methodName);
                obj = Activator.CreateInstance(fullType);
                methodCache.Add(dataType, new ExecutableInfo {Method = method, Object = obj});
           }
       } else {
            var calcType = GetCalcType(dataType);
            var fullType = type.MakeGenericType(new[] {dataType, calcType});
            method = fullType.GetMethod(methodName);
            obj = Activator.CreateInstance(fullType);
            var methodCache = new Dictionary<Type, ExecutableInfo> {{dataType, new ExecutableInfo {Method = method, Object = obj}}};
            Cache.Add(type, methodCache);
       }
       return (K) method.Invoke(obj, parmeters);
   }
....
}

That’s basically it. We can now add double/float/complex/complex32 matrices using either 1-D or jagged arrays for storage. Example:

var defaultMatrix = new Matrix<double>(10,10);
var oneDMatrix = new Matrix<double>(new Dense<double>(10,10));
var jaggedMatrix = new Matrix<double>(new JaggedDense<double>(10,10));
var result = denseMatrix + jaggedMatrix;

Doing some quick benchmarking, there is almost no performance difference between this approach and the non-generic approach. We now need add a similar vector class and all the BLAS and LAPACK algorithm classes – each of which will require a class similar to MatrixCalculator.

Is it worth it? I don’t know. There is a lot of plumbing code – but it does seem to be less code than if we created a whole set of non-generic classes: DenseMatrix, SparseMatrix, FloatDenseMatrix, FloatSparseMatrix, ComplexDenseMatrix, etc.

[1] See https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=94264

Tweet This Post  Plurk This Post  Delicious This Post 

This project has been pushed back a bit. I’ve been busy working on the dnAnalytics’ parallel Map code and the QuantPro data storage. I’ll get back to the ERF rewrite the first week of June. I wanted to start hashing out the sites requirements. User are most interested in viewing, charting, and downloading a particular time series, so I’m going to focus on the time series page requirements.

For a given time series, the series page:

1. should display information about the series – such as its date range, frequency, and unit of measure.
2. should display a chart of the series, where
   • users can set the time range.
   • users can change the title.
   • users can add a second series of the same frequency to the chart.
   • [Optional]users can select the chart format – PNG, GIF, or JPG.
3. should display a summary of the data, that:
   • displays the series’ last 10 values.
   • displays the series’ last 10 period over period values.
   • displays the series’ last 10 year over year values.
   • if the series is not an annual series, convert to an annual series and display
     the last ten values.
   • if the series is not an annual series, convert to an annual series and display
     the last ten values as an annual rate.
4. should allow users to download the series, where:
   • users can select Excel or CSV as the download format.
   • users can include the period over period and year over year values in the download.
   • users can select other series in the same group to be included in the download.
   • users can include the series descriptions in the download.

Tweet This Post  Plurk This Post  Delicious This Post 

While writing the data acquisition programs, I ran into a couple of issues with the current model.  Most notably, NHibernate will not persist null values for a list.  This looks like a NHibernate optimization, since it keeps the values ordered with an index column. So when recreating the list, it knows where to put the null values. The problem is that if the last element in a list is null, that information is lost. That doesn’t work for us, so we’ll use a magic value (double.MinValue) to represent missing values.  This isn’t as bad as it sounds since we had to use Nullable.HasValue calls anyway.  Now we just do a value != double.MinValue (we’ll add a double extension method HasValue that wraps that).

Another change is that we are now storing the date for each observation.  Previously, we were only storing the first observation and then using offsets to determine each elements date.  This was to save storage space. But I forgot that NHibernate stores a position index for each observation anyway.  We might as well store the date and that will remove the need for all the offset calculations.

The new TimeSeries class:

public class TimeSeries : DataItem, IComparable<TimeSeries>, IEquatable<TimeSeries> {
        public TimeSeries()
        {
            Data = new SortedList<DateTime, double>();
        }

        public virtual IndexType IndexType { get; set; }

        public virtual Group Owner { get; set; }

        public virtual bool Reload { get; set; }

        public virtual IDictionary<DateTime, double> Data { get; private set; }
}

The new Mapping:

<hibernate-mapping xmlns="urn:nhibernate-mapping-2.2" assembly="Erf.Data" namespace="Erf.Data.Model">
  <class name="TimeSeries">
    <id name="Id">
      <generator class="native" />
    </id>
    <property name="Enabled" not-null="true"/>
    <property name="Reload" not-null="true"/>
    <property name="DisplayOrder" not-null="true"/>
    <property name="IndexType" not-null="true"/>

    <many-to-one name="Owner" not-null="true" lazy="false"/>

    <map name="Titles" table="TimeSeriesTitles" cascade="all" lazy="false" outer-join="true">
      <key column="ID"/>
      <index column="Language" type="int"></index>
      <element type="string" column="Title" not-null="true"/>
    </map>

    <map name="Data" lazy="true" sort="natural" batch-size="1000" collection-type="sorted-list" generic="true">
      <key column="ItemId"/>
      <index type="DateTime" column="Period"/>
      <element type="double" column="Value" not-null="true"/>
    </map>

  </class>
</hibernate-mapping>

Tweet This Post  Plurk This Post  Delicious This Post 

Our data model is pretty simple.   We have Agencies that provide TimeSeries (the actual data) that are categorized in high level Categories and lower level Groups.  We break our model into four enity types: Agency, Category, Group, and TimeSeries.  An Agency class represents an entity that produces data (such as the Central Bank of Turkey or the Turkish Treasury Department).  Each agency will have a collection of Category objects.  A Category is a high level grouping of data (such Public Finance, Price Index, or Production index).  Categories will have a collection for Group objects.  Groups are a lower level grouping of closely related time series (such as GDP, Consumer Price Index, etc.).  A group also maintains properties of the time series they contain, such the data’s frequency and unit of measure.  A TimeSeries object represents a single series of time ordered data.  So we basically have a tree going from Agencies to Categories to Groups to Time Series.

Each of this classes expose a common set of functionality – the item’s Id, its display title in multiple languages, its display order, and if the item is enabled or not.  We’ll place this functionality in a base class called DataItem.

public abstract class DataItem  {
  protected DataItem() {
    Titles = new Dictionary<Language, string>();
  }

  public virtual int Id { get; protected set; }

  public virtual IDictionary<Language, string> Titles { get; private set; }

  public virtual bool Enabled { get; set; }

  public virtual int DisplayOrder { get; set; }

  public override int GetHashCode()  { return Id.GetHashCode(); }
}

Language is an enumeration of supported languages – currently only English and Turkish.

The Agency class (comments and some code removed for brevity):

public class Agency : DataItem, IEnumerable<Category>, IEquatable<Agency> {
  public Agency() { Categories = new SortedSet<Category>(); }

  protected internal virtual ISet<Category> Categories { get; private set; }

  public virtual IEnumerator<Category> GetEnumerator(){...}

  public virtual void AddCategory(Category category) {
    category.Owner = this;
    Categories.Add(category);
  }

  public virtual void RemoveCategory(Category category)  {
   Categories.Remove(category);
  }

  public virtual void RemoveCategory(int id) {...}

Agency NHibernate mapping file:

<hibernate-mapping xmlns="urn:nhibernate-mapping-2.2" assembly="Erf.Data" namespace="Erf.Data.Model">
  <class name="Agency">
    <id name="Id"> <generator class="native" /></id>
    <property name="Enabled" not-null="true"/>
    <property name="DisplayOrder" not-null="true"/>

    <map name="Titles" table="AgencyTitles"  cascade="all" lazy="false">
      <key column="ID"/>
      <index column="Language" type="int"></index>
      <element type="string" column="Title" not-null="true"/>
    </map>

    <set name="Categories" inverse="true" cascade="all" lazy="true" sort="natural">
      <key column="Owner"/>
      <one-to-many class="Category"/>
    </set>

  </class>
</hibernate-mapping>

We don’t lazy load the titles since the amount of data is small and they’ll be used often. The Category class is almost identical in structure, so we’ll skip it.

Group class:

 public class Group : DataItem, IComparable<Group>, IEnumerable<TimeSeries>, IEquatable<Group> {
  public Group() {
    Series = new SortedSet<TimeSeries>();
    FrequencyLabels = new Dictionary<Language, string>();
    UnitLabels = new Dictionary<Language, string>();
  }

  public virtual Frequency Frequency { get; set; }

  public virtual Category Owner { get; set; }

  public virtual IDictionary<Language, string> FrequencyLabels { get; private set; }

  public virtual IDictionary<Language, string> UnitLabels { get; private set; }

  protected internal virtual ISet<TimeSeries> Series { get; private set; }

  public virtual void AddSeries(TimeSeries series) {
    series.Owner = this;
    Series.Add(series);
  }

  public virtual void RemoveSeries(TimeSeries series){
    Series.Remove(series);
  }

  public virtual void RemoveSeries(int id){...}
}

Similar structure as the Agency class, but now we add information about the time series the group holds – frequency and unit of measure.

<hibernate-mapping xmlns="urn:nhibernate-mapping-2.2" assembly="Erf.Data" namespace="Erf.Data.Model">
  <class name="Group" table="GroupTable">
    <id name="Id">
      <generator class="native" />
    </id>
    <property name="Enabled" not-null="true"/>
    <property name="DisplayOrder" not-null="true"/>

    <many-to-one name="Owner" not-null="true"/>

    <map name="Titles" table="GroupTitles" cascade="all" lazy="false">
      <key column="ID"/>
      <index column="Language" type="int"></index>
      <element type="string" column="Title" not-null="true"/>
    </map>

    <map name="FrequencyLabels" cascade="all" lazy="false">
      <key column="ID"/>
      <index column="Language" type="int"></index>
      <element type="string" column="Title" not-null="true"/>
    </map>

    <map name="UnitLabels" cascade="all" lazy="false">
      <key column="ID"/>
      <index column="Language" type="int"></index>
      <element type="string" column="Title" not-null="true"/>
    </map>

    <set name="Series" inverse="true" cascade="all" lazy="true" sort="natural">
      <key column="Owner"/>
      <one-to-many class="TimeSeries"/>
    </set>

    <joined-subclass name="TcbGroup">
      <key column="TcbGroupId"/>
      <property name="Database" column="DB" not-null="true" unique="true"/>
    </joined-subclass>

    <joined-subclass name="TreasuryGroup">
      <key column="TreasuryGroupId"/>
      <property name="Url" column="URL" not-null="true" unique="true"/>
      <property name="LastUpdated" column="UPDATED" not-null="true" />
    </joined-subclass>

  </class>
</hibernate-mapping>

The mapping file differs from the Agency mapping file in that we now have subclasses of Group. These subclasses (TcbGroup and TreasuryGroup) store information we use to retrieve the group’s series. If you are building a system off of this, you would modify the mapping to include your own subclasses. Here using Fluent NHibernate would be better than mapping files, since users extending the system wouldn’t have to muck around with these mapping files (replacing my subclasses with theirs). We would put the subclass mapping into another DLL and users wouldn’t reference it. I just couldn’t get Fluent NHibernate to work with joined subclasses. I’ll probably remove the mapping files once Fluent NHibernate can support them.

Finally, we have our TimeSeries class:

 public class TimeSeries : DataItem, IComparable<TimeSeries>, IEquatable<TimeSeries> {
  private DateTime _firstPeriod;

  protected TimeSeries() { Data = new List<double?>(); }

  public TimeSeries(DateTime firstPeriod) : this()  { _firstPeriod = firstPeriod; }

  public virtual DateTime FirstPeriod  {
            get { return _firstPeriod; }
            set { _firstPeriod = value; }
  }

   public virtual double? this[DateTime date] { get{...} }

   public virtual IndexType IndexType { get; set; }

   public virtual Group Owner { get; set; }

   public virtual bool Reload { get; set; }

   public virtual IList<double?> Data { get; private set; }
}

Again similar structure. The IndexType enumeration gives us information about the series. Are the series values at an annual rate, are the series an annual over annual series, both, or just a plain series? The Reload properties tell us that we should reload all the series data at the next update. Generally, I only update the last year of a series’ data when adding new data. Sometimes there is a major revision and we want to update it all.

For storing the actual time series data, we are not using a map (DateTime, double?). The reason is to save database space. Our database is currently about 400MB and adding a date for each observation, would increase the storage significantly. The only issue with this approach is that you have to make sure you have a data point (a null for missing dates) for each date. We’ll use the Erf.Data.Extensions.DateExtensions DateTime extension methods to help us here.

TimeSeries mapping file:

<hibernate-mapping xmlns="urn:nhibernate-mapping-2.2" assembly="Erf.Data" namespace="Erf.Data.Model">
  <class name="TimeSeries">
    <id name="Id">
      <generator class="native" />
    </id>
    <property name="Enabled" not-null="true"/>
    <property name="Reload" not-null="true"/>
    <property name="DisplayOrder" not-null="true"/>
    <property name="IndexType" not-null="true"/>
    <property name="FirstPeriod" not-null="true"/>

    <many-to-one name="Owner" not-null="true" lazy="false"/>

    <map name="Titles" table="TimeSeriesTitles" cascade="all" lazy="false" outer-join="true">
      <key column="ID"/>
      <index column="Language" type="int"></index>
      <element type="string" column="Title" not-null="true"/>
    </map>

    <list name="Data" cascade="all" lazy="true" batch-size="1000" generic="true" >
      <key column="ItemId"/>
      <index column="Position"/>
      <element type="double" column="Value" not-null="false"/>
    </list>

    <joined-subclass name="TcbTimeSeries">
      <key column="TcbTimeSeriesId"/>
      <property name="Mnemonic" not-null="true" unique="true"/>
    </joined-subclass>
  </class>
</hibernate-mapping>

The only difference here is that we don’t lazy load the series owner/group. The reason is that we need to the series frequency and unit when displaying the series data. We also have a subclass, TcbTimeSeries, that exposes the FAME mnemonic for the series (used to retrieve the data). Again, you’ll have to create subclasses of TimeSeries if you need to store any additional information and create your own joined subclass mapping.

Our data repository classes are really simple: Add, Update, Remove, and GetById. 

public interface IRepository<T> where T : DataItem {
  void Add(T item);
  void Update(T item);
  void Remove(T item);
  T GetById(int id);
}

The also have a class specific method to load a detached object’s collection. For example the AgencyRepository:

public class AgencyRespository : Repository<Agency>{
  public void LoadCategories(Agency agency) {
    using (var session = SessionFactory.OpenSession()) {
      session.Lock(agency, LockMode.None);
        NHibernateUtil.Initialize(agency.Categories);
    }
  }
}

The data model appears pretty anemic, but it is sufficient for what we need – retrieve time series data for displaying, charting, placing into data files (Excel, delimited, etc), and passing the data off to analysis programs.

I’ll be spending next couple days rewriting our data retrieval and update programs to work with the new model. That code is commissioned by the ERF and I’m not allowed (yet) to release it. Once that’s done, we’ll work on porting the web site the new model and ASP.NET MVC.

Tweet This Post  Plurk This Post  Delicious This Post 

I’m starting a short series of posts on rewriting the Economic Research Forum (ERF) data site.  This a brief introduction to the project.  I’ll write about the data model in the next day or so.

What is the Economic Research Forum Data Site
The purpose of the Economic Research Forum’s data web site is to serve as a central repository for Turkish economic data.  The site currently hosts data from the Central Bank of Turkey and Koc University’s Economics Department.  The site will be adding more data sources over time with the Turkish Treasury Department next on the list.

Background
The site was originally developed using DevExpress’ ASP.NET controls and eXpress Persistent Objects (XPO) hosted on Windows Server 2003 with SQL Server 2005.  Our goal is to convert the site to 100% free [1] and/or open source software running under Linux.  We’ll be migrating from DevExpress’s ASP.NET controls to ExtJS, XPO to NHibnernate, and SQL Server to MySQL.  We’ll be running the site on Debian 5 using Mono 2.4/2.6.  The move is not about open source idealism, but dollars and cents.

Source Code
The code for site is released under the GPL v3 with an open source exception.  The exception allows any OSI approved open source project to use the code without having to GPL their own code.  The code is up on GitHub – http://github.com/cuda/erf/tree .  So far only the data model and related tests have been checked in.  We’ll start adding the web site code next week.

[1] free as in beer, not the Free Software Foundation’s definition of free (as in speech).

Tweet This Post  Plurk This Post  Delicious This Post 

dnAnalytics 2009.4 will be released at the end of the month. So its time to start planning on what I can add to the next release.

One tedious task is going to be updating the unit tests.  The unit tests were started in 2003 and are a mess. They are mixed in with integration tests and numerical accuracy tests, and have inconsistent naming and Assert patterns.  I’d like to break the tests into three separate projects and standardize the unit test naming.  Also, I’d like to add some type of random testing to help catch corner case problems.  I’ve been planning on using FsCheck to handle this, but recently learned about the Theory tests in NUnit 2.5.   I need take a deeper look at them and see if they can accomplish the same thing.  It would be nice to use only one tool.

As for new code, I’m looking to add parallel map extension methods and genetic optimization.  The map methods will allow users to something like the following on vectors, matrices, and IList<double> types:

obj.Map(x=>1/x};
obj.Map(Math.Sqrt);

The first inverts the each element of the that object and the second would replace their values with their square root.  We’ll parallelize this so it uses all available cores.

A project I completed last month made heavy use of genetic optimization.  It was amazing how simple it was to solve complex problems.  I think this will a very helpful addition to the library.  I’ll start a blog series on creating the optimization framework when I start working on the code.

If I can, I’d like to squeeze in matrix and vector views.   Views will allow users to treat sub-matrices/vector and arrays as actual matrices and vectors.  This has two advantages – less copying overhead and reduced memory usage. Also, if I can pick up a relatively cheap NIVIDA GTX 200 series video card, I can start adding NVIDIA CUDA support to the native version.

Tweet This Post  Plurk This Post  Delicious This Post 

The next release of dnAnalytics is going to provide a Map class that applies a “function” to each element of a vector or matrix.   Since each mapping of the function is independent of the others, it’s a great candidate to parallelize.  .NET 4.0 provides a parallel for loop (Parallel.For) that is perfect for this.  However, we need to support .NET 2.0.  The plan is to create a drop in replacement for Parallel.For that can be used under .NET 2.0.  Then depending on the end user’s environment, dnAnalytics will use either the .NET 4.0 Parallel.For or our clone.  We should be clear that this is NOT a general replacement for the .NET 4.0 parallel for loop.  The clone will be optimized for dnAnalytics, not for general applications.

Below is my thought process (or thinking out loud) of creating the Parallel.For clone.  The code is a work in progress and has not been optimized.

For our Parrallel.For loop, we have three options:

1. Create threads on the fly.
2. Use the .NET ThreadPool.
3. Create our own thread pool.

We’re going with option 3, since it gives us control over the number of running threads and minimizes thread creation.

internal static class ThreadQueue{
   private static readonly Queue _queue = new Queue();
   private static int _count;
   private static bool _running = true;
   private static Thread[] _threads = SetThreads(2);

   public static int ThreadCount{
      get { return _count; }
   }

   private static Thread[] SetThreads(int count){
      _count = count;
      var threads = new Thread[_count];

      for (var i = 0; i < _count; i++) {
         threads[i] = new Thread(Start);
         threads[i].IsBackground = true;
         threads[i].Start();
      }
      return threads;
   }

   public static void Enqueue(Task task){
      lock (_queue){
         _queue.Enqueue(task);
         Monitor.Pulse(_queue);
      }
   }

   private static void Start() {
      while (true){
         Task task = null;
         lock (_queue){
            if (_queue.Count > 0){
               task = _queue.Dequeue();
            } else{
               Monitor.Wait(_queue);
            }
         }
         if (task != null){
            task.Compute();
            task.Set();
         }
         if (!_running){
            break;
         }
      }
   }
}

The thread queue starts up N threads and the threads wait for a Task to be queued. In the sample code the number of threads is set to 2, but we’ll make that configurable and default to the number of CPU cores. Next, we need to create the Task class. Here we are going to deviate from the System.Threading.Task design by having the Task class extend EventWaitHandle. This will simplify thread notification. Our Task implementation is going to be for internal use only, so deviating is not a problem.

internal class Task : EventWaitHandle{
   private readonly Action _body;

   public Task(Action body) : base(false, EventResetMode.ManualReset){
      if (body == null){
         throw new ArgumentNullException("body");
      }
      _body = body;
   }

   public Exception Exception { get; private set; }

   public bool ThrowException { get { return Exception != null; }  }

   public void Compute()  {
      try {
         _body();
      } catch (Exception e){
         Exception = e;
      }
   }
}

The Task class will swallow any exceptions thrown by the mapped function (Compute method), but it will be aggregated with any other exceptions in the actually Parallel.For loop and re-thrown as an AggregateException.

Now we can create the actual For loop.

public static class Parallel{
   public static void For(int fromInclusive, int toExclusive, Action body){
      if (body == null){
         throw new ArgumentNullException("body");
      }
      var count = toExclusive - fromInclusive;
      var size = count/ThreadQueue.ThreadCount;
      if (count < 1) {
         return;
      }

      var actions = new List();
      for (var i = 0; i < ThreadQueue.ThreadCount - 1; i++){
         var start = fromInclusive + (i*size);
          var stop = fromInclusive + ((i + 1)*size);
          actions.Add( () =>; {
             for (var j = start; j < stop; j++) {
                body(j);
             }
          } ) ;
       }
       actions.Add( () =>  {
          for (var i = fromInclusive + ((ThreadQueue.ThreadCount - 1)*size); i < toExclusive; i++){
             body(i);
          }
       } );
       Invoke(actions.ToArray());
   }

   public static void Invoke(params Action[] actions) {
      var tasks = new List();
      foreach (var action in actions){
         tasks.Add(new Task(action));
      }
      foreach (var task in tasks){
         ThreadQueue.Enqueue(task);
      }
      WaitHandle.WaitAll(tasks.ToArray());

      var exceptions = new List();
      foreach (var task in tasks) {
         if(task.ThrowException) {
            exceptions.Add(task.Exception);
         }
      }
      if( exceptions.Count > 0) {
         throw new AggregateException(exceptions);
      }
   }
}

Real simple. Break the loop into X number of tasks, queue up the tasks, wait for them to finish, and then check for any exceptions. Using only X number of thread tasks works well for us, since these are CPU bound loops. If that wasn’t the case, there would be idle threads. You could then increase performance by breaking up the loop into smaller tasks (feeding the idle threads).

So how does this perform? Lets compare it to serial code and against the Parallel Extension Library (PEL) June CTP (I don’t have access to the VS 2010 CTP). We’ll look at two cases, (A) a computational light operation (1/i) and then (B) a slightly heavier operation (Sinh(90) + Cosh(90)). We’ll also use two array sizes, 1000 and 1000000.

Case A
N=1000, our Code is 3.2x slower than the serial code and the PEL code is 7.7x slower.
N=1000000, our Code is 37% faster than the serial code PEL code is 18% faster.

Case B
N=1000, our Code is 1.1x slower than the serial code and the PEL code is 1.6x slower.
N=1000000, our Code is 43% faster than the serial code PEL code is 33% faster.

These results make sense.  When the size of the array is relatively small, the threading overhead outweighs the parallel gain.   And as the the operation performed gets more computationally intense, the threading overhead becomes smaller relative to the computing time.

Folks might be wondering why our code seems to out perform the Parallel Extension Library.  This is probably due to two reasons. First, PEL is a CTP release and isn’t optimized for speed.  Second, we are focusing only one type of usage – tight numerical loops.  PEL has to be optimized to handle any type of loop.

The code stills needs some work before it is integrated into the dnAnalytics code base.

Update: A modified version of the code above has been added to dnAnalytics.

Tweet This Post  Plurk This Post  Delicious This Post 

I upgraded to Sandcastle Help File Builder (SHFB) 1.8 and it wasn’t finding our LaTeX build component.   It looks like SHFB now longer looks for the “BuildComponents” directory in the program directory but in %ProgramData%.  After moving the component there, it works fine.  I’ve updated the documentation to reflect this change.

I started compiling mimeTeX with Intel’s C++ compiler and having it auto-vectorize (using SSE2) its loops where possible.  This should speed it up a bit.  The new build can be downloaded from http://cuda.net/downloads/latex_sandcastle.zip.

The source has also been moved over to GitHub and the I setup a Lighthouse issue tracker for the project.

Tweet This Post  Plurk This Post  Delicious This Post