Intent

Gradient Descent in Dynamo

2017-09-27T09:26:00.002-07:00

Hello and Welcome back. So far we have been able to develop a simple regression model and solved it using linear algebra. In this article, I am going to explore another alternative approach called Gradient Descent. It’s an iterative method to optimize a cost function by finding a local/global minimum by taking steps proportional to the gradient of the cost function. To apply the Gradient Descent algorithm we need to first define our cost function for our machine learning problem.

We have a set of features for which the actual output is given we want to develop a prediction model to predict the output for the given set of features. We can represent our feature vectors for the complete training set as matrix X, the corresponding output values as vector y, and the predicted values as vector \hat{y}. So our prediction model is \hat{y} = X*a and the error vector for the training set in the model is err = \hat{y} - y. We would like to choose a cost function such that the overall error of the model could be minimized. Thus we choose the mean square error as our cost function. The dot product of the error vector with itself gives the sum of the square of errors and dividing it by the number of training examples gives us the mean square error. Here is the Dynamo graph to compute mean square error for a given training set.

Since the cost function we have chosen is quadratic (square of error), it has one global minima and we can find it by traversing along the gradient. Now the gradient of the cost function is obtained by finding the partial derivation of the cost function with the coefficient vector and it can be simplified as G = \alpha (\hat{y} - y) * X^{T}, the numeric constant alpha is the learning rate. Using this gradient we can now update the value of coefficient vector ‘a’ as a := a - \alpha (\hat{y} - y) * X^{T} The Dynamo graph to update any given coefficient vector ‘a’ is shown below. Note that one extra iterator input node, when we wrap it into a custom node LinearRegressionGradientDescent. This iterator is used to run an iteration in Dynamo using List.Reduce node.

We now assemble these pieces together to evaluate our regression model. We need to initialize the coefficient vector with ones and create a list of repeated items with this coefficient vector and the amount of iteration we want to do for evaluation. This list is then fed to the List.Reduce node with initial/seed coefficient vector and the custom node function as the reducer. This is a unique way to run imperative loop and update the value using nodes in Dynamo. We could have used imperative code block and run a loop to achieve the same, but I prefer to use map, reduce a concept from functional programming paradigm in Dynamo. To get a better result and faster conversion we should also normalize our feature vectors. The complete Dynamo graph is shown below. We can again use test and validation set to find the optimum alpha for our dataset.

Conclusion: In this article, I have shown how to define the cost function and implement gradient descent in Dynamo. In fact, the gradient descent concept can be used to optimize any other cost function for which we know how to compute the gradient. I have also used List.Reduce higher-order function node to do the iterations. I hope you liked this post, please do send your feedback and comments.

Polynomial regression and model comparison

2017-09-13T00:24:00.000-07:00

Welcome back, in my previous post I described how we can perform linear regression using normal equation in Dynamo and I left with a question "what if the input and output are not linearly dependent?” Let’s say we have a hypothesis that the housing price doesn’t depend on floor area and number of rooms linearly but it has a following relationship as y = a_{0} * x_{0} + a_{1} * x_{1} + a_{2} * x_{2} + a_{3} * x_{1} * x_{2} + a_{4} * x_{1}^2 + a_{5} * x_{2}^2 \: where x_{0} = 1, \: x_{1} is floor area and x_{2} is number of rooms. Then we can introduce few new parameters x_{3} = x_{1} * x_{2}, \: x_{4} = x_{1}^2, \: x_{5} = x_{2}^2 and then perform the linear regression to find the coefficient matrix.

The Dynamo graph to setup the feature vector looks as follows. Once the feature vector is setup rest all is same as previous linear regression example.

Note that the price prediction for a given floor area and rooms we need to again construct the same feature vector. The price prediction based on the complete dataset and this new hypothesis for a 3 room house of area 1650 sq. ft. is 307,176 whereas the price predicted based on previous linear regression model is 293,081. Now the sellers can argue that the polynomial model is good because it is fetching them the higher price and similarly buyers can argue that linear regression model is better because it gives a lower price for his desired house. To identify the best model we need to build a test set and use that test set to figure out which model gives less error. We can use 80% of the dataset to evaluate the regression model and rest 20% we can use to test both the models. While splitting the dataset into a training set and test set we must shuffle the data before the split just to avoid any bias present in the dataset.

The Dynamo graph to split the dataset into training and testing is shown below. We shuffle the data row-wise and then extract sub matrix based on the percentage of the dataset to be used for training.

Now we can solve the linear regression model using Matrix.SolveLinearEquation node, which uses the same technique to evaluate the regression model as discussed in the previous blog post on Linear Regression. Once we have computed the two models (linear and polynomial) we can now run this model with the test set to get the error value. The total error for a model is the sum of the absolute error for each of the predictions as compared to the actual price data. The comparison of the two errors shows that the linear regression model performs better on the test set hence we should choose linear regression as compared to the polynomial regression hypothesis we discussed here. The error computation graph in Dynamo is shown below.

The final graph to run both the models and error comparison is shown below and it is shared at Github.

Conclusion: In this article, I have demonstrated how we can model polynomial equation based hypothesis and solve for the prediction model using linear regression and compare the two hypothesis based on the error value on a test set which is different than the training set. As always, please do share your feedback and suggestions on this article as well as the DynamoAI package on this page or on Github.

Linear Regression in Dynamo

2017-09-03T20:05:00.003-07:00

Today we are collecting more data than ever but making sense out of the data is very challenging. Machine learning helps us analyze the data and build an analytical model to help us with future prediction. To create a machine learning prediction model we usually develop a hypothesis based on our observation of the collected data and fine tune the model to reduce the cost function by training the model. One of the very simple hypothesis we can develop by assuming a linear relationship between input and output parameters.

Suppose we have data on housing price and we assume that housing prices are linearly related to the floor area of the house then we can use linear regression to predict the price of a house with specific floor area. One of the most important steps towards building the hypothesis is being able to visualize the data and understand the trend. So let first draw a scatter plot of our data as shown in the figure below. I am using Grapher package to draw the scatter plot.

The scatter plot shows that the relationship between floor area and price is not quite linear but it can be approximated to a linear curve. In another words the linear regression is a curve fitting problem. Let’s say that the desired curve is y = mx + c
Hence to build our model we need to find out the values of m and c. We can use linear algebra and represent this equation as y = A*X^{T} where A represents a matrix of coefficients i.e. A = \begin{bmatrix}c & m\end{bmatrix} and \:X = \begin{bmatrix}1 & x\end{bmatrix} and X^{T} is transpose of X. Now for a give y we can build our model to find the coefficient vector A, which can be obtained as A = (X^{T} * X)^{-1} * X^{T} * y.

Let’s say that we have a hypothesis that says the house price not just depend on floor area but it also depends on number of bedrooms. The parameters such as floor area or number of bedrooms are called features. To generalize we can express y as y = a_{0} + a_{1}*x_{1} + a_{2}*x_{2} + a_{3}*x_{3}... where the feature vector X can be represented as X = \begin{bmatrix} 1 & x_{1} & x_{2} & x_{3} &...\end{bmatrix} and the coefficient vector A can be represented as A = \begin{bmatrix} a_{0} & a_{1} & a_{2} & a_{3} &...\end{bmatrix}. Suppose we have ‘m’ samples of data to fit the curve and y depends on 'n' features, then the size of A is '1 x (n+1)' whereas the size of X is ‘m x (n+1)’ where the first column has all 1. To fit the optimal curve we would like to find out the coefficient vector A so that we can predict the value of y for any new feature vector X.

So now let's look at the Dynamo graph compute the coefficient vector A.

To predict new price based on the input feature vector we need to take a dot product with the coefficient vector. This provides a simple Linear regression model which works well as long as the inverse of (X^{T} * X) can be computed efficiently. For a very large number of features, computing inverse could be very expensive also if any two or more features are inter-dependent then also inverse can’t be computed because of singularity. If we take care of these two aspects then it’s pretty effective model.

What if the linearity is not a good hypothesis and we think that there is a polynomial relationship between input and output. We can still use the same computational model by making the polynomial terms as a new feature in the model.

Note: The Dynamo Graph demonstrated here, uses DynamoAI package and is shared at GitHub. I hope you would find it useful and please do send me your feedback on GitHub or on this page.

DesignScript

2017-08-16T16:40:00.001-07:00

It’s been a really long time that I wrote about something. For past many years, I have been busy working on a domain specific language called DesignScript and it's visual programming interface Dynamo. I am now moving to a world of reality-capture and I’ll miss working on this technology. I find this technology very useful and every now and then I’ll keep coming to it for some geometric and programming experiment. Recently I am learning machine learning and wondering if I can use Dynamo to explain some of the concepts of machine learning. I’ll explore it further but in the mean time let me explain what is DesignScript and Dynamo.

DesignScript is a scripting language intended to provide programming capability to designers. Designers can now describe their design-intent effectively using this language. Unlike other procedural languages, it provides two great features such as associativity and replication. The associativity in expressions helps it preserve the intent whereas the replication makes it agnostic to the dimensionality of the data. One can establish relationships easily by defining expressions where left-hand side depends on the parameters used in the right-hand side of the expression. e.g.

a = 5;
b = 2 * a; //‘b’ is twice the value of ‘a’, thus ‘b’ depends on ‘a';

The relationship defined by such expressions are associative in nature, i.e. whenever the dependent parameter(s)/variable(s) is(are) modified the left-hand side variable is automatically updated. And the evaluation of this expression is agnostic to the dimensionality of the parameters used in the expression. The parameter ‘a’ in above expression is a singleton value, but later in the program, if we change the value of a to a collection then the expression gets re-evaluated and ‘b’ is updated to a collection, which means we don’t need to run a for-loop to evaluate the expression.

a= 1..5; //‘a’ is a collection of integers from 1 to 5 and now b = {2, 4, 6, 8, 10} due to associativity of ‘b = 2 * a;’ expression.

There are few other features such as zero touch library import allows users to import any .NET library into DesignScript without modifying the library and the public class and methods defined in the library can be called seamlessly from DesignScript.

Dynamo provides a visual programming interface to DesignScript users. They can define their programming logic as a data flow program. Dynamo also support python scripts and users can creates a reusable script using custom nodes. They can also publish their workflow for others to consume it in their design using package manager. More about DesignScript and Dynamo can be learned from http://dynamoprimer.com/ or http://dynamobim.org

Associativity in AutoCAD

2011-01-29T17:41:00.000-08:00

CAD applications often implement features or entities which are associative in nature, for example creating an extrude surface in AutoCAD 2011, by extruding a profile creates associative extrude surface. When you modify the profile the extruded surface will update itself to follow the profile.
AutoCAD developers now can develop fascinating applications using new associative framework. There are APIs available in C++ as well as .NET to implement associative features in AutoCAD. You can download the latest ObjectARX to make use of associative framework.
The associative framework in AutoCAD helps us to maintain relationship between objects. Relations are represented as hierarchical Networks of Objects, Actions and Dependencies between them. The core building blocks are:

Action - Defines behavior or intelligence of application or feature.
Dependency- Maintains associativity/ relations between objects and notifies changes on dependent objects.
Network- The associative model
Object- A regular AutoCAD AcDbObject/AcDbEntity being controlled by an associative action.

Now let's code a simple action where we want to establish a relationship between a line and a circle, where length of line controls diameter of the circle. For this we need to implement an Action class derived from AcDbAssocActionBody, which provides the evaluation logic.

//A custom action derived from AcDbAssocActionBody, establishes relationship
//between a line and a circle. Length of the line drives diameter of the circle.
//
class MyActionBody : public AcDbAssocActionBody
{
public:
ACRX_DECLARE_MEMBERS(MyActionBody);

virtual void evaluateOverride();

static Acad::ErrorStatus createInstance(AcDbObjectId lineId, AcDbObjectId circleId, AcDbObjectId& actionBodyId);
};

Establishing relationship between objects:

This action depends on both line and circle, so we need to attach dependencies on both line and circle. Since we are going to read the length of line and write/update diameter of the circle, so we attach read dependency on line and write dependency on circle. These dependencies are owned by our action and they will ensure that line is driving entity and circle is the driven entity.

Acad::ErrorStatus MyActionBody::createInstance(AcDbObjectId lineId, 
    AcDbObjectId circleId, AcDbObjectId& actionBodyId)
{
    //Get the space to which network/action needs to be added.
    AcDbObjectId ownerSpaceId;
    {
        AcDbObjectPointer< AcDbEntity > pEntity(lineId, AcDb::kForRead);
        if(!eOkVerify(pEntity.openStatus()))
            return pEntity.openStatus();
        ownerSpaceId = pEntity->ownerId();
    }

    //create a new action and action body on the given space.
    AcDbObjectId actionId;
    AcDbAssocActionBody::createActionAndActionBodyAndPostToDatabase(
        MyActionBody::desc(), ownerSpaceId, actionId, actionBodyId);

    AcDbObjectPointer< AcDbAssocAction > pAction(actionId, AcDb::kForWrite);
    if(!eOkVerify(pAction.openStatus()))
        return pAction.openStatus();

    //Read dependency on line, to read it's length.
    AcDbAssocDependency* pDep = new AcDbAssocDependency();
    pAction->database()->addAcDbObject(pDep);
    pDep->attachToObject(lineId);
    pDep->setIsReadDependency(true);
    pDep->setIsWriteDependency(false);
    pAction->addDependency(pDep->objectId());
    pDep->close();

    //Write dependency on circle to set it's radius equal to length of line.
    pDep = new AcDbAssocDependency();
    pAction->database()->addAcDbObject(pDep);
    pDep->attachToObject(circleId);
    pDep->setIsReadDependency(false);
    pDep->setIsWriteDependency(true);
    pAction->addDependency(pDep->objectId());
    pDep->close();

    return Acad::eOk;
}

Evaluation logic:

Now the evaluation of action needs to ensure that the desired behavior is maintained after every evaluation. This requires implementation of evaluateOverride() method on the action body class.

void MyActionBody::evaluateOverride()
{
    AcDbAssocEvaluationCallback* pCallback = this->currentEvaluationCallback();

    AcDbObjectIdArray depIds;
    Acad::ErrorStatus es = getDependencies(true, true, depIds);
    AcDbObjectId lineId, circleId;
    for(int i = 0; i < depIds.length(); ++i)
    {
        AcDbObjectPointer< AcDbAssocDependency > pDep(depIds[i], AcDb::kForRead);
        if(!eOkVerify(pDep.openStatus()))
        {
            //Reporting error to the callback.
            if(NULL != pCallback)
            {
                AcDbObjectPointer< AcDbAssocAction > pAction(this->parentAction(), AcDb::kForRead);
                pCallback->setActionEvaluationErrorStatus(pAction, pDep.openStatus(), depIds[i]);
            }
            setStatus(kFailedToEvaluateAssocStatus);
            return;
        }

        if(pDep->status() == kErasedAssocStatus)
        {
            //If any dependent object is erased, the corresponding dependency
            //will be mared as kErasedAssocStatus. For this example, we need to
            //erase the action if dependency is to be erased.
            setStatus(kErasedAssocStatus);
            return;
        }

        if(!pDep->isWriteDependency()) //read only dependency
            lineId = pDep->dependentOnObject();
        else if(!pDep->isReadDependency()) //write only dependency
            circleId = pDep->dependentOnObject();
    }

    assert(!lineId.isNull() && !circleId.isNull());
    if(lineId.isNull() || circleId.isNull())
    {
        setStatus(kFailedToEvaluateAssocStatus);
        return; //you may report some error here.
    }
    
    //Evaluate dependencies
    evaluateDependencies();

    double length = 0.0;
    //Use AcDbAssocObjectPointer to open line for read, so that
    //we get the cloned copy created by dragger, when line is dragged.
    AcDbAssocObjectPointer< AcDbLine > pLine(lineId, AcDb::kForRead);
    if(!eOkVerify(pLine.openStatus()))
    {
        setStatus(kFailedToEvaluateAssocStatus);
        return; //you may report some error here.
    }

    AcGePoint3d start, end;
    es = pLine->getStartPoint(start);
    es = pLine->getEndPoint(end);

    length = start.distanceTo(end);

    //Use AcDbAssocObjectPointer to open circle for write, so that
    //we get the clone of circle managed by dragger used to draws updated 
    //circle when line is dragged.
    AcDbAssocObjectPointer< AcDbCircle > pCircle(circleId, AcDb::kForWrite);
    if(eOkVerify(pCircle.openStatus()))
        pCircle->setRadius(length/2);

    setStatus(kIsUpToDateAssocStatus); //Finally set the status as uptodate.
}

What’s Intent?

2011-01-28T18:49:00.001-08:00

Every action has equal and opposite reaction but today we need to know the intent behind an action to react properly. If you can understand the intent you are smart. So what’s the intent, how do you understand it? It’s a thought, an idea, a concept which can be derived from the context by reading between the lines or understanding the gestures. It’s understanding of “I in you”.
Understanding intent leads to providing the best solution for a problem, it leads to innovation. For example, searching for a restaurant on mobile phone has an intent and providing the best search result based on the context/location is innovation. In design paradigm every entity drawn on the canvas has some intent and it can be modified meaningfully only if one understand the intent of the entity. But when author is changed or the tools to modify these entities are changed, most often intents are lost. Capturing intent from actions develops knowledge base and developing knowledge base helps us understand the intent better.
Welcome to my first edition of Intent.