Los Techies

Python Web Framework Series – Pylons Part 7: Refactoring, Deployment And Wrap-Up

Ryan Svihla — Sat, 04 Jul 2009 12:00:00 GMT

Lets take a look at our existing site and what we can do to clean it up and add some badly needed functionality, however that is beyond the scope of my series. So I’m going to leave some hints for the remaining functionality to get you started.

Refactoring Ideas

Replace default authkit login with custom login. HINT: make a sign-in link on your public page. then redirect to a custom login page that uses authkit. Trying it any other way will make you lose hair.
Refactor database access into a class or two. This will make the db access more testable and allow you to monkey patch in different behavior later. HINT: If your controllers can avoid having to know anything about SQLAlchemy you’ve done your job.
Replace return “rsvihla” in the users.py module with actual authkit user. HINT: just access pylons request object: “request.environ.get("REMOTE_USER")”
Create admin ui for users, with roles. HINT: look at Pylons Book for Authkit ideas here.
Split threads into categories and then give the ability to respond to posts. HINT: just more controllers and views.
Makes parts of your view visible depending on if a user has certain rights or not. HINT: while this is not what you’ll find in the Pylons Book I highly suggest not have gobs of if/else checks in your views. Make small functions that you can unit test easily which will change what is visible depending on role, then have your views call these. See what I did with my user.py module for a concept.

Deployment

This almost completely depends on your situation with hosting. If you are using something like WebFaction you just have to upload files and follow their directions. I found it trivial to have a production site up and running with them.

Now if you are using your own dedicated server or want to shrink wrap the project I suggest using Python Eggs, this is not pylons specific at all and I’ve built them for a number of other projects. There are some snags you can run into depend on your platform and how portable you want to make this but as long as your dev platform and production platform are the same all you need is “python setup.py bdist_egg” and you should have a egg file in the dist directory which can be installed with easy_install.

Finally, you’ll want to build a production.ini . By and large you can copy your development.ini but you must must make your debug = false or you’ll give people access to your application in all sorts of bad ways. Also, don’t forget to make your secret strings actually secret strings.

Overall Impressions

Things I like:

Pretty easy to get up and running quickly.
paster and nose integration is pretty slick.
being able to functional test response including view is actually nice. Really wish I had this in Monorail as easily.
Component based to the hilt. If you really don’t like something you can switch it out.

Things I don't like:

AuthKit is like all auth frameworks I’ve used, just a lot to make work the way I want.
Too much violation of dry in the config. AuthKit being the main culprit here, while I may want to change which datasource I use, I’m not sure how helpful it is to have to select “form, cookie” for each ini file I make.
Explicitly calling the view in my actions. Should have some convention over configuration here.

So I could certainly see using Pylons more once I get my own auth solution worked out, and get my own code flow working I could actually probably get work done more quickly in it than most web frameworks I’ve learned. The majority of the time I wasted learning was dealing with authkit, in fact that took more time than everything else I studied combined.

Thank you for taking the time to get this far with my posts. If you have any specifics parts you want me to dive into more depth and/or suggestions for me on the next series (which will be Pyxer with Google App Engine) let me know.

Theory Of Constraints: Productivity Metrics in Software Development

derick.bailey — Fri, 03 Jul 2009 17:38:16 GMT

In the past, I’ve been a true believer that software development is not really possible to measure from a productivity perspective. I was ignorant, basically. I’m now a bit wiser and I understand that software development is no different than any other product development process. We can and should measure productivity of software developers by understanding that we are building business value via functionality that the end-user wants. So, we should essentially be measuring our progress toward the end user facing functionality (as an over-simplification).

A Paper On The Theory Of Constraints

Several months ago, I wrote up a paper for an internal company effort to help define productivity metrics for software developers. This paper is largely based on the work of David J. Anderson, in “Agile Management For Software Engineering”. It also includes some of my own interpretations and understandings of the Theory of Constraints.

The original intent of this paper was to facilitate the discussion of productivity and metrics in the Development Department at McLane Advanced Technologies, LLC., where I work. I decided to release it to the world, to hopefully help others understand how we can measure productivity as software developers. I did not intend this to be a comprehensive or exhaustive discussion of the points within, but am trying to spur additional research and conversations. I hope you enjoy reading it as much as I enjoyed writing it.

Download The Paper

You can download the PDF here:

The Theory of Constraints: Productivity Metrics In Software Development

How not to do Dependency Injection, in NerdDinner

bogardj — Fri, 03 Jul 2009 16:19:29 GMT

Checking out the NerdDinner code the other day, I found a common Dependency Injection anti-pattern. One of the core concepts of DI is that components are not responsible for locating their own dependencies. The code went part of the way to full-on DI, but not quite far enough. Here’s the offending code:

public class SearchController : Controller {

IDinnerRepository dinnerRepository;

//
// Dependency Injection enabled constructors

public SearchController()
    : this(new DinnerRepository()) {
}

public SearchController(IDinnerRepository repository) {
    dinnerRepository = repository;
}

The second constructor is correct – the SearchController’s dependencies are passed in through the constructor. This is because the IDinnerRepository is a required, or primal dependency. SearchController depends on IDinnerRepository to function properly, and won’t work without it. But on the first constructor, we violate DI by having the SearchController create a concrete DinnerRepository! We’re now back to concrete, opaque dependencies. We have a small benefit of easier testability, but we still force our controller to locate its own dependencies.

So why is this a Bad Thing?

For one, it’s confusing to have two constructors. Why go through all the trouble of creating the IDinnerRepository interface if I’m just going to depend directly on an implementer? Now what if DinnerRepository now needs some dependency? What if it now needs some ILogger, security or policy dependency? Do I now need to go fix all of my calls to “new”?

And that’s the whole point of Dependency Injection, and the Dependency Inversion Principle. I only know about my first level dependencies, and abstractions on top of that. If we check out DinnerRepository, we see another issue:

public class DinnerRepository : NerdDinner.Models.IDinnerRepository {

    NerdDinnerDataContext db = new NerdDinnerDataContext();

A private, opaque dependency to NerdDinnerDataContext. If we wanted to make that opaque dependency explicit, we’d have to fix our SearchController (and all the other controllers as well). It’s these kinds of ripple effects that prevent refactoring and improvement.

Fixing it

But we can fix this, quite easily. From MvcContrib, we can grab any one of the ControllerFactory implementations for the IoC container of our choice. For me, that’s StructureMap. First, we’ll need to configure StructureMap to wire up all of our dependencies. The preferred way to do so is to create a Registry:

public class NerdDinnerRegistry : Registry
{
    public NerdDinnerRegistry()
    {
        Scan(scanner =>
        {
            scanner.TheCallingAssembly();
            scanner.WithDefaultConventions();
        });

        ForRequestedType<NerdDinnerDataContext>()
            .CacheBy(InstanceScope.HttpContext)
            .TheDefault.Is.ConstructedBy(() => new NerdDinnerDataContext());
    }
}

In the first part, we just tell StructureMap to scan the calling assembly with default conventions. This will wire up IDinnerRepository to DinnerRepository. Probably 99% of our dependencies will be taken care of in that one call. Next, we need to wire up the NerdDinnerDataContext (for reasons we’ll see soon). Since that class has multiple constructors, StructureMap likes to use the greediest dependency, with the most arguments. I don’t want that, so I override it to use the no-arg constructor. Finally, I cache it by HttpContext, though I could probably go Singleton if it’s expensive to instantiate.

Next, I need a wrapper class to initialize StructureMap:

public static class IoCContainer
{
    public static void Configure()
    {
        ObjectFactory.Initialize(init =>
        {
            init.AddRegistry<NerdDinnerRegistry>();
        });
    }
}

Initialize only needs to get called once per AppDomain, and all I need to do is add the NerdDinnerRegistry I created earlier. I could wire up everything here, but again, the preferred configuration method is through registries. The next piece is to wire up ASP.NET MVC to both call our configuration method and use the StructureMapControllerFactory:

void Application_Start()
{
    RegisterRoutes(RouteTable.Routes);

    IoCContainer.Configure();
    ControllerBuilder.Current.SetControllerFactory(new StructureMapControllerFactory());

    ViewEngines.Engines.Clear();
    ViewEngines.Engines.Add(new MobileCapableWebFormViewEngine());
}

I put all this in with the other MVC configuration in the Global.asax Application_Start method. One final piece is to remove the extra constructors on our controller:

public class DinnersController : Controller {

    IDinnerRepository dinnerRepository;

    //
    // Dependency Injection enabled constructors

    public DinnersController(IDinnerRepository repository) {
        dinnerRepository = repository;
    }

And make our DinnerRepository expose its dependencies explicitly:

public class DinnerRepository : NerdDinner.Models.IDinnerRepository {
    
    private readonly NerdDinnerDataContext db;

    public DinnerRepository(NerdDinnerDataContext db)
    {
        this.db = db;
    }

Note that there are no no-arg constructors to be found! When StructureMap creates a DinnersController, it will look to resolve the IDinnerRepository dependency. That’s a DinnerRepository, which in turn needs the NerdDinnerDataContext. But that’s all hidden from us, we never need to wire up an entire graph, as we would if we stuck to the “new” operator. Just to make sure everything works, I can navigate to view upcoming dinners:

Wrapping it up

In the original NerdDinner code, dependency inversion was only partially implemented as the original controllers still contained a constructor that called a constructor of a concrete class. To fix this, we added StructureMap to the mix, configuring ASP.NET MVC to use StructureMap to create controllers instead of the default controller factory. Finally, we pushed the dependency inversion principle all the way down to our repository, removing the opaque dependencies where we found them. When we finished, all of our classes exposed their dependencies explicitly through their constructor. No one class knew more than one level of depth, and our controllers now properly depended exclusively on the IDinnerRepository abstraction.

In the future, we can add things like logging, policies and the like to custom IDinnerRepository implementations, without needing to change any of our controllers. Once we introduce inversion of control to our application, we open a lot of doors for functionality and simplicity, but only going halfway won’t give us the full benefit.

Python Web Framework Series – Pylons: Part 6 Basic Authorization With AuthKit

Ryan Svihla — Fri, 03 Jul 2009 12:00:00 GMT

Last post we left off with very basic database access, and testing story completed. Now we're going to look at basic Authorization and Authentication with AuthKit. NOTE: most of this post is just an aggregation of a couple of chapters in the Pylons Book since this setup is a good base starting point. Read the previous link to the Pylons Book for more in depth coverage of this topic.

Setting Up AuthKit

First lets make sure we have AuthKit installed: easy_install AuthKit. For this post we're working with AuthKit 0.4.3, your mileage may vary if you read this post in the future and are using a different version. Now that we have Authkit installed open up pylonsforum\config\middleware.py add the following imports:

import authkit.authenticate

from authkit.permissions import ValidAuthKitUser

and then add somewhere inside the if asbool(full_stack): code block.

permission = ValidAuthKitUser()

app = authkit.authorize.middleware(app, permission)

app = authkit.authenticate.middleware(app,app_conf)

in development.ini add this to the end of your [app:main] section

authkit.setup.enable = true

authkit.setup.method = form, cookie

authkit.form.authenticate.user.type = authkit.users.sqlalchemy_driver:UsersFromDatabase

authkit.form.authenticate.user.data = pylonsforum.model

authkit.cookie.secret = secret string

authkit.cookie.signoutpath = /home/signout

Open your home.py controller and for now add a “signout” action:

def signout(self):

    return "You've been signed out"

Now in your websetup.py we have a ton to add to get the basic setup working. Start right after imports and add these line.

from authkit.users.sqlalchemy_driver import UsersFromDatabase

next add the following in your “setup_app” method after load_environment

    from pylonsforum.model import meta

    meta.metadata.bind = meta.engine

    filename = os.path.split(conf.filename)[-1]

    log.info("Adding the AuthKit model...")

    users = UsersFromDatabase(model)

    meta.metadata.create_all(checkfirst=True)

    log.info("Adding roles and uses...")

    users.user_create("admin", password="admin")

For the final piece delete your development.db file and run paster setup.app development.ini to recreate it with the AuthKit user model. Now you have very basic authentication working in your site

http://localhost:5000 reveals:

type in “admin” for the username and password and it should let you pass. Note going back to the site will not bring up a password box again.

http://localhost:5000/home/signout

will remove your cookie and you’ll see the sign in form once more if you go to http://localhost:5000 . Stayed tuned for more posts as I go more in depth with the different features and customizations of AuthKit.

Python Web Framework Series – Pylons: Part 5 Testing Models

Ryan Svihla — Thu, 02 Jul 2009 20:09:11 GMT

When we last left of with our Pylons forum we had a had just successfully created a post, and then could immediately retrieve that same post. However, we kind of skimped on the testing story so lets fill in the gaps and do some refactoring as a bonus.

Functional Testing Models

Open your functional controller test located at pylonsforum\tests\functional\test_home.py and add the following import line

from pylonsforum.model import Thread, Post, meta

and the following methods

def setUp(self):

    meta.Session.remove()

    meta.metadata.create_all(meta.engine)

    self._createpost(5)

def _createpost(self,times):

        i = 0

        while i < times:

            i = i +1

            thread = Thread()

            thread.subject = "subject " + str(i)

            post = Post()

            post.author = "posting away"

            post.content= ""

            post.isparent = True

            thread.posts.append(post)

            meta.Session.add(thread)

        meta.Session.commit()

and change the test_recent_posts method to

   def test_five_most_recent_threads_show_in_homepage(self):

        response = self.app.post(url(controller="home", action="index"))

        assert "subject 1" in response

        assert "subject 2" in response

        assert "subject 3" in response

        assert "subject 4" in response

        assert "subject 5" in response

In summary here we’re adding a _createpost factory method that generates some basic test threads and parent posts. running nosetests –-with-pylons=test.ini should result in failed tests.

Changing The View and Controller

Now adjust your index.mako view to the following:

<%def name="title()">Pylons Forum</%def>

<%def name="head_tags()"></%def>

<div id="recentposts">

<table>

<thead><tr><th>subject</th><th>author</th><th>date submitted</th></tr></thead>

<tbody>

%for t in c.threads:

    <tr><td>${t.subject}</td><td>${t.parentpost.author}</td><td>${t.dateadded}</td></tr>  <!-- changed posts properties to thread properties and added parentpost -->

%endfor

</tbody>

</table>

</div>

So once you’ve changed the to using a “threads” object, added parentpost (where did that come from? I’ll get to that in minute) and changed our properties around we need to change the our home.py controller index action for our cosmetic changes and to work with our new tests:

 def index(self):

        c.username = "rsvihla"

   # removing this line    c.posts = [Post("jkruse", "New Kindle", "06/24/2009")]

        thread_query = meta.Session.query(model.Thread)

        c.threads = thread_query.order_by(model.Thread.dateadded.desc()).limit(5).all()

        return render('index.mako')

So our index action now is querying for the 5 most recent threads and storing in our new c.threads context variable. Lets strip the query down into pieces.

thread_query = meta.Session.query(model.Thread)

we’re working with SQLAlchemy Session object. We call then store a Thread query object inside the thread_query variable. Now through the thread_query variable we can get access Thread objects stored in the database.

c.threads = thread_query.order_by(model.Thread.dateadded.desc()).limit(5).all()

Here we call “order_by” on our query object and then pass in the Thread model specifying a descending order of the dateadded property. Next we limit it to 5 rows and then call “all()” which returns our results in a nice python “list” object.

Custom Properties on the Model

Ok so back a few paragraphs ago in the index.mako view I stuck in a “parentpost” property on the thread, and I’m quite certain you’re wondering how I did that. So I’ve created the following class tests\test_model.py

from pylonsforum.model import Thread, Post, meta

from pylonsforum.tests import *

class TestThreadParent(TestController):

    def _makethread(self, hasparent):

        thread = Thread()

        thread.subject = "test"

        parentpost = Post()

        parentpost.content = ""

        parentpost.author = "first post"

        parentpost.isparent = hasparent

        childpost = Post()

        childpost.content = ""

        childpost.author = ""

        childpost.isparent = False

        thread.posts.append(childpost)

        thread.posts.append(parentpost)

        return thread

    def test_should_find_parent_post(self):

        thread = self._makethread(True)

        assert thread.parentpost.author == "first post"

    def test_should_not_find_parent_when_none_set(self):

        thread = self._makethread(False)

        assert thread.parentpost is None

Our test has a _makethread factory method (another one of those maybe time for a factory class soon!) which can make a parent post or not depending on its parameter, then two tests documenting its behavior in each situation.

So I’ve gone back to our models\__init__.py class and changed the Thread class to look like this:

class Thread(object):

    @property

    def parentpost(self):

        for p in self.posts:

            if p.isparent == True:

                return p

Not the best method however its VERY interesting for static language veterans. We are searching an instance variable that we have not even defined! Remember only our SQLAlchemy mapper even makes mention of a “posts” variable through our relationship mapping, yet our “parentpost” property is searching through self.posts. Once it finds a parent post it returns true, this is not perfect and again not the ideal way to enforce a database constraint but for a demo it works fine.

Calling nosetests –-with-pylons=test.ini should now give you all passing tests. Running paster serve –-reload development.ini and opening http://localhost:5000 , adding a couple of threads and you should have something that looks like this:

Summary

We’ve completed a home page now that has some dynamic data, added a custom property on our database class, and addressed functional testing with models.

Congrats to the Los Techies MVPs!

bogardj — Thu, 02 Jul 2009 01:03:38 GMT

This morning, I and fellow Los Techies alumni Eric Hexter and Chris Patterson learned that we are MVPs! Well, technically I learned last night through an IM of a re-tweet of my MVP lead, but hey, close enough, right? I know it’s cheesy, but I think these awards are a testament to the great community Los Techies has built over the years, so for me, this is really an award for Los Techies. The roundup includes:

Jimmy Bogard, MVP ASP.NET
Eric Hexter, MVP ASP.NET
Chris Patterson, MVP C#

Personally, I’ve viewed my community involvement as simply a desire to share my journey with others, and in turn learn where others have been and are going. My favorite experiences thus far have been the open space conferences, book clubs, dojos and code camps, as it gives me a great chance to meet and talk with a lot of smart people. Thanks again everyone!

I'm a Microsoft C# MVP!

Chris Patterson — Thu, 02 Jul 2009 00:14:35 GMT

Opening my Inbox this morning resulted in a pleasant surprise from Microsoft. I have been presented with the 2009 Microsoft® MVP Award as a recognition for sharing my expertise in Visual C# with others. It is an honor to be recognized by Microsoft for doing something that I truly enjoy - building really cool software.

Microsoft Most Valuable Professionals (MVPs) are exceptional technical community leaders from around the world who are awarded for voluntarily sharing their high quality, real world expertise in offline and online technical communities. Microsoft MVPs are a highly select group of experts that represents the technical community's best and brightest, and they share a deep commitment to community and a willingness to help others.

A community-based award like this would not be possible without the support of my peers, and I look forward to meeting up with some of those very peers at the next MVP Summit! In the meantime, congratulations to all the new and renewed MVPs!

Updated TDD Productivity Plug-in for Resharper

erichexter — Wed, 01 Jul 2009 03:41:03 GMT

I first want to thank JetBrains for being pretty awesome. I have complained a lot about how they are constantly chaining their APIs to Resharper and as a result it makes keeping plugins very hard to maintain but they went way out of their way to help. I received and email from one of their developers offering to help on my plugin. Their was a change made in the latest version of resharp which made my plugin incompatible and their telemetry showed them a pattern with this problem. Just this week they made a commit to the source code and updated the project. I did not have the time to get into the internals of this change and I was really motivated by their help.

As a result, if you were using the plugin I recommend you download the latest version and install it!

Download it Here

For those of you who do not use it yet I will run down the features that are available.

1. Code forward, create a class in a referenced project.

When prompted with Quick Fixes for a non-existent class You get the following menu. This adds menu items to create the class in all referenced projects.( If this menu does not show up… you may need to add the project references to you unit test project).

After selecting a menu item. The class is created in the project you selected but the IDE stays in your test class. And you are prompted with the quick fix for adding the using for your classes namespace.

The class file is created in the corresponding project under the correct folder and namespace. It is that easy!

2. Move Class/Interface to referenced project

If you prefer working with your Class or Interface under test in the same file as your test class and move the class to your referenced project after you get your tests passing this feature will reduce the number of steps it takes to move the class to the referenced project. This eliminates the need to fumble around in the Solution Explorer window.

Opinionated Input Builders – Part 9 override the default Date Time picker

erichexter — Tue, 30 Jun 2009 13:07:23 GMT

Part 1 – Overview
Part 2 – the Label
Part 3 – the Source Code
Part 4 – the Partial View
Part 5 – the Required Field Indicator
Part 6 – the Performance
Part 7 – the Performance Take 2
Part 8 – the AutoForm
Part 9 – override the default Date Time Picker

I received a comment from Scott Hanselman about how would a better date time picker look using the opinionated input builders. I knew that this would be a complex problem just for the fact that there are currently very few good solutions to this problem now. While JQuery provides a great Date picker I am not very happy with the time picker. So here is a version of what this could look like. What I like about this approach is that it takes all the complexity including the multiple form elements and javascript and pushes it to a small partial that can be easily reused as well as it could be easily tested using QUnit.

The user interface I came up with is a combination of the Jquery UI datepicker and a set of dropdowns to select the time. I trimmed down the minute select box so that it only contains fifteen minute increments for this example.

In order to implement this I added a call to the Partial( ) method and passed in the name of my opinion for how a datetime should be rendered.

The next step was to add a partial control with the same name to my Shared view folder. This could have been placed in the Home folder if I only wanted to have this input available for that controller.

The code for the Partial view looks like the following. The view page is strongly typed to a DateTime model property. Than comes some jquery to pull it all together.

I rendered a hidden field, this field will be used to databind when being posted back to my Save Action. The other elements I appended some fixed names so that I can wire up an event that updates the hidden field when any of the values of the dropdowns or the date pickker text box changes. I also write a dynamice method named after the input field in order to reduce the client side code.

This is one approach to solve this problem, if you did not want to include this javascript and do the client side wire up of updating the hidden field this same work could be done in a Custom Model Binder that is wired up for DateTime objects that could look for fields with these names and than it could do the formatting. So there you go a few ways to tackle this problem.

How we do MVC – View models

bogardj — Tue, 30 Jun 2009 03:06:36 GMT

A while back, I went over a few of the patterns and opinions we’ve gravitated towards on our current large-ish ASP.NET MVC project, or, how we do MVC. Many of these opinions were forged the hard way, by doing the wrong thing many times until we found the “right” opinion. Of course, many of these opinions are only really valid in the constraints of our project. While the domain of this project isn’t important, here are some key aspects to consider:

AJAX is used very, very sparingly. Section 508 compliance is required
XHTML compliance is also required
XHTML DTD validation is also required
All (well, 99%) operations revolve a single uber-entity. Think customer relationship management, where everything you do deals with exactly one customer
Snippets of information repeated across many screens
Screens are either edit, or view, but never both. (99% never)

Given these constraints, these opinions may or may not apply to the project you work on. Again, patterns are all about tradeoffs, benefits and liabilities. But, opinionated software is like building a bullet train. It goes extremely fast, but only in the direction you build it.

That said, I’m going to go over some of the main aspects of our MVC usage in a series of posts – starting with ViewModels.

ViewModel design

For our application, the ViewModel is a central aspect of our MVC architecture. One of the first dilemmas facing MVC developers is to decide what the “M” in MVC means in ASP.NET MVC. In Rails, this is fairly clear, the M is ActiveRecord (by default). But in ASP.NET MVC, the “M” is silent! Its out-of-the-box architecture offers no guidelines nor advice on what the M should be. Should it be an entity? Data access object? DTO? Something else?

Sidenote – the term DTO is far overused. DTO is a Data Transfer Object. The pattern describes the usage, not the shape of a type. Just because an object is all properties and no methods does NOT mean it’s a DTO.

For us, the ViewModel is inextricably linked to Views, which leads us to our first rule:

Rule #1 – All Views are strongly-typed

I think I’ve gone over this one enough, as I really can’t stand magic strings and loose contracts. A dictionary as ViewModel is a very loose contract between the Controller and View. While on rare occasions we still need to pass information in the dictionary part, we’ve limited this to supporting information to help render some of the lower-level pieces of the View, and are used for some “plumbing” pieces, these pieces do not show up in our Controller action nor are they visible when you design the view.

Rule #2 – For each ViewModel type, there is defined exactly one strongly typed View

We’ll get into how we do this soon, but this rule has a lot of implications:

ViewModel types are distinct from our Domain Model types
The choice of what View to show can be decided strictly on the shape of your ViewModel
Re-used pieces in a View (through Partials) can be decided through re-using ViewModel types

On the first point, we never pass an Domain Model entity straight into the view. Most of the time, we only show a slice of information from a single entity. And many other times, the same snippet is shown in many places.

Rule #3 – The View dictates the design of the ViewModel. Only what is required to render a View is passed in with the ViewModel.

If a Customer object has fifty properties, but one component only shows their name, then we create a custom ViewModel type with only those two properties. Because we only have one ViewModel type per View, we shape our ViewModel around only what is displayed (or used) in that View. Why is this a Good Thing?

For one, just having a ViewModel points us to the right View. We need any other information other than your ViewModel type to pick the correct View. This also means we no longer need to concern ourselves with locating views by some arbitrary name, as the ViewModel is the View. Things like RenderPartial, which I have to select a name, become rather pointless at that point.

Rule #4 – The ViewModel contains only data and behavior related to the View

For the most part, our ViewModel contains only data. Most, if not all aggregations/calculations or shaping is done through our Domain Model. But occasionally, we have View-specific behavior or information, and that rightly belongs on our ViewModel.

We’ve looked at how we design our ViewModel and what it looks like, but how does it get there? If we create all these distinct ViewModel types separate from our Domain Model, didn’t we just create a bunch of work for ourselves? We thought so too, which is why we developed AutoMapper on this project.

Building the ViewModel

When we introduced AutoMapper into our MVC pipeline, we had a real problem. Do Controllers need to do the mapping between Domain Model and ViewModel in each action method? That becomes rather annoying for unit testing, as the mapping operation could warp things to a state that it becomes difficult to pull things back out. For example, or EditModels (ViewModels for forms) are very string-y, where DateTimes, Ints, Decimals etc are represented as strings. This comes from us using Castle Validators (future post, I promise) for validation.

So more moving parts, a dependency across all controllers? No, mapping in our Controller action just won’t do. Instead, we’ll use an Action Filter to do the work for us:

A request comes in, handled by an Action. The Action does its thing, ultimately deciding how to respond to the request. In many cases, this means rendering a view (ViewResult). From there, our Action Filter comes into play. On our Action method, we decorate it with an AutoMap attribute to configure the source/destination type pair to be mapped:

[AutoMap(typeof(Product), typeof(ShowProduct))]
public ActionResult Details(int id)
{
    var product = _productRepository.GetById(id);

    return View(product);
}

Very trivial, yes, but here we see that we still use the strongly-typed version of the View method, so that means that our model on the Action side, which I call the Presentation Model (feel free to pick a better name), is the strongly-typed ViewModel for the moment. The Presentation Model, which the Action creates, can be an entity, an aggregate root, or some other custom aggregate component that we build up.

From there, we decorated our action with a filter that specified we need to map from Product to ShowProduct. Why do we have to specify the source type? Well, many ORMs, including NHibernate, rely on proxy types for things like lazy loading. Instead of relying on the runtime type, we’ll explicitly specify our source type directly. This also helps us later in testing, as we can whip through all of our controller actions using reflection, and test to make sure the source/destination type specified is actually configured.

The filter attribute is very simple:

[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class AutoMapAttribute : ActionFilterAttribute
{
    private readonly Type _sourceType;
    private readonly Type _destType;

    public AutoMapAttribute(Type sourceType, Type destType)
    {
        _sourceType = sourceType;
        _destType = destType;
    }

    public override void OnActionExecuted(ActionExecutedContext filterContext)
    {
        var filter = new AutoMapFilter(SourceType, DestType);

        filter.OnActionExecuted(filterContext);
    }

    public Type SourceType
    {
        get { return _sourceType; }
    }

    public Type DestType
    {
        get { return _destType; }
    }
}

We simply capture the types and delegate to the real action filter for the work. This is again because I believe in separating metadata in attributes from the behavior they perform. Attributes just don’t work well for behavior. Instead, I’ll create a separate action filter:

public class AutoMapFilter : BaseActionFilter
{
    private readonly Type _sourceType;
    private readonly Type _destType;

    public AutoMapFilter(Type sourceType, Type destType)
    {
        _sourceType = sourceType;
        _destType = destType;
    }

    public override void OnActionExecuted(ActionExecutedContext filterContext)
    {
        var model = filterContext.Controller.ViewData.Model;

        object viewModel = Mapper.Map(model, _sourceType, _destType);

        filterContext.Controller.ViewData.Model = viewModel;
    }
}

The BaseActionFilter is just a class that implements the various filter methods as virtual members, so I can override just the ones I need to use. The AutoMapFilter pulls the original PresentationModel out of ViewData, performs the mapping operation, and puts the mapped ViewModel into the ViewData.Model property. From there, the strongly-typed view for our specified ViewModel type is rendered.

Because AutoMapper can flatten source types, we often find our ViewModel to simply follow a property chain for various pieces of information. Again, we let our View shape that piece. If we decide not to flatten, it’s usually because we’re creating a partial that re-uses a ViewModel type across other parent ViewModel types.

Wrapping it up

Designing ViewModels is quite ambiguous with MVC, as the shipped platform doesn’t offer any guidance or opinions in that area. But by forming rules around our ViewModel, we can create a path and direction for our innovation. Our rules are designed to strengthen the relationship between the View and the Model, with a concept of ViewModel – a Model designed exclusively for exactly one View.

In the next post, we’ll look at designing our views – for both viewing and editing data, and how we’ve crafted opinionated HTML builders to eliminate a lot of duplication and enforce standardization.

Python Web Framework Series – Pylons: Part 4 Introduction For Database Support With SQL Alchemy.

Ryan Svihla — Mon, 29 Jun 2009 02:54:00 GMT

We last left off with Views with Mako, now Pylons does not enforce on you an ORM at all, so you can use hand crafted SQL if you prefer. However, since I’ve done enough of that for a career or two we’re going to use my Python ORM of choice and the preferred one for Pylons SQLAlchemy.

Where does SQLAlchemy fit in as an ORM?

If you used NHibernate, you should feel pretty close to right at home with SQLAlchemy. If you come from an ActiveRecord, Entity Framework, Subsonic, or even Linq2Sql background this will be a bit more hand management than you are used to. If that is the case, I suggest Elixir (which I’ve heard great things about) or SQLObject (which I’ve used as well and works fine).

Mapping and Setup

There are three different ways to map tables to classes with SQLAlchemy, I will pick the most commonly used one, but will show the other two in a later post. In our PylonsForum project open model\__init__.py and change the file to look like so:

"""The application's model objects"""

from sqlalchemy import orm

import sqlalchemy as sa

import datetime

from pylonsforum.model import meta

def now():

    return datetime.datetime.now()

def init_model(engine):

    """Call me before using any of the tables or classes in the model"""

    meta.Session.configure(bind=engine)

    meta.engine = engine

class Post(object):

    pass

class Thread(object):

    pass

posts_table = sa.Table("posts", meta.metadata, 

		sa.Column("id", sa.types.Integer, primary_key=True),

		sa.Column("threadid", sa.types.Integer,sa.ForeignKey('threads.id')),

		sa.Column("content", sa.types.String(4000), nullable=False),

		sa.Column("author", sa.types.String(255), nullable=False),

		sa.Column("created", sa.types.TIMESTAMP(), default=now()),

		sa.Column("isparent" , sa.types.Boolean, nullable=True)

		)

threads_table = sa.Table("threads", meta.metadata,

		sa.Column("id", sa.types.Integer, primary_key = True),

		sa.Column("subject", sa.types.String(255))

		)

orm.mapper(Post, posts_table)

orm.mapper(Thread, threads_table,properties={'posts':orm.relation(Post, backref='thread')})

Not the best table structure and you’re welcome to improve this on your own but I wanted to create a default setup that was easy to read. Lets take a bit to recap the pieces:

def init_model(engine):

    """Call me before using any of the tables or classes in the model"""

    meta.Session.configure(bind=engine)

    meta.engine = engine

Straightforward method here sets up a Session object with the database engine passed into the method. Pylons will call init_model itself when the site is accessed.

class Post(object):

    pass

class Thread(object):

    pass

So a couple of empty classes? Python being a dynamic language can get away with this and just add the properties at runtime. These are the objects we’ll be interacting with when we want to store data.

posts_table = sa.Table("posts", meta.metadata, 

		sa.Column("id", sa.types.Integer, primary_key=True),

		sa.Column("threadid", sa.types.Integer,sa.ForeignKey('threads.id')),

		sa.Column("content", sa.types.String(4000), nullable=False),

		sa.Column("author", sa.types.String(255), nullable=False),

		sa.Column("dateadded", sa.types.TIMESTAMP(), default=now()),

		sa.Column("isparent" , sa.types.Boolean, nullable=True)

		)

threads_table = sa.Table("threads", meta.metadata,

		sa.Column("id", sa.types.Integer, primary_key = True),

		sa.Column("subject", sa.types.String(255))

		)

Ok these table declarations are providing the data definition logic, including some basic relationship, nothing too interesting here post in comments if you have specific questions.

orm.mapper(Post, posts_table)

orm.mapper(Thread, threads_table,properties={'posts':orm.relation(Post, backref='thread')})

Here the orm.mapper calls take the Page and Thread classes and map them with the table data definitions typed in earlier. You can also specify relationships here as we have done in the thread mapping, the properties argument is referencing the Post class and mapping it to a property called posts on the Thread class, while also mapping the other direction and putting thread on the Post class.

Finally run paster setup-app development.ini from the root pylonsforum directory and you should see a bunch of SQL flying by which indicates it has build the database schema for us:

Making Our New Thread Store In The Database

In the interest of space and time I’ll skip the testing story for another post.

Open up controllers\home.py .

Remove the Post class we created several posts ago
under the imports add import pylonsforum.model as model
under the imports add import pylonsforum.model.meta as meta
change the submitnewthread method to the following

Finally run the newthread action

http://localhost:5000/home/newthread

then create a thread

http://localhost:5000/home/submitnewthread

Should be no change in the actual outward appearance of from what we were doing before.

Summary and Recap

This was a very quick and basic introduction to SQLAlchemy and I will do more with it over the next couple of posts, but please add any comments to things I did not make clear. SQLAlchemy and ORM’s in general are a very large subjects and those of us that have used them for a long time tend to forget not all of this was so obvious when we started.

Real World Refactoring

Joshua Flanagan — Mon, 29 Jun 2009 00:35:29 GMT

If you’ve ever asked, or been asked, for an example to illustrate a software design principle, you know how frustrating it can be to work with a contrived example. There is rarely any depth to the example, and it lines up so neatly with the concept being explained that the student may have trouble recognizing similar situations in the wild. I’m going to try and relate a real world example of a recent refactoring effort, with the hope that the extra context and narrated walkthrough will help someone make a connection that has failed before.

A little background

My discussion revolves around Docu, which is an open source project for converting .NET XML comments into HTML documentation. (Note: I want to make clear is that this is not a criticism of any of the Docu code. All code is written with specific goals and constraints in mind. As the goals evolve, some designs that worked perfectly in early iterations may start to create friction, and deserve to be reconsidered.) .NET XML comments are constructed by applying a set of top-level tags to code elements such as classes, methods, or properties. Top-level tags are things like <summary/>, <remarks/>, or <param/>. Within the contents of the top-level tags, you can use embedded tags to provide additional contextual information. Embedded tags are things like <see/>, <para/>, or <paramref/>. The following example shows a top level <remarks/> tag, with embedded <paramref/> and <para/> tags:

///<remarks>Make sure the <paramref name=”maxValue” /> is a positive number.
///<para>Do not call more than once.</para></remarks>

In Docu, the job of translating the XML contents of a single top-level tag into a semantic model is handled by the CommentParser. Let’s take a look at the original implementation:

public class CommentParser : ICommentParser
{
    private readonly IDictionary<Func<XmlNode, bool>, Func<XmlNode, bool, bool, IComment>> parsers =
        new Dictionary<Func<XmlNode, bool>, Func<XmlNode, bool, bool, IComment>>();

    private readonly InlineTextCommentParser InlineText = new InlineTextCommentParser();
    private readonly InlineCodeCommentParser InlineCode = new InlineCodeCommentParser();
    private readonly MultilineCodeCommentParser MultilineCode = new MultilineCodeCommentParser();
    private readonly SeeCodeCommentParser See = new SeeCodeCommentParser();
    private readonly ParagraphCommentParser Paragraph;
    private readonly ParameterReferenceParser ParameterReference = new ParameterReferenceParser();
    private readonly InlineListCommentParser InlineList;

    public CommentParser()
    {
        Paragraph = new ParagraphCommentParser(this);
        InlineList = new InlineListCommentParser(this);
        parsers.Add(node => node is XmlText, InlineText.Parse);
        parsers.Add(node => node.Name == "c", InlineCode.Parse);
        parsers.Add(node => node.Name == "code", MultilineCode.Parse);
        parsers.Add(node => node.Name == "see", See.Parse);
        parsers.Add(node => node.Name == "para", Paragraph.Parse);
        parsers.Add(node => node.Name == "paramref", ParameterReference.Parse);
        parsers.Add(node => node.Name == "list", InlineList.Parse);
    }

    public IList<IComment> Parse(XmlNodeList nodes)
    {
        var blocks = new List<IComment>();

        int count = nodes.Count;
        for(int i = 0; i < count; i++)
        {
            XmlNode node = nodes;
            bool first = (i == 0);
            bool last = (i == (count - 1));

            foreach(var pair in parsers)
            {
                var isValid = pair.Key;
                var parser = pair.Value;

                if(!isValid(node))
                    continue;

                var block = parser(node, first, last);

                if(block != null)
                {
                    blocks.Add(block);
                    continue;
                }
            }
        }

        return blocks;
    }

You can see (lines 18-24 above) that in its constructor it builds up a collection of function (Func<>) pairs: one function that is used to identify a specific embedded tag, paired with another function that knows how to parse that tag into its model representation (an instance of IComment). The collection is used by the primary method in CommentParser, which iterates over all of the child nodes of a given chunk of XML, finds a parsing function that can handle that node, invokes the function, and collects the results.

The functions that perform the parsing of individual embedded tags are implemented in separate classes (I’ll call them “node parsers”). The CommentParser creates an instance of each node parser and stores it in a field where it can be referenced by the collection of function pairs. The benefit of moving the embedded tag parsing into separate classes is that they can be developed and tested independent of CommentParser. Unfortunately, you need to make a number of changes to CommentParser every time you add a new node parser. Since there are still quite a few embedded tags that are not yet recognized by Docu, and each embedded tag (generally) requires a new node parser, the CommentParser class will be continuously modified and unstable. The primary goal of my refactoring effort is to create a more stable CommentParser that is open for extension (support for new embedded tags can be added) but closed for modification.

High Cohesion

As stated above, the logic for determining when a specific parsing function should be applied was inside of CommentParser, while the logic to implement that function was in the individual parsing classes. These two pieces of logic are tightly related to each other. You cannot safely apply an arbitrary node parser to any node – it only makes sense to apply a parser to the type of node it was designed for. We can make the node parsers and the CommentParser more cohesive by moving both pieces of logic into the node parsing classes. I’ll add a CanParse(XmlNode) method to each node parser. The method returns true if the parser knows how to parse a given comment node. I implement them by copying the logic for identifying specific tags from the Func<> pairs in CommentParser.

Low Coupling

We still have the problem that the CommentParser creates and stores an instance of each specific node parser class (lines 6-11 above). This high coupling between the CommentParser and the node parsing classes makes it impossible to execute in isolation. You cannot use a CommentParser without bringing all of the functionality of all node parsers along. You also have to modify the CommentParser every time a new type of node parser is added (as when adding support for a new type of comment tag).

I’ll reduce coupling between CommentParser and individual node parsers by introducing an interface (ICommentNodeParser) to describe the functionality exposed by the node parsers. The CommentParser will only interact with this interface, which exposes the CanParse and Parse methods. I then change the node parsers so that they implement the new interface. Since they already had all of the needed functionality, it was just a matter of making sure the method signatures matched correctly.

Slight detour

When I started to add the interface to all of the node parsing classes, I noticed they all derived from CommentParserBase. However, none of the code in the application referred to these classes through this base class. That’s a pretty good indicator that inheritance is being used to share common functionality rather than for polymorphism. Using inheritance just to share some common functionality can lead to more inflexible designs and less cohesive classes. You are usually better off using composition instead of inheritance in these scenarios. In this case, the only shared functionality was a single helper method that had some special logic for string trimming. Since the method didn’t make use of any instance data, it was easy to move it to a separate class as an extension method on string. Now that CommentParserBase was empty, there was no reason to keep it around, so it was eliminated. The fact that no code broke when the base class was deleted (without using a refactoring tool) is a good indicator that it was the right decision.

Composition through dependency injection

At this point we have a bunch of node parsers that all implement a common interface. But the CommentParser is still tightly coupled to the various implementations because it has to create the instances. This is the perfect opportunity to use dependency injection to pass the node parser instances into the CommentParser. I just change the constructor of CommentParser to require an array of ICommentNodeParsers, and delete all the code that was creating the node parser instances. Whoever creates an instance of CommentParser will need to pass in the collection of node parser instances. CommentParser is no longer coupled to the specific node parsers and can be more easily used in isolation. You can see that the newer version is greatly simplified:

public class CommentParser : ICommentParser
   {
       private readonly ICommentNodeParser[] _parsers;

       public CommentParser(ICommentNodeParser[] parsers)
       {
           _parsers = parsers;
       }

       public IList<IComment> Parse(XmlNodeList nodes)
       {
           var blocks = new List<IComment>();

           var count = nodes.Count;
           for(var i = 0; i < count; i++)
           {
               var node = nodes;
               var first = (i == 0);
               var last = (i == (count - 1));

               var parser = _parsers.FirstOrDefault(p => p.CanParse(node));
               if (parser == null) continue;

               var block = parser.Parse(this, node, first, last);
               if (block != null)
               {
                   blocks.Add(block);
               }
           }

           return blocks;
       }

At this point you may be thinking I just “passed the buck” for creating the individual node parser instances to another class up the stack. Someone still needs to create them, and will therefore be tightly coupled to the implementations. That is true. However, if we can pass that responsibility up the stack far enough, it can be handled by code that has no other responsibility than to configure and bootstrap our application. That type of code doesn’t typically have any logic that would be re-used and doesn’t have the same concerns about designing for maintainability. Luckily, in the case of Docu, we already use StructureMap to compose our object instances so we get this functionality for free. I simply had to add a single line to the StructureMap configuration to tell it to make use of any implementation of ICommentNodeParser it finds in the Docu assembly.

public DefaultRegistry() {
  Scan(x =>  {
    x.AssemblyContainingType<DocumentationGenerator>();
    x.WithDefaultConventions();
    x.AddAllTypesOf<ICommentNodeParser>();

Now when an instance of CommentParser is requested from StructureMap, it will automatically have the array of all node parsers injected to its constructor.

Wrap up

So what did we gain? Let’s compare the stories for adding the ability to parse a new XML documentation tag. In the original implementation, we had to:

Create a class with code that parses the new tag
Create a new field in CommentParser to hold an instance of the new parsing class
Modify CommentParser’s constructor to register the new class, along with a predicate that determines when it should be used

And now we simply:

Create a class that implements ICommentNodeParser with code that recognizes and parses the new tag

Notice that you didn’t have to touch CommentParser? We just gave it new functionality without having to change (and potentially destabilize) the code. That’s the open-closed principle in action. The increased cohesion of the node parsers makes them easier to understand and helps localize any future changes to their implementation. We also saw that the use of dependency injection and a composition tool (StructureMap in this case) made it painless for us to pull code apart into separate classes. The code was made easier to maintain by applying a few established object oriented design principles. Hopefully this example helped clarify the application of these principles. Your feedback is appreciated.

For additional context, you can view the commit that contained the changes discussed above.

Walking through the creation of a complex installer package

Gabriel N. Schenker — Fri, 26 Jun 2009 12:05:21 GMT

Introduction

In this post I want to describe the issues and pain points I had when trying to generate a setup package for a complex product with many thousand of files. This is a post mainly for self documenting purposes but hopefully it might add some other people trying to create a complex product installer.

I received the following list of requirements

The install process must be transactional
Repair of a corrupt installation must be possible
Patching and Upgrading must be possible
Some files (mainly config files) must be backed up during an upgrade
Must be able to automatically launch an application at the end of the install
Must support localization
Can automatically create a virtual directory and an application pool for IIS
Can assign write permissions for specific folders to users/groups

In the past our company had used Install Shield to generate the setup packages. But this was overly complex and became a maintenance burden with the time. Not to mention that Install Shield is rather expensive compared to the two products discussed in this post which are both – ahhh – free! The products I want to have a look at in this post are WIX and NSIS.

WIX – Windows Installer XML

WIX is an open source project of Microsoft. It is based on the Microsoft Windows Installer technology. Everything about WIX (download and documentation) can be found here.

Tutorial

If you are new to WIX I absolutely recommend to read this wonderful and extensive tutorial.

“… Instead of a tool with a graphical interface that allows the developers to collect the files and other related tasks making up the installation process manually, it is much more like a programming language. Integrating perfectly with the usual process of creating applications, it uses a text file (based on the increasingly popular XML format) to describe all the elements of the installation process. The toolset has a compiler and a linker that will create the setup program just like our usual compiler creates our application from the source files. Therefore, WiX can be made part of any automated application build process very easily, be that based either on the classical technology of makefiles or the similar features of contemporary integrated development environments. …”

Analyze an existing MSI package with ORCA

If you ever want to analyze the .msi package created by WIX there is a tool called ORCA available in the Microsoft Windows SDK. With this tool any detail of such a package can be analyzed in details, e.g. the components included, the icon used or the install execution sequence.

Let’s now have a look at various elements of a setup project in WIX.

Defining Variables

If you are used to e.g. Nant or MSBuild you certainly define properties which later on you use in the script instead of hard coded values. WIX has a similar concept of variables. With the aid of pre-processor <?define …> statements we can define variables that later on can be used at the place of hard coded values

<?define BuildPath = ".\..\..\..\build" ?>

<?define Help = "$(var.BuildPath)\Help" ?>

<?define Skin = "$(var.Help)\Skin" ?>

<?define Data = "$(var.Help)\Data" ?>

Note that once a variable is defined it can be used by applying the following syntax: $(var.VariableName). It took me a while to find this all out since unfortunately it is not well documented in the WIKI.

Properties

To define a property and assign a (constant) value use a statement similar to this

<Property Id='NOTEPAD'>Notepad.exe</Property>

Note that when using an all caps id for the property the property is considered to be global and can be used every where in the setup files.

To define the value of a property with a nested statement like a registry search

<Property Id="IIS">

  <RegistrySearch Id="IISInstalledVersion"

                  Root="HKLM"

                  Key="SOFTWARE\Microsoft\InetStp"

                  Type="raw" Name="MajorVersion" />

</Property>

In the above snippet the property IIS is assigned the value of the registry key HKLM\SOFTWARE\Microsoft\InetStp\MajorVersion which is a DWORD in this case. If IIS 7.0 is installed the property IIS would contain the value “#7”.

Such a property can then e.g. be used in a condition. The following condition tests whether IIS 6.0 or 7.0 is installed on the current machine

<Condition Message="This setup requires IIS 6.0 or 7.0 is installed.">

  <![CDATA[IIS="#7" OR IIS="#6"]]>

</Condition>

Use environment variables to set properties, e.g.

<Property Id='ROOTDRIVE' Value='$(env.SystemDrive)' />

For a full list of all possible system directory values take a look here.

Components

The component is the atomic unit of things to be installed. It consists of resources—files, registry keys, shortcuts or anything else—that should always be installed as a single unit. Installing a component should never influence other components, removing one should never damage another component or leave any orphaned resource on the target machine. As a consequence, components cannot share files: the same file going to the same location must never be included in more than one component.

A component has to have its own Id identifier as well as its own GUID.

<Component Id='Component 1' Guid='170A4245-9DE3-4bd7-B68A-110000000010'>

...

</Component>

Once defined a component can be referenced by its id, e.g.

<ComponentRef Id='Component 1' />

Component Groups

A component group is a handy way to tie together a group of related components.

<ComponentGroup Id='Help'>

    <ComponentRef Id='Help'/>

    <ComponentRef Id='Help.Content'/>

    <ComponentRef Id='Help.Content.Administration'/>

...

</ComponentGroup>

later on in the setup package this group of components can be easily referenced, e.g.

<Feature Id='Core' Title='Core' Level='1'>

...

    <ComponentGroupRef Id='Help' />

...

</Feature>

Conditions

Often we need to define some conditions, e.g. launch conditions, conditional installation of some components, etc. For this purpose we have the <Condition Message=’…’>Inner text</Condition> construct. Here at the place of ‘Inner text’ one would define the condition (logic). If the condition is a launch condition and the evaluation of the condition logic result in false then the install process would be aborted and the text in the Message attribute would be reported. Typical launch conditions are

Request for sufficient privileges

<Condition Message="You need to be an administrator to install this product.">

  Privileged

</Condition>

a specific version of a framework (e.g. .NET 3.5) must be installed

<Condition Message="This application requires .NET Framework 3.5. Please install the .NET Framework then run this installer again.">

  <![CDATA[Installed OR NETFRAMEWORK35]]>

</Condition>

IIS 6.0 or above must be installed

<Condition Message="This setup requires IIS 6.0 or 7.0 is installed.">

  <![CDATA[IIS="#7" OR IIS="#6"]]>

</Condition>

For the above condition to work we have to define the property IIS since this is a custom property. We can do so with the following snippet

<Property Id="IIS">

  <RegistrySearch Id="IISInstalledVersion"

                  Root="HKLM"

                  Key="SOFTWARE\Microsoft\InetStp"

                  Type="raw" Name="MajorVersion" />

</Property>

here the value of the property IIS is set by the nested registry search which looks for a required key in the registry.

Permissions on folders

If we want to create a folder during setup which has special permissions we can use the following snippet inside a component

<CreateFolder Directory='index'>

  <!-- pay attention when using on non-english system! -->

  <Permission User="Everyone" GenericAll="yes" />

</CreateFolder>

Permissions and globalization/localization issues

Regarding localization issues have a look at this article.

The following snippet would cause problems on a computer having a say German Windows operating installed

<CreateFolder Directory='index'>

  <Permission User="Users" GenericAll="yes" />

</CreateFolder>

since the group ‘Users’ is called ‘Benutzer’ in German. Thus we need a more sophisticated way using custom actions and conditions. First we define a global(!) property with a default value

<Property Id="USERGROUP_USERS" Value="Users"/>

then we define a custom action to apply the localized name for the group ‘Users’

<CustomAction Id="SetUserGroup_Users_De"

              Property="USERGROUP_USERS"

              Value="Benutzer"

              Return="check" />

and then we execute the correct custom action based on the condition on which system language the operating system uses

<InstallUISequence>

    <Custom Action="SetUserGroup_Users_De" After="LaunchConditions">

        SystemLanguageID = "1031"

    </Custom>

    <Custom Action="SetUserGroup_Users_Fr" After="LaunchConditions">

        SystemLanguageID = "1036"

    </Custom>

...

</InstallUISequence>

finally the Permission tag uses our global property which now should contain the correct value

<CreateFolder Directory='index'>

  <Permission User="[USERGROUP_USERS]" GenericAll="yes" />

</CreateFolder>

Wow, this is a lot of work. Way too complicated in my opinion. If somebody know a more elegant way to do it, I am open for any suggestions!

Extensions

WIX provides some useful extension for common tasks. The namespace of an extension must be defined in the file where the respective functionality is used. If we want to use the IIS extensions then use the following definition

<Wix xmlns='http://schemas.microsoft.com/wix/2006/wi'

     xmlns:iis="http://schemas.microsoft.com/wix/IIsExtension">

...

</Wix>

Now the commands provided by the extensions can be used like this

<!-- Create an application pool on IIS -->

<iis:WebAppPool Id='TestAppPool' Name='Test Application Pool' Identity='networkService'/>

which creates a new application pool for IIS during installation.

Define an application pool for IIS

On task of our setup package is to create a new application pool for IIS during installation. It comes handy that WIX provides an extension providing IIS related tasks which among other tasks allows us to just do that. First we have to define the namespace for the extension

<Wix xmlns='http://schemas.microsoft.com/wix/2006/wi'

     xmlns:iis="http://schemas.microsoft.com/wix/IIsExtension">

then we can use all available commands. In our case this is the one we need

<iis:WebAppPool Id='SampleAppPool' Name='Sample Application Pool' Identity='networkService'/>

The above command creates a new application pool and assigns it the network service identity. Note that the command has to be a child of a component.

Create a virtual directory for IIS

Not only do we have to create a new application pool but also we need a new virtual directory for IIS. With the same extension we can use this fragment to do just that

<iis:WebVirtualDir Id="MyWebApp" Alias="MyWebApp"

                   Directory="INSTALLDIR"

                   WebSite="DefaultWebSite">

  <iis:WebApplication Id="SampleWebApplication" Name="Sample" WebAppPool="SampleAppPool" />

</iis:WebVirtualDir>

The above command creates a new virtual directory for the physical folder [INSTALLDIR] and calls it MyWebApp. The new virtual directory is defined for the DefaultWebSite. It then creates an Web application and assigns this application to the application pool created previously. Again this command has to be a child of a component.

Now finally we have to define what is our DefaultWebSite. This is done with the following command

<iis:WebSite Id='DefaultWebSite' Description='Default Web Site' Directory='TARGETDIR'>

  <iis:WebAddress Id="AllUnassigned" Port="80" />

</iis:WebSite>

Note that the above command usually should not be put inside a Component otherwise if put it inside the Component, a new Web Site will be created and uninstall will remove it.

Modularize the setup

When building a complex setup package it is advisable to modularize it. SRP anybody…? We have modules, includes and fragments for this purpose. There will be one main .wxs file and several satellite .wxs or .wxi files.

The main installation file

We have to have one main wxs file which defines our overall setup package. The schema of this file contains the following main parts

   1: <?xml version="1.0" encoding="utf-8"?>

   2: <Wix xmlns='http://schemas.microsoft.com/wix/2006/wi'>

3:

   4:   <Product Id='170A4245-9DE3-4bd7-B68A-100000000000'

   5:            UpgradeCode='170A4245-9DE3-4bd7-B68A-200000000000' ...>

6:

   7:     <Package .../>

8:

   9:     <!-- Define the directory structure on target computer -->

  10:     <Directory Id='TARGETDIR' Name='SourceDir'>

  11:       <Directory Id='ProgramFilesFolder' >

  12:         <Directory Id='INSTALLDIR' Name='TOPAZ TE6'>

  13:             <!-- define more (sub-) directories if needed -->

  14:             ...

  15:         </Directory>

  16:       </Directory>

  17:     </Directory>

18:

  19:     <!-- Define the components and component groups -->

  20:     <DirectoryRef Id="INSTALLDIR">

  21:       <Component Id='Component 1' Guid='170A4245-9DE3-4bd7-B68A-110000000010'>

  22:         ...

  23:       </Component>

  24:     </DirectoryRef>

25:

  26:     <!-- Define the list of features available to install -->

  27:     <Feature Id='Complete' Level='1'>

28:

  29:       <Feature Id='Feature 1' Title='Feature 1' Level='1'>

  30:         <ComponentGroupRef Id='Group 1'/>

  31:             ...

  32:         <ComponentRef Id='Component 1'/>

  33:         <MergeRef Id='Merge Module 1'/>

  34:       </Feature>

35:

  36:       <Feature Id='Feature 2' Title='Feature 2' Level='1000'>

  37:         ...

  38:       </Feature>

39:

  40:     </Feature>

  41:   </Product>

  42: </Wix>

The root node on line 2 is <Wix> and we have to define at least the root namespace “http://schemas.microsoft.com/wix/2006/wi”.

The wix node has a single child node <Product> which defines such things as the name of the product, the manufacturer, the version, etc. Very important are also the Id and the UpgradeCode of the product tag which uniquely define an installed product. Both of them are GUIDs. The UpgradeCode is very important if you plan to patch or upgrade an installation in the future since (to work properly) the patch- or upgrade package will have to reference this UpgradeCode.

On line 7 we define the (setup-) package in more details. There we define the required version of the Microsoft Windows installer, whether or not the package is compressed, the manufacturer and other things.

Starting from line 10 we have the definition of the directory structure we want to create on the target computer.

Starting from line 20 we define the components that shall be included in our setup package.

On line 27 and following we define the list of features this setup package provides. This list is visible to the user during the setup and he can enable or disable certain features. Normally during a setup the user is presented with 3 choices

The Level attribute of a feature determines whether a feature is included in the typical or only in the complete setup. In our sample Level=’1’ means that the feature is included in the typical setup whilst Level=’1000’ means it is not included.

Include Files

The content found in an Include file is just imported and placed at the position where you put an <?include …> pre-processor directive, e.g.

<?include MyIncludeFile.wsi?>

A include file usually has the extension .wsi. As an example I have defined an include file which contains (some of) the launch conditions of our setup

<?xml version='1.0'?>

<Include xmlns='http://schemas.microsoft.com/wix/2006/wi'

     xmlns:iis="http://schemas.microsoft.com/wix/IIsExtension">

  <!-- check whether .NET 3.5 SP1 is installed -->

  <PropertyRef Id="NETFRAMEWORK35"/>

  <PropertyRef Id="NETFRAMEWORK35_SP_LEVEL"/>

  <Condition Message="This application requires .NET Framework 3.5. Please install the .NET Framework then run this installer again.">

    <![CDATA[Installed OR NETFRAMEWORK35]]>

  </Condition>

  <Condition Message="This application requires .NET Framework 3.5 SP1. Please install the .NET Framework then run this installer again.">

    <![CDATA[Installed OR (NETFRAMEWORK35_SP_LEVEL and NOT NETFRAMEWORK35_SP_LEVEL = "#0")]]>

  </Condition>

  <Condition Message="You need to be an administrator to install this product.">

    Privileged

  </Condition>

...

<Include>

Please note that the root tag of an include file has to be <Include> and not <Wix>

Fragment

Useful if you want to define a component or component group that later on is referenced in the main .wxs file. As an example I take the component responsible to install the online help for the product

<?xml version='1.0'?>

<?define BuildPath = ".\..\..\..\build" ?>

<?define Help = "$(var.BuildPath)\Help" ?>

<?define Skin = "$(var.Help)\Skin" ?>

<?define Data = "$(var.Help)\Data" ?>

<Wix xmlns='http://schemas.microsoft.com/wix/2006/wi'>

  <Fragment Id='FragmentHelp'>

    <ComponentGroup Id='Help'>

      <ComponentRef Id='Help'/>

      <ComponentRef Id='Help.Skin'/>

      <ComponentRef Id='Help.Data'/>

    </ComponentGroup>

    <DirectoryRef Id="Help">

      <Component Id="Help" Guid="170A4245-9DE3-4bd7-B68A-110000000003">

        <File  Id='Reviews.pdf' Name='Reviews.pdf' Source='$(var.Help)\Reviews.pdf' DiskId='1'/>

      </Component>

      <Directory Id='Help.Skin' Name='Skin'>

        <Component Id="Help.Skin" Guid="170A4245-9DE3-4bd7-B68A-110000000013">

          <File  Id='Blank.htm' Name='Blank.htm' Source='$(var.Skin)\Blank.htm' DiskId='1' />

          <File  Id='BrowseSequences.htm' Name='BrowseSequences.htm' Source='$(var.Skin)\BrowseSequences.htm' DiskId='1' />

          <File  Id='Favorites.htm' Name='Favorites.htm' Source='$(var.Skin)\Favorites.htm' DiskId='1' />

          <File  Id='Index.htm' Name='Index.htm' Source='$(var.Skin)\Index.htm' DiskId='1' />

...

        </Component>

      </Directory>

      <Directory Id='Help.Data' Name='Data'>

        <Component Id="Help.Data" Guid="170A4245-9DE3-4bd7-B68A-110000000023">

          <File  Id='Alias.xml' Name='Alias.xml' Source='$(var.Data)\Alias.xml' DiskId='1' />

...

        </Component>

      </Directory>

    </DirectoryRef>

  </Fragment>

</Wix>

At the top I define a component group which in turn is then referenced in the main .wxs file by its it (here ‘Help’). Below I define the components with their content. In this case I want to install some files in a Help subfolder of the installation folder. The help subfolder in turn has two child folders Skin and Data which in turn will contain other files. The whole definition is wrapped by a Fragment tag.

Merge Module

Merge modules are used to deliver shared code, files, resources, registry entries, and setup logic to applications as a single compound file.However, a merge module cannot be installed alone, it must be merged into an installation package. A very basic module definition is given below

<?xml version='1.0'?>

<Wix xmlns='http://schemas.microsoft.com/wix/2006/wi'>

   <Module Id='TestModule' Language='1033' Version='1.0.0.0'>

      <Package Id='170A4245-9DE3-4bd7-B68A-100000000000'

               Description='Basic Merge Module'

               Comments='A basic Windows Installer Merge Module'

               Manufacturer='TOPAZ Techonlogies'

               InstallerVersion='200' />

      <Directory Id='TARGETDIR' Name='SourceDir'>

         <Directory Id='MyModuleDirectory' Name='.'>

            <Component Id='MyModuleComponent' Guid='170A4245-9DE3-4bd7-B68A-100000000001'>

               <File Id='readme2' Name='readme2.txt' Source='readme2.txt' />

            </Component>

         </Directory>

      </Directory>

   </Module>

</Wix>

The above merge module just installs a single file ‘readme2.txt’ into the target directory. The main difference between the wxs file of an normal install and the wxs file of a merge module is that the former has a <Product> tag whereas the latter has a <Module> tag.

Standard Actions

There are many standard actions available but not scheduled by default. ScheduleReboot, for instance, will instruct the user to reboot after the installation:

<InstallExecuteSequence>

  <ScheduleReboot After='InstallFinalize' />

</InstallExecuteSequence>

If the need to reboot depends on a condition (for instance, the operating system the installer is running on), we can use a condition:

<InstallExecuteSequence>

  <ScheduleReboot After='InstallFinalize'>VersionNT = 400</ScheduleReboot>

</InstallExecuteSequence>

A reboot will be required if the OS is Windows NT 4.0.

Custom Actions

It's not only these so-called standard actions that you can schedule and re-schedule. There are a couple of custom actions as well (custom here means that they don't appear in the standard course of events but we can use them wherever and whenever we like). A very common need is to launch the application we have just installed or to show a readme.txt file in Notepad.exe at the end of the installation? To do the latter we just add the following code to the .wxs file

<!-- show the readme.txt file at the end of the installation -->

<Property Id='NOTEPAD'>Notepad.exe</Property>

<CustomAction Id='LaunchFile' Property='NOTEPAD' ExeCommand='[SourceDir]readme.txt' Return='asyncNoWait' />

<InstallExecuteSequence>

  <Custom Action='LaunchFile' After='InstallFinalize'>NOT Installed</Custom>

</InstallExecuteSequence>

The “NOT Installed” condition means that the action is only executed if the product has not already been installed at a previous time.

Features

When installing an application we usually are presented with a list of features we can install. Some of them are mandatory and some of them are optional. The list

<Feature Id='Complete'

         Title='TOPAZ Enterprise 6.0'

         Description='The complete package'

         Display='expand'

         ConfigurableDirectory='INSTALLDIR'

         Level='1'>

...

</Feature>

Please note the attribute ConfigurableDirectory which defines the directory that can be configured by the user during setup. In our case this would be the INSTALLDIR which by default points to the sub-folder “TOPAZ TE6” of the program files folder (refer to chapter: the main installation file).

The attribute Level=’1’ indicates that this feature is included in the typical installation.

The features can be (hierarchically) nested. That is each feature can have as many child features as needed. Inside a feature tag we have to reference the components, component groups or merge modules which pertain to the said feature. E.g.

<Feature Id='Core' Title='Core' Level='1'>

  <ComponentRef Id='Backup' />

...

  <ComponentGroupRef Id='Help' />

...

  <MergeRef Id='MyModule' />

</Feature>

In the above case the feature ‘Core’ would contain the component Backup, the component group Help and the merge module MyModule amongst others.

Compile and link the installation package

The creation of an .msi package is a two step process. First we have to compile the .wxs and .wxi files and then link them. The compiler is called candle.exe and the linker light.exe.

Assuming we only have one .wsx file called Product.wxs and we use functionality of the two extensions IISExtension and WixUtilExtension the creation would be

candle.exe Product.wxs -ext IISExtension -ext WixUtilExtension

light.exe Product.wixobj -ext IISExtension -ext WixUtilExtension

which results in a Product.msi file.

If our Product.wxs file references components or component groups defined in another fragment file (e.g. called FramgmentHelp.wxs) then our calls have to be modified as follows

candle.exe Product.wxs FragmentHelp.wxs -ext IISExtension -ext WixUtilExtension

light.exe Product.wixobj FragmentHelp.wixobj –out Product.msi -ext IISExtension -ext WixUtilExtension

Please note the “–out Product.msi” part of the second command. Since we have more than one input file for the linker we have to define the name for the output file.

If we have many include- and fragment files and/or merge modules to include in our compile and link process then we can also define a (text) file from where the candle.exe and the light.exe take their input and thus our commands will be

candle.exe @candleInput.txt

light.exe @lightInput.txt

Dealing with many files

Our setup package contains many hundred files. It would be a tedious task to add them all manually to the respective wxs files. Unfortunately it is not possible to use file names with wildcards (IMHO this is one of the major disadvantages of WIX compared to NSIS). Each file has to be explicitly added to the setup. Thus I decided to write a small application that automatically creates those entries for me.

Starting from a given base directory the program loops through all files and recursively through all sub-folders and creates the appropriate XML fragments. Having a directory structure like this

ParentFolder

    File1

    File2

    ChildFolder1

        File3

        GrandChildFolder1

            File4

            File5

    ChildFolder2

        File6

the application creates an XML fragment like this

<ComponentGroup Id='ParentFolder'>

  <ComponentRef Id='ParentFolder'/>

  <ComponentRef Id='ParentFolder.ChildFolder1'/>

  <ComponentRef Id='ParentFolder.ChildFolder1.GrandChildFolder1'/>

  <ComponentRef Id='ParentFolder.ChildFolder2'/>

</ComponentGroup>

<DirectoryRef Id='ParentFolder'>

  <Component Id='ParentFolder' Guid='...'>

    <File Id='...' Name='File1' Source='$(var.ParentFolder)\File1' DiskId='1' />

    <File Id='...' Name='File2' Source='$(var.ParentFolder)\File2' DiskId='1' />

  </Component>

  <Directory Id='ParentFolder.ChildFolder1' Name='ChildFolder1'>

    <Component Id='ParentFolder.ChildFolder1' Guid='...'>

      <File Id='...' Name='File3' Source='$(var.ParentFolder)\ChildFolder1\File3' DiskId='1' />

    </Component>

    <Directory Id='ParentFolder.ChildFolder1.GrandChildFolder1' Name='GrandChildFolder1'>

      <Component Id='ParentFolder.ChildFolder1.GrandChildFolder1' Guid='...'>

        <File Id='...' Name='File4' Source='$(var.ParentFolder)\ChildFolder1\GrandChildFolder1\File4' DiskId='1' />

        <File Id='...' Name='File5' Source='$(var.ParentFolder)\ChildFolder1\GrandChildFolder1\File5' DiskId='1' />

      </Component>

    </Directory>

  </Directory>

  <Directory Id='ParentFolder.ChildFolder2' Name='ChildFolder2'>

    <Component Id='ParentFolder.ChildFolder2' Guid='...'>

      <File Id='...' Name='File6' Source='$(var.ParentFolder)\ChildFolder2\File6' DiskId='1' />

    </Component>

  </Directory>

</DirectoryRef>

where at the place of the ‘…’ the application puts a GUID. The code is really simple; it’s just a recursive walk through the directory structure and its files.

In the main wxs file I then just reference the component group e.g.

<ComponentGroupRef Id='ParentFolder' />

Nullsoft Scriptable Install System – NSIS

I will touch this framework only briefly since in the end we decided to take WIX for our purposes. Everything about NSIS (download and documentation) can be found here.

“… NSIS is a professional open source system to create Windows installers. It is designed to be as small and flexible as possible and is therefore very suitable for internet distribution. …

NSIS is script-based and allows you to create the logic to handle even the most complex installation tasks. Many plug-ins and scripts are already available: you can create web installers, communicate with Windows and other software components, install or update shared components and more. …”

NSIS is NOT based on the Microsoft Windows Installer technology! Thus no msi files are produced but rather exe files. My first impression of the product was very good. I especially liked the speed of an installation. Even complex installations take a fraction of the time a MS Windows Installer based installation needs.

The Installer Script File

As said above NSIS is a script base install system. Our installation script we have to place in a file with the extension nsi.

Such a file can be compiled to a setup package with this simple command line

makensis.exe TOPAZ.nsi

where TOPAZ.nsi contains our installation scripts. The compilation is very quick. The creation of a setup package can thus be very easily integrated in an automated build.

Define Variables

We can define variables for further usage as follows

!define COMPANY "TOPAZ Technologies"

!define COMPANYLEGAL "Ldt"

!define URL "http://www.topazti.com"

!define PRODUCT "TE6"

we can also conditionally define a variable depending on the fact whether or not the variable has already been defined e.g. as a command line parameter when calling the NSIS make

!ifndef VERSION

  !define VERSION "0.1.0"

!endif

later on a variable can be referenced by using the ${VariableName} syntax, e.g.

;Set default installation folder

InstallDir "$PROGRAMFILES\${COMPANY}\${PRODUCT}"

This is very similar to what we are used from tools like Nant.

Predefined variables are accessed with the syntax $VariableName; that is no curly braces are needed. An example would be

SetOutPath "$INSTDIR"

which sets the output directory to the content of the pre-defined variable INSTDIR.

Sections

A nsi file consists of several sections. At least two of them are needed; one to define the install actions and one to define the un-install actions.

Each section contains zero or more instructions.
Sections are executed in order by the resulting installer, and if ComponentText is set, the user will have the option of disabling/enabling each visible section.
If a section's name is 'Uninstall' or is prefixed with 'un.', it's an uninstaller section.

Section "Core" SEC_CORE

...

    SetOutPath "$INSTDIR"

...

SectionEnd

Adding Files to the Setup Package

To add a single file to the setup package use a command similar to this

File readme.txt

To add a whole tree of files (while maintaining the hierarchy) I use a command similar to this

File /r /x .svn /x index /x log ..\..\build\*.*

The above command starts from a base directory (here a relative path ‘..\..\build\*.*’) and traverses the whole sub-tree and adds all files found including their relative folders. By using the ‘/x folderName’ command line parameter I can exclude certain files or folders. Compared to how much effort is needed with WIX to accomplish the same thing is is really a “killer feature”.

Creating Directories on the target machine

to conditionally create a directory on the target machine I use the following syntax

IfFileExists $INSTDIR\index +2 0

  CreateDirectory $INSTDIR\index

User Interface

There are several different UI available for the installer; standard, modern UI and ultra modern UI. In the sample we will use the modern UI. To use this UI we have to include the corresponding library with this command

!include "MUI2.nsh"

later in the script file we have e.g.

!insertmacro MUI_PAGE_WELCOME

!insertmacro MUI_PAGE_LICENSE ".\Eula.rtf"

!insertmacro MUI_PAGE_COMPONENTS

!insertmacro MUI_PAGE_DIRECTORY

!insertmacro MUI_PAGE_INSTFILES

!define MUI_FINISHPAGE_NOAUTOCLOSE

!define MUI_FINISHPAGE_SHOWREADME "$INSTDIR\readme.txt"

!insertmacro MUI_PAGE_FINISH

!insertmacro MUI_UNPAGE_WELCOME

!insertmacro MUI_UNPAGE_CONFIRM

!insertmacro MUI_UNPAGE_INSTFILES

!define MUI_UNFINISHPAGE_NOAUTOCLOSE

!insertmacro MUI_UNPAGE_FINISH

The above fragment defines that we want to have a set of pages defined in the modern UI library included during our setup and during the uninstall of the product. Of course we should not forget to provide resources of the desired language used by the installer. This can be done by adding this statement

!insertmacro MUI_LANGUAGE "English"

Functions

We can define functions in our installer file. As an example I provide the function used to detect whether IIS is installed and whether it is at least IIS 6.0.

Function CheckIISVersion

    ClearErrors

    ReadRegDWORD $0 HKLM "SOFTWARE\Microsoft\InetStp" "MajorVersion"

    ReadRegDWORD $1 HKLM "SOFTWARE\Microsoft\InetStp" "MinorVersion"

    IfErrors 0 NoAbort

    Abort "Setup could not detect Microsoft Internet Information Server v5 or later; this is required for installation. Setup will abort."

    IntCmp $0 6 NoAbort IISMajVerLT6 NoAbort

    NoAbort:

        DetailPrint "Found Microsoft Internet Information Server v$0.$1"

        Goto ExitFunction

    IISMajVerLT6:

        Abort "Setup could not detect Microsoft Internet Information Server v6 or later; this is required for installation. Setup will abort."

    ExitFunction:

FunctionEnd

The script language is simple but powerful. It remembers a little bit the old days we still used to develop in assembler.

A function like the above one can then be called from a section of the installation file

Section "Install Virtual Directory" SEC_IIS

    Call CheckIISVersion

    Call ...

...

SectionEnd

Register Product and prepare Uninstaller

NSIS does not automatically register an installed product on the system such as that it is visible in the list of installed products. We have to manually do this. I use the following section to just accomplish this task. The section creates some registry keys and finally creates an uninstaller exe in the install directory.

Section "-Common Items" SEC_COM

  ;Store installation folder

  WriteRegStr HKCU "Software\${COMPANY}\${PRODUCT}" "" $INSTDIR

  ;Add uninstall information to Add/Remove Programs

  WriteRegStr HKLM "Software\Microsoft\Windows\CurrentVersion\Uninstall\${COMPANY} ${PRODUCT}" \

                 "DisplayName" "${COMPANY} ${PRODUCT}"

  WriteRegStr HKLM "Software\Microsoft\Windows\CurrentVersion\Uninstall\${COMPANY} ${PRODUCT}" \

                 "URLInfoAbout" "${URL}"

  WriteRegStr HKLM "Software\Microsoft\Windows\CurrentVersion\Uninstall\${COMPANY} ${PRODUCT}" \

                 "Publisher" "${COMPANY} ${COMPANYLEGAL}"

  WriteRegStr HKLM "Software\Microsoft\Windows\CurrentVersion\Uninstall\${COMPANY} ${PRODUCT}" \

                 "DisplayVersion" "${VERSION}"

  WriteRegStr HKLM "Software\Microsoft\Windows\CurrentVersion\Uninstall\${COMPANY} ${PRODUCT}" \

                 "UninstallString" "$\"$INSTDIR\uninstall.exe$\""

  ;Create uninstaller

  WriteUninstaller "$INSTDIR\Uninstall.exe"

SectionEnd

In my opinion this puts an unnecessary burden onto the shoulders of the developer since every single installed product has to be registered. Thus I would expect that this happens automatically. Also there is no such thing as an automatic repair functionality. Compared to WIX we have a clear disadvantage here.

Summary

In this post I have shown in detail the various steps needed to create a setup package with WIX for a product consisting of many hundreds of files. The setup package also assigns write access for certain folders to specific users and/or groups. It creates a virtual directory for the application and defines a new application pool to which the virtual directory is assigned. Furthermore the setup creates a backup of pre-existing configuration files when upgrading a previous version of the product. The whole setup is fully transactional and can be changed or repaired at a later time. The product is ready for patches or upgrades if needed. I have also shown how the setup can be localized.

We use WIX to create our setup packages. WIX is free and open source. Some tasks could be simpler or more streamlined (like e.g. the usage of properties) and the documentation could be more detailed. Once all missing pieces of information are collected the creation of a setup package is straight forward and easy to do.

IMHO comparing NSIS with WIX is similar to comparing C++ (no .NET) with C# on .NET. With NSIS you have much more fine grain control of the installation process than with WIX. On the other hand you have to invest more time in providing such useful features as transactional install and repair functionality as well as fail-safe de-installation. This was the main reason that we finally decided to take WIX as our tool for the creation of setup packages. Still I think NSIS is a very powerful and mature product with lightening speed.

How To? Highly Complex Query Generating Based On Security Needs

derick.bailey — Thu, 25 Jun 2009 20:26:29 GMT

I have the following object model:

An Office belongs to one Office Group. An Office also belongs to one Office Region. There is no relationship between Office Group and Office Region. They are two separate groupings for various reasons. A Counselor belongs to one Office. A Veteran is assigned to one Office.

I need to implement the following security rules for a Counselor looking at Veterans, starting with “deny access to all Veteran records, unless any of the following are true”:

If a Counselor is an Administrator (a role of the Counselor), that Counselor can see all Veteran records
A Counselor can see any Veteran that has no current Office assignment
A Counselor can see Veterans that are assigned to an Office within the Counselor’s Office Group
If a Counselor belongs to a “Regional Office”, that Counselor can see Veterans that are assigned to that Region
… a few others in similar fashion …

I also need to implement the following around the above rules:

When a Counselor searches for Veterans, they are only allowed to see search results that meet the above criteria
When a Counselor views the detail of a Veteran, they are only allowed to see the the Veteran if the above criteria is met. This includes, if a Counselor tries to manually load a Veteran (manually type in url “id=1234”)

… and the Counselor is not allowed to see the Veteran, show a “Not Authorized” message
… and the Vet does not exist, show a “Vet Does Not Exist” message

so…

how do you create a search process around a security need like this? How do you go about creating a very complex rule system to determine which Veterans the Counselor can see? In my ideal world, I’d have some way of checking this security need against the search results and against an individual veteran record, without duplicating the code or logic for that check anywhere… is this a realistic expectation? What “standard” security implementations are out there, that would help me resolve my needs?

This is being done in C# in an ASP.NET Webforms app, using NHibernate for our ORM. I don’t expect NHibernate to be able to solve this problem for us, but if it can, that’s even better. And, I do expect to be able to write some sort of Unit or Integration tests around whatever the process is.

Any advice or help is appreciated.

Python Web Framework Series – Pylons: Part 3 Views with Mako

Ryan Svihla — Thu, 25 Jun 2009 04:53:00 GMT

This is a huge post and I should have split this into several smaller ones so please bear with me while I get my series format tweaked. We last left off with Controllers, Views and Testing with a basic test, basic view and basic controller. Now with our basic scaffold built we can focus on making our views reusable, dynamic, and more pleasant to look at.

First things first

First before we dive too deeply into Mako, lets setup a default home page by doing the following:

delete the index.html in the pylonsforum\public\ folder.
in pylonsforum\config\routing.py add the following line somewhere under the line “# CUSTOM ROUTES HERE” but before “return map”.
1. map.connect("home", "/", controller="login", action="index")
assuming you have your Pylons command we used earlier paster server –-reload development.ini then open your browser to http://127.0.0.1:5000 and you should see the view we created in the last lesson.

Basics

Mako uses standard python mixed with HTML to display views. This gives you both a great deal of flexibility and a great deal of ability to hang yourself. So be warned, avoid putting to much logic into your view, since you can do about anything there.

Open up templates\index.mako replace the table rows in the table body with the following:

<tbody>       
%for p in c.posts:       
<tr><td>${p.author}</td><td>${p.subject}</td><td>${p.dateadded}</td></tr>       
%endfor  
</tbody>       

for good measure in the beginning of the body before the “recent posts” div place the following

<body>       
username: ${c.username}                 
<div id="recentposts" style="float: right">       

Looking at this, for the tables we’ve done a for loop over some variable called c.posts. In the second we’re accessing another variable c.username. “c” is shorthand for Template Context, similar to ViewData in Asp.Net MVC and PropertyBag in Monorail, except we’re not accessing a string dictionary.

Add the following tests in “pylonsforum\tests\functional\test_home.py” for what we need to add to the controller:

    def test_username(self):       
        response = self.app.get(url(controller='home', action='index'))       
        assert "username: rsvihla" in response       
      
    def test_recent_posts(self):       
        response = self.app.get(url(controller='home', action='index'))       
        assert "<tr><td>jkruse</td><td>New Kindle</td><td>06/24/2009</td></tr>" in response       

running nosetests –-with-pylons=test.init should give you two assertion failures.

now change “pylonsforum\controllers\home.py” to look like so:

import logging       
      
from pylons import request, response, session, tmpl_context as c       
from pylons.controllers.util import abort, redirect_to       
      
from pylonsforum.lib.base import BaseController, render       
      
log = logging.getLogger(__name__)       
      
class Post(object):       
      
    def __init__(self, author, subject, dateadded):       
        self.author = author       
        self.subject = subject       
        self.dateadded = dateadded       
      
class HomeController(BaseController):       
      
    def index(self):       
        c.username = "rsvihla"       
        c.posts = [Post("jkruse", "New Kindle", "06/24/2009")]       
        return render('index.mako')       

We’ve added a Post class with basic attributes and placed them in an array in the c.posts variable, also we’ve hardcoded the username “rsvihla”. I know the more experience developers here are cringing at my awful little Post class, don’t worry its a just a place holder and will be removed later. The point here is building functionality in steps with test coverage. Now run nosetests just as before and you should have all tests passing. For bonus measure refresh http://127.0.0.1:5000 .

Base Layouts

We’ve built a very basic page now, but suppose we want to build several and have some bits of information show up over and over again, like user name or menu structure.

create a base template named “base.mako” in the templates directory that has the following in it:

<%namespace name="user" module="pylonsforum.model.users" />

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html>

<head>

<title>${self.title()}</title>

${self.head_tags()}

<style type="text/css">

#header{ 

	height: 70px;

	width: 100%;

	border-width: 0.5px;

	border-style: solid;

	font-size:30px;

	text-align: center;

	background-color: #9B2C25;

}

#sidebar{

	background-color: #C1AD72;

	border-width: 0.5px;

	border-style: dotted;

	width: 150px;

	height: 600px;

	float: right;

	font-size: 15px;

	padding-top:10px;

	padding-left:5px;

}

#content{

	margin-top:10px;

}

body {

background-color: #DACEAB;

}

</style>

</head>

<body>

<div id="header">Pylons Forum</div>

<div id="sidebar">

username: ${user.get_current_user()}

</div>

	<div id="content" >

	${self.body()}

	</div>

</div>

</body>

</html>

Most everything is standard HTML so lets restrict this to the interesting bits:

<%namespace name="user" module="pylonsforum.model.users" />

The namespace directive here is basically doing an import of a custom module and giving it an alias of user. This is no different than in normal python code typing:

import pylonsforum.model.users as user

This is used later in the div sidebar by pulling the current user from my customer users module:

username : ${user.get_current_user()}

the module source only contains a simple hard coded value for now

def get_current_user(self):

    return "rsvihla"

Now onto the next non-HTML tidbits

<title>${self.title()}</title>

${self.head_tags()}

${self.body()}

Everyone of the above methods establishes a base method that the child templates must now call with the exception of self.body (which is called when the templates actually render the content anyway). So lets adjust “index.mako” template to the following:

<%inherit file="/base.mako"/>

<%def name="title()">Pylons Forum</%def>

<%def name="head_tags()"></%def>

<div id="recentposts">

<table>

<thead><tr><th>subject</th><th>author</th><th>date submitted</th></tr></thead>

<tbody>

%for p in c.posts:

    <tr><td>${p.author}</td><td>${p.subject}</td><td>${p.dateadded}</td></tr>

%endfor 

</tbody>

</table>

</div>

The inherit line at the very top references our base.mako template. The next two lines with the <%def tag are where our previous self.head_tags() and self.title() methods are referenced.

You’ll notice the title method is passing in a new title for the page, and our head_tags method is doing nothing, so no changes there. Browse to your pylons home page and you should see something resembling this:

Your tests should also still pass with no modification.

Forms and Web Helpers

Lets add a basic submit form and a link in the base.mako template to go to the new form page.

Add the following tests to test_home.py

    def test_new_thread(self):

        response = self.app.get(url(controller="home", action="newthread"))

        assert "<input id=\"subject\" name=\"subject\" type=\"text\" />" in response

    def test_submit_new_thread(self):

        response = self.app.post(url(controller="home", action="submitnewthread"), params={'subject':'test','content':'testcontent'})

        assert "post submitted" in response

Install FormBuild with the following command:

Next open lib\helpers.py and change the imports to as follows:

In the sidebar div under username add

<br/>

${h.link_to('new thread', h.url_for(controller='home', action='newthread'))}

Now add an action on the home controller for :”new thread” and create a view in your templates folder called newthread.mako.

Add the following to your home.py controller:

    def newthread(self):

        return render('newthread.mako')

The newthread.mako file should have the following in it:

<%inherit file="/base.mako"/>

<%def name="title()">Make A New Thread</%def>

<%def name="head_tags()"></%def>

${h.form_start(h.url_for(controller='home', action='submitnewthread'), method="post")}

    ${h.field("Subject", field=h.text(name='subject'))}

    ${h.field("Content" ,field=h.textarea(name='content'))} 

    ${h.field(field=h.submit(value="Create Thread", name='submit'))}

${h.form_end()}

Sooooooo what are h values everywhere? Do I need them? They come from our earlier imports in lib\helpers.py , and no you do not actually need to use them, a typical HTML form tag is all you actually need, this is an available option for those interested.

h.form_start and h.form_end are pretty self explanatory
h.field takes label text as argument one, and then a url for argument two.
h.text, h.textarea, h.hidden and h.submit represent their html counterparts.\
h.url_for takes a controller and action to make a url for you.

We now need another action for actually posting our data. so add another method to our home.py controller:

     def submitnewthread(self):

        c.username = uses.get_current_user(self)

        c.subject = request.POST['subject']

        c.content = request.POST['content']

        return render('submitnewthread.mako')

With the following import at the top

import pylonsforum.model.users

Our submitnewthread.mako view should have the following:

<%inherit file="/base.mako"/>

<%def name="title()">Thank You For Your Submission</%def>

<%def name="head_tags()"></%def>

<p>post submitted </p>

data was as follows <br/>

author:  ${c.username} <br/>

subject: ${c.subject} <br />

content: ${c.content} <br />

newthread view should look like so

and submitnewthread page should look like this

Validation

Validation is a huge subject in Pylons, I’m going to just focus on the most basic common case.

Open home.py controller again and add these two imports to the top

import formencode

from pylons.decorators import validate

Then in the same file for now below the post object add

class PostForm(formencode.Schema):

    allow_extra_fields = True

    filter_extra_fields = True

    subject = formencode.validators.String(not_empty=True)

    content = formencode.validators.String(not_empty=True)

Finally on top of the submitnewthead action in the same class file add

    @validate(schema=PostForm(), form='newthread', post_only=True, on_get=True)

    def submitnewthread(self):

Now if you attempt to leave either the subject or content fields blank PylonsForum will now notify you of what you missed.

When done your tests should all pass

Summary

We went through the 80% cases for views, controllers and a brief bit about testing the response object. We still haven’t done a lot of unit testing in the traditional sense but we’ve been focusing exclusively on UI. Stay tuned next for my SQL Alchemy piece.