Kevin Miller's blog

Getting a ASP.Net request URL into a log4net context

kmiller — Fri, 05 Oct 2012 15:29:48 +0000

We make a lot of web applications and we use a lot of logging to diagnose problems on production systems. Often I find myself wishing I knew what URL was being rendered for a given log. In this post I’ll take you through a good solution for adding the current request URL to the log4net context. Better yet we will do this a configurable way so that you can turn this on and off when needed.

One perfectly good way to do this would be to add a debug log to the start of each web request. What I don’t like about this is that it can be tricky to scan logs with simultaneous requests and thus URLs being handled and keep track of which thread is being used to render what URL. Log4net has a context mechanism where you “push” context into “stacks”.

The push method returns a log context which implements IDisposible. When the context is disposed the context is popped off the stack. This allows you to push many different contexts onto the stack.

Let’s take a look at a simple HttpModule which you can use to push the current request’s url into the log context

This module registers two event handlers on begin and end request where begin gets a logger from an IoC container pushes the request url into the context. When the request completes (or the module is disposed) the context is cleaned up effectively removing the url off the log context stack.

A nice side-effect of making this a HTTPModule is that you can add or remove this HttpModule to your application as needed using IIS Manager or by editting your web.config.

And make sure you configure log4net appenders to include the context in log output pattern.

Finally here is an example of this module in action with the configuration settings you are seeing above.

I hope you find this handy. If so let me know. Not so much? Go watch kitten videos on youtube

Dovetail Bootstrap – Now With Service

kmiller — Thu, 13 Sep 2012 20:20:24 +0000

A customer recently asked me how to best create a Windows service using our Dovetail SDK. To me this sounds like a task for Dovetail Bootstrap. We have already provided a nice example of using Bootstrap of to build a web application. We did not, until now, have an example of using Bootstrap and Dovetail SDK to create a Windows service. Bootstrap makes it easy to pull data out of your Clarify database and Topshelf makes it easy to roll your own Windows services . Let’s put them both together and build an example Windows service that monitors case creation in your Clarify database.

If you'd like to skip right to the example you can find it in our Dovetail Bootstrap source repository.

Topshelf

At Dovetail we are big fans of Topshelf and its ability to easily create Windows services from plain .Net console applications. Our Dovetail Seeker and Dovetail Carrier products both use Topshelf for this purpose. First let’s take a look at how Topshelf configuration of your Windows service is done when your application starts up.

The code above informs Topshelf on how to configure Windows when you install this application as a Windows service. As the comment notes. The installation of the Windows service is done using the command line. It is easy to have your deployment script or windows installer automate this step. The most important part here is the service registration. You are not limited to registering only one service. BootstrapService wires everything up so let’s take a look at that.

Topshelf is using an interface here to ensure you have methods for starting and stopping the service. In this example the start method constructs the IoC container and uses it to create a CaseMonitor which will do all the work. The IoC container is there because Bootstrap uses it to bring a lot of value. Things like automatic population of settings objects, getting a ClarifySession for the current user, logging support, and much more. Additionally using IoC containers make writing tests easier. Lets move on and take a look at the CaseMonitor.

Polling The Database

CaseMonitor starts a timer that pulses every 15 seconds polling the database using a model map filtered by a timespan. It them simply writes to a logger what it finds. The IoC container is in charge of providing the CaseMonitor’s constructor with instances of the types it is dependent on. The more interesting thing here is the IModelBuilder or RecentCaseModel. This builder uses a Model Map to know how to pull data from a Clarify database.

Model Maps

At Dovetail we use Model Maps extensively in our applications when we need to pull data out of the database into models (Projection) which we’ll be using in our MVC based applications.

Wrap Up

You might notice how little data access code we actually wrote. That is the beauty of Model Map. To make the Windows service aspect of your application you only need to give Topshelf some details and implement a ServiceControl based class. The two Dovetail applications I mentioned use the basic pattern demonstrated here but push TopShelf harder registering many Services using the IoC container and utilizing a message bus architecture MassTransit from the same smart people Dru Sellers and Chris Patterson that brought you Topshelf. This extra messaging sauce which makes it easy to produce and consume a lot of work across many threads making the most of your server hardware.

FubuMVC Hijacking Behaviors

kmiller — Tue, 24 Apr 2012 22:24:21 +0000

Have you every wanted a convention to wrap an endpoint and conditionally transfer the original chain to a different one? No server redirect here just plain old hijack your currently scheduled response with another one. This is the FubuMVC equivalent of the classic ASP Server.Transfer. Here are a couple of examples on how to do this.

Scenario: Web Service API Exception Handling

We have a web services application where when things go wrong we want a pretty error page for the developer type who is consuming the service. You don’t want an ugly yellow and red YSOD you have too much style for that.

Answer: Wrap the Behavior Chain

In FubuMVC you can easily wrap one behavior with another.

This simple convention pulled from Dovetail Bootstrap wraps all of our API actions with the exception handling behavior. The generic argument is the type of the request you wish to redirect to when something goes wrong.

The behavior does the heavy lifting putting a try/catch around a wrapped behavior. When an exception is thrown and Asp.Net custom errors are enabled we need to hijack the behavior and return a different result. The generic argument type is used to build the target chain’s input model and invoke that behavior chain using a FubuPartial.

A Partial? What the fsck is a FubuPartial?

A FubuPartial is a really silly way of saying that we want to render the result of an input model. A partial can be rendered two ways: with or without authorization nodes. Invoke() checks the authorization nodes and InvokePartial() does not. Why wouldn’t you want to authorize? Well your behavior is likely already positioned after the authorization nodes in the behavior chain so it might not benefit anyone to do yet another authorization. Unless that is you do for this particular part of your application.

In the end a FubuPartial is a handy way to get and render the contents of any behavior chain given you have the input model. Tldr; Ok there you go this was the big take away from this post. The rest is window dressing and example code and chest thumping.

Scenario: Single Page Web App

Often with a single page application using a client side framework like Backbone.js you want to support deep linking using browser History and pushState. Let’s support transferring web service requests coming from search engines and user’s bookmarks to the site’s actual home page while leaving the URL intact so the client side framework can be in-charge of rendering the page.

Background

A single page app does not reload it uses AJAX requests for everything. Your single page FubuMVC application will likely have RESTful-ish web services returning JSON for each of the views presented to the user. Urls like:

/customer/new
/customer/1
/customer/1/edit

Your web services only return JSON and likely don’t have HTML versions of their results. If a user bookmarks one of these links they will only get back JSON. Yuck!

Enter TransferAPIRequestsTo Behavior

How about we Hijack web service requests that come from actual browsers. We do this in similar fashion as the first example:

The API requests are wrapped with a behavior that checks to see if the request’s mimetype want’s HTML. If so, just like before we get the partial for the generic argument’s input model and render it.

Conclusion

A couple of interesting scenarios that leverage the flexibility of FubuMVC in similar yet very different ways.

Dovetail SDK C# Helper Extensions

kmiller — Wed, 29 Feb 2012 17:40:52 +0000

Dovetail SDK is quite useful for doing data access with your Clarify databases. When using our ClarifyGenerics there is some ceremony to the syntax there for backwards compatibility that always bothered me. I’ve written a few helper extensions for Dovetail SDK that I use a lot these days so I thought I should share them with you.

ClarifyGeneric Extensions

I use these to help me filter generics in a fluent way. Create and Traverse to generics without needed a separate line to set which fields should be selected.

ClarifyDataRow Extensions

Row fields are plain objects but you always want to put them into a strongly typed variable. I use these to get at fields without having to do a lot of conversions and checking for DBNull.

An example of these extensions in play

Here I create a generic. Filter it in my fancy new way and project the results into many result objects.

Automating Deployment For Clairfy Using Rake

kmiller — Mon, 06 Feb 2012 19:03:02 +0000

Gary posted recently on Automating Deployment For Clarify. A customer recently had an automation need that I answered in a similar way but rather than using NAnt I pulled from our fancy favorite new build automation tool Rake. In this post we’ll take a look at automating common database setup tasks like applying schema scripts and executing arbitrary SQL using Rake. I hope you find it useful.

Where is the Code?

I put together a Dovetail Automation gist with everything you need to solve two problems.

Applying Dovetail Schema Editor schema scripts.
Applying all SQL scripts in a directory.

Check out the readme for setup and execution instructions. Next I’ll talk a little about what each task does.

Applying Schema Scripts (apply_schemascripts)

This task looks for a child directory named schema below where the rakefile.rb is located. Every file in this directory ending in schemascript.xml will be applied to your target database.

Applying SQL Scripts (apply_sql)

This task looks for the child directory named database again below the current working directory. Every file ending in .sql is

Ordering

Both these tasks apply files to your database in the order they are found in the file system. This means you’ll want your scripts to be orthogonal to each other.

I’ll leave it as an exercise to the reader to customize these scripts to follow a conventional file ordering. If you do fork the gist and let me know.

My Quick Argument For Rake over Nant

The main difference between to Nant and Rake is that NAnt encodes its behaviors using XML and Rake uses the scripting language Ruby. I am a self proclaimed NAnt Ninja but I do everything I can in Rake these days. Why? XML is pretty terrible at expressing useful behavior compared to a top shelf language like Ruby with lots of extensibility via its deep repository of Gems.

Introducing Dovetail Bootstrap

kmiller — Tue, 24 Jan 2012 23:24:46 +0000

Over the years we’ve grown a lot of infrastructure for building web applications on Dovetail SDK. These infrastructure pieces get used by a lot of different Dovetail products. Today we are launching Dovetail Bootstrap which packages up many handy capabilities like dynamic case history and case creation on top of common infrastructure like: model projection, authentication, logging, and session management. The core goal of Dovetail Bootstrap is to get Dovetail SDK developers up and creating web applications quickly.

How do I get it?

The code for Bootstrap is open and available on GitHub. Feel free to watch, fork and improve the code. We’d love to work with you to add great new things to Bootstrap. I recommend reading the ReadMe as it takes you through setting up a development environment.

What Can Bootstrap Do?

Bootstrap is not a product but a starting point giving you the building blocks of your own applications. Let’s take a look at the basic features that are there now.

Authentication

Implementing authentication is a pain. Bootstrap’s “Hello Bootstrap” web application has a simple one. When a user is not authenticated they are redirected to the Sign In page.

After signing in, the user is taken to the URL they originally requested. Your code can easily get the current user of the current incoming web request. This is all pretty typical stuff for web developers but a pain to write and wire up. Below, the master layout uses a partial to display the current user.

Case Listing

The home screen, above, demonstrates showing the user’s open cases. To accomplish this we are using a Model Map to pull case details from a view (qry_case_view) to a view model object which the Spark view engine will consume.

Clicking on the case ID takes us to the case history.

Dynamic History

We have invested a lot in our dynamic history infrastructure and here we are sharing it with you. Getting good case history is a tricky problem. There is the static history on the case record which is not that useful as its length maxes out at 32K. The better alternative is to use the activity entries logged for every action taken, but hand building history via activity entry queries and collating results is a tricky and manual data access task.

All the work to build this view is done by Bootstrap’s history assembler and it works for cases, subcases, and solutions. Better yet the output of the history generated can be extensively customized:

· Add or remove which history (activity) entries are shown

· Edit history contents, including getting records related to the activity entry.

· Child object history can be mixed in e.g. case history can contain selected sub case history items.

The output of the history assembler is a history view model and is rendered by the appropriate view.

Create Case

Most everyone wants to create a case at one time or another. Bootstrap comes with an example of doing this using our Dovetail SDK

Look in the /api/cases/create handler directory. The get_handler class is in charge of showing the form for GET requests of /api/cases/create url. The post_handler class handles POSTs to the same URL and creates the case based in the input model.

Take a look a the drop downs for the case Type, Severity and Priority. The dropdown values are automatically populated by a feature of FubuMVC called HTML Conventions.

In this case we set an attribute on the output model of the create case GET. We have trained the web application to render special Html when trying to display an input for this “special” attributed property. HTML Conventions are another whole blog post. I put together a gist containing all the moving parts that make this happen.

Application List Listing (try to say that fast)

Application lists (GBSTs) are used a lot in Clarify. In fact, I just talked a bit too much about conventionally showing drop downs for them above. As an example in Bootstrap we provide a simple listing and detail view of application lists.

Take a look at how easy it is to do the ClarifyGeneric query used to generate the listing. Here is a detail view of one of the lists showing off its elements.

To show this list detail view we project a view model from the Dovetail SDK’s list cache ordered by rank highlighting the default list element.

That finishes our fly through of the Bootstrap Web example. There is an extremely exciting About page that I did not show off. I’ll let you grab the code and take it for a spin to see that gem.

What exactly is in Bootstrap?

Bootstrap is written in .Net 4.0. It currently consists of 3 libraries and a web application pulling the libraries together showing it all off.

Bootstrap – SDK session management, history builder, settings (configuration) support.
ModelMap – map view models to your database schema and easily populate them (awesome).
Fubu – FubuMVC specific infrastructure: list rendering conventions, forms and token based authentication.
Web – An example Agent authenticated web application with a user customized home page, case history rendering, create case example, application lists

Retrieving data using ModelMap

Long ago I talked about the first iteration of ModelMap called DataMapper.

“~~DataMap~~ ModelMap makes it easy to populate model objects from a Clarify/Dovetail CRM database. It is a sort of one-way Clarify specific object relational mapper tool.”

Model Map saves you from having to write a lot of data access code for querying and mapping data from query results into view models. For more details take a look at my original post and at the example map shipped with Bootstrap which is used by the home page to build a list of the current user’s open cases. If you don’t find anything else useful about Bootstrap do stop and take a look at the power of ModelMap at Dovetail we use it everywhere.

Web Application Framework

The Bootstrap web application is built on ASP.Net using the FubuMVC web framework. At Dovetail we use FubuMVC a lot. My secondary goal of Bootstrap is to make our customers more aware of “Fubu” by shipping great and useful examples of using it.

By no means are you required to write only FubuMVC apps using Bootstrap. In fact, I’ve updated our Dovetail Mobile product (based on ASP.Net MVC 2) to use Bootstrap to prove out this assertion. I may later provide guidance for using Bootstrap with ASP.Net MVC. Let me know if this interests you.

Wrap Up

That is about it for the main features and examples in the Bootstrap web application example. Bootstrap is by no means is done. We are starting to use Bootstrap as a foundation for our products at Dovetail and it has been really handy publishing the Bootstrap libraries as Nuget packages for use by us internally.

The name of the project is a blatant rip-off of Twitter Bootstrap which we use in the web application.

I apologize for glossing over a lot of FubuMVC detail there is an ever improving about of educational information on Fubu out there. Go ahead and follow them on the Twitters.

If you have any suggestions or find bugs hit me on or Twitter or create a GitHub issue. If you are stuck please do send me a line.

Compositional Architecture FTW

kmiller — Thu, 09 Jun 2011 22:01:55 +0000

While doing a deployment of our Dovetail Carrier product our customer was having connectivity issues with their email server. We ran into situation where we needed prove that the product could connect and simply firing up an email client was not as good as proving that our code worked. The compositional architecture of Carrier allowed me to quickly build a little console application to detect email configuration problems.

Recompose It

Plucking the necessary pieces of code out of our application and recomposing a utility to solve the problem at hand was easy. Here is the gist of the console code.

Writing this code did not take long.

Lean on the container

Allowing the container to compose my types for me frees me from having to worry about minutia. There are other things going on behind the scenes that I don’t have to worry about. For better output I pulled in our logging infrastructure. Way better than writing to standard out.

Validation?

Not really. A minute before I posted the console app to the customer figured out the problem was on their side. But had the mystery still been afoot we’d had a tool on our belt.

Dovetail Software is Multi-Core Aware

kmiller — Tue, 12 Apr 2011 21:31:08 +0000

A customer recently asked if they should provision multiple CPUs on virtual machines hosting Dovetail applications. The short answer is Yes most of our applications do a pretty good job taking advantage of multiple CPU cores. Let’s take a look at this in a bit more detail.

Our Clarify products all use our .Net Dovetail SDK. The SDK is not itself multi-threaded (each thread runs on a CPU core) but most Dovetail products using the SDK utilize multiple cores in one way or another.

Web Applications

Most of our customers use one of our web applications. Dovetail Agent, Agent Lite, and Dovetail Mobile, and Dovetail Administration are implicitly multi-threaded because the web server (IIS) that they run on uses a thread per web request being processed.

Note: There is a limit to the number of threads IIS will put into play.

Message Based Applications

Dovetail Seeker and Dovetail Carrier are message based applications which run as Windows services. They both use multiple threads to produce and consume messages. This architecture does a very good job at parallelizing the work being done. For example Dovetail Seeker is a bit CPU intensive when indexing a large amount of content. We have seen a linear increase in the amount of content that can be indexed concurrently when adding additional cores. Here is a screenshot from my development machine doing some heavy Dovetail Seeker indexing.

Seeker’s usage is not purely CPU as there is still a bit of waiting that occurs while talking to the database and file system. It doesn’t hurt to have a fast database server and local file system where the search indexes get written.

Dovetail RuleManager

Dovetail RuleManager is very much like our message base applications. It is a Windows service and uses multiple threads. We wrote this application before our experience with message based architectures. If I did major rework this application would become message based. RuleManager uses a producer consumer multi-threading model where one thread polls the database for events feeding consumer threads work to do when events are found.

Conclusion

Should you use multiple CPUs for Dovetail applications? Yes. How many? I would recommend at least 2 cores for web applications. Our message based applications can in theory use as many CPUs as you throw at them.

Making ASP.Net Custom Errors Less Dookie

kmiller — Thu, 13 Jan 2011 22:15:51 +0000

I wanted to share some lessons learned from an experience recently adding custom error pages to one of our FubuMVC projects deployed to IIS7.

Treat your users well

Having good looking error pages in your app is a nice sign of polish. They mean that you care about your user’s experience even when things go wrong. Github, who always seems to knock it out of the park, recently updated their error pages. I love the 404 page complete with parallax JavaScript effects.

ASP.Net has this all handled right?

Sure they do. Well sort of. Lets take a look at your web.config.

httpErrors – good for static content

An httpError seems to allow you to customize error pages for static sites and the parts of your app that do not flow through the ASP.Net runtime. For us this worked well for our 404. The main problem here is that you cannot use a virtual path to your error page.

Unfortunately, this requires that we have a deployment step to set the correct value in the path attribute of the error element.

customErrors – almost good but not quite great

A customError tells ASP.Net to avoid the built in IIS (404, 403) and Yellow Screens of Death (500) error pages and instead use what you provide.

We ran into one major problem with this. What actually happens under the hood is that an error on your page generates a 302 redirect to your custom error page.

This doesn’t seem too bad. Until you think about it. You have an error page with is ultimately returning a 200 OK status. It is saying everything is OK when it is NOT. The mismatched status code breaks the semantic contract your error page has with your browser.

Where this goes terribly wrong is with AJAX requests. AJAX will now receive no error when things go wrong. This has the effect of really messing up AJAX requests that pull in server side generated views.

See how the error page is plopped into the middle of the website. Lovely. This happened because jQuery’s ajax method dutifully followed the error redirect pulled in the error page’s html content into results div.

We have error handlers which look much nicer. Let’s make them work.

Fixing the AJAX error redirect problem

We tried way to hard to get this to work. In the end it was something simple that our Dovetail colleagues had already found. The solution is to prevent the custom error page from rendering for AJAX requests. Turns out this is pretty easy.

This code is cancelling the error and rendering a short error page with the proper 500 error when something goes wrong. Now our example AJAX request can handle the error.

Conclusion

Hopefully this was useful and we didn’t miss too many obvious alternative solutions. Down the road I’d rather have a behavior in FubuMVC that skips the custom errors if they are enabled and also supports 403 authorization errors behind AJAX requests. Better yet I wish that custom errors would do the right thing and serve up the correct status by default.

How To Setup hMailServer To Use a SSL Certificate

kmiller — Tue, 17 Aug 2010 15:17:35 +0000

I am adding IMAP support to one of our products. Likely more that one person out there a needed to do this, so enjoy. I’ll take you from creating an SSL certificate to configuring hMailServer to work with both secure and regular connections to testing your setup.

Creating a Self Signed SSL Certificate

First things first you’ll need to download OpenSSL. I downloaded the 64bit 1.0 light version which required Visual C++ 2008 Redistributables (x64) to be installed first. I told the installer to put OpenSSL in my c:\utilites\ folder.

Create a Key

Next up you’ll need to create a key. I recommend you replace <host> with your machine name.

>openssl genrsa -out <host>.key 1024

Certificate Request

Now you need to create a certificate request. This is the file you normally send off to your SSL certificate provider.

>openssl req -new -nodes -key <host>.key -out <host>.csr

Oops if you’re like me here you’ll get this error:

Unable to load config info from C:/lan/ssl/openssl.cnf

Looks like the default config location is not cutting it. You can clue OpenSSL in with an environment variable:

SET OPENSSL_CONF=c:\utilities\OpenSSL-Win64\bin\openssl.cfg

Let’s try that again…

>openssl req -new -nodes -key <host>.key -out <host>.csr

Answer the questions the best you can.

For the Common Name you should put the fully qualified domain name of the server where you will deploy the certificate.

Self Signing the Certificate Request

If you are like me this certificate is for development and you won’t really need a certificate authority to sign it for testing purposes. Luckily it is easy to generate your own certificate.

>openssl x509 -req -days 1024 -in <host>.csr -signkey <host>.key -out <host>.cert

Configuring hMailServer With Your New SSL Certificate

Bring up your hMailServer administrator UI and add a SSL certificate

Next create new TCP/IP ports which use SSL for each protocol you are interested in testing.

Notice the port numbers I use for IMAP (993) POP3 (995) seem to be the default secure ports for these two protocols. Each change will restart hMailServer.

Test Your Secure Connection

Go ahead and fire up your favorite email client. I use Windows Live Mail which I’ve configured to pull email from one of the test accounts I’ve setup on hMailServer.

And now for a test message or two?

I’ve setup a few pop3 accounts and an IMAP and had a lot of fun sending my virtual test personas emails. The dialog is not so riveting.

You’ve Got SSL

There you go. Hopefully you didn’t run into anything weird that I did not. This post is basically a roll up of the hMailServer SSL certificate documentation. And Generating Certificate Signing Requests from Jeremy Mate