Kevin Miller – Clarify Solutions | Dovetail Software

TikaOnDotnet now supports Tika 1.7

Kevin Miller — Fri, 06 Feb 2015 23:01:09 +0000

TikaOnDotNet has been updated to the latest Tika 1.7. As always a Nuget has been published.

Two things were great about this release.

Jason Rotello did the heavy lifting grabbing an issue I had marked up for grabs and got it done. This is his second time updating the libray. Thank you very much Jason!
It fixes a known problem where indexing mp4 video files would cause the file to be locked.

Great way to end the week.

The post TikaOnDotnet now supports Tika 1.7 appeared first on Clarify Solutions | Dovetail Software.

FubuMVC using ModifyChainAttributes

Kevin Miller — Fri, 30 Jan 2015 15:34:25 +0000

Let’s take a look at a handy way to quickly modify a FubuMVC behavior chain by simply placing an Attribute on the action method. This technique is handy when you have a one off change you need to make to a specific action and rolling your own site wide convention is too much.

The Problem

FubuMVC uses content negotiation to control how your output model is formatted. Great stuff when you are creating endpoints to be used as an API for your fancy JavaScript framework of choice.

We ran into a problem on a specific API endpoint which supports file uploads where IE9 would barf on a JSON response after uploading a file. In this case the response was not being consumed by the client. No big deal let’s force the output to be XML to ensure that IE9 is happy.

To do this we will need to remove all non XML output formatters from the behavior chain.

Create a Modify Chain Attribute

Implementing a chain modifying attribute is easy.

Subclass ModifyChainAttribute.
Edit the parent behavior chain (via the ActionCall)
Done

You then simply apply your new custom attribute to your action. Take a look at FubuDiagnostics and see the changes to your chain. In this case there is only one output formatter XmlFormatter.

FubuMVC Wisdom: Bottles and StructureMap Registries

Kevin Miller — Thu, 29 Jan 2015 21:11:56 +0000

FubuMVC Bottles are pretty great. You can mix code, web endpoints and views in a “Bottle” assembly. Package them up as Nugets and you can re-use application functionality between your web applications just by adding a reference to a Bottle assembly.

Auto Import Referenced Bottle Registries

It is a really common pattern when you are using an IoC container like StructureMap in your Bottle that you want a registry imported when your bottle is loaded. Add the following code to the StructureMap Registry which you use when spinning up your FubuMVC application and your Bottle registries will be imported.

More On Bottles If you Like

Josh Arnold has a
post
or two about Bottles.

Also Jeremy Miller did a nice presentation on Bottles at Monkey Square.

The post FubuMVC Wisdom: Bottles and StructureMap Registries appeared first on Clarify Solutions | Dovetail Software.

TikaOnDotNet 1.4 Released as a Nuget

Kevin Miller — Fri, 12 Jul 2013 20:29:16 +0000

A while back I shared a post about how we were successfully able to use the excellent , yet Java based Tika text extraction library in our .NET based applications. Along side that post I also created a GitHub repo where the code for TikaOnDotNet lives and is maintained. At Dovetail Software we continue to use this library with great success on the .NET platform. Externally I’ve gotten responses from people using the ideas in the project but often they have problems creating their own release or getting Tika up and running in their projects. Today I gave the project some love and some polish and updated it to support being consumed as a Nuget package which makes it really easy to use from your code base. Let’s take a look at how to use Tika in your .Net projects.

From Zero to Text Extraction

Step 1 – Install the Nuget

Step 2 – New up a TextExtractor

Something to note here. The text extractor class uses Tika’s auto detection mechanism. We are not explicitly use Tika’s Office document extraction parsers. Tika has the capability to detect the incoming content and extract the text from it. This is very useful for search engines like Dovetail Seeker. Tika is an extensive library and I have just wrapped this one mechanism. If you like you can use the full power of Tika directly from your .NET code. Check out Tika In Action for more about what Tika can do.

Finale – Compare the text with the original

The code above runs an ancient Word document through Tika text extraction engine. This document is part of our testing of Tika’s compatibility with… ancient word documents.

Original

Extracted Text

The post TikaOnDotNet 1.4 Released as a Nuget appeared first on Clarify Solutions | Dovetail Software.

Getting a ASP.Net request URL into a log4net context

Kevin Miller — Fri, 05 Oct 2012 20:29:48 +0000

We make a lot of web applications and we use a lot of logging to diagnose problems on production systems. Often I find myself wishing I knew what URL was being rendered for a given log. In this post I’ll take you through a good solution for adding the current request URL to the log4net context. Better yet we will do this a configurable way so that you can turn this on and off when needed.

One perfectly good way to do this would be to add a debug log to the start of each web request. What I don’t like about this is that it can be tricky to scan logs with simultaneous requests and thus URLs being handled and keep track of which thread is being used to render what URL. Log4net has a context mechanism where you “push” context into “stacks”.

The push method returns a log context which implements IDisposible. When the context is disposed the context is popped off the stack. This allows you to push many different contexts onto the stack.

Let’s take a look at a simple HttpModule which you can use to push the current request’s url into the log context

This module registers two event handlers on begin and end request where begin gets a logger from an IoC container pushes the request url into the context. When the request completes (or the module is disposed) the context is cleaned up effectively removing the url off the log context stack.

A nice side-effect of making this a HTTPModule is that you can add or remove this HttpModule to your application as needed using IIS Manager or by editting your web.config.

And make sure you configure log4net appenders to include the context in log output pattern.

Finally here is an example of this module in action with the configuration settings you are seeing above.

I hope you find this handy. If so let me know. Not so much? Go watch kitten videos on youtube

The post Getting a ASP.Net request URL into a log4net context appeared first on Clarify Solutions | Dovetail Software.

Dovetail Bootstrap – Now With Service

Kevin Miller — Thu, 13 Sep 2012 20:15:33 +0000

A customer recently asked me how to best create a Windows service using our Dovetail SDK. To me this sounds like a task for Dovetail Bootstrap. We have already provided a nice example of using Bootstrap of to build a web application. We did not, until now, have an example of using Bootstrap and Dovetail SDK to create a Windows service. Bootstrap makes it easy to pull data out of your Clarify database and Topshelf makes it easy to roll your own Windows services . Let’s put them both together and build an example Windows service that monitors case creation in your Clarify database.

If you’d like to skip right to the example you can find it in our Dovetail Bootstrap source repository.

Topshelf

At Dovetail we are big fans of Topshelf and its ability to easily create Windows services from plain .Net console applications. Our Dovetail Seeker and Dovetail Carrier products both use Topshelf for this purpose. First let’s take a look at how Topshelf configuration of your Windows service is done when your application starts up.

The code above informs Topshelf on how to configure Windows when you install this application as a Windows service. As the comment notes. The installation of the Windows service is done using the command line. It is easy to have your deployment script or windows installer automate this step. The most important part here is the service registration. You are not limited to registering only one service. BootstrapService wires everything up so let’s take a look at that.

Topshelf is using an interface here to ensure you have methods for starting and stopping the service. In this example the start method constructs the IoC container and uses it to create a CaseMonitor which will do all the work. The IoC container is there because Bootstrap uses it to bring a lot of value. Things like automatic population of settings objects, getting a ClarifySession for the current user, logging support, and much more. Additionally using IoC containers make writing tests easier. Lets move on and take a look at the CaseMonitor.

Polling The Database

CaseMonitor starts a timer that pulses every 15 seconds polling the database using a model map filtered by a timespan. It them simply writes to a logger what it finds. The IoC container is in charge of providing the CaseMonitor’s constructor with instances of the types it is dependent on. The more interesting thing here is the IModelBuilder or RecentCaseModel. This builder uses a Model Map to know how to pull data from a Clarify database.

Model Maps

At Dovetail we use Model Maps extensively in our applications when we need to pull data out of the database into models (Projection) which we’ll be using in our MVC based applications.

Wrap Up

You might notice how little data access code we actually wrote. That is the beauty of Model Map. To make the Windows service aspect of your application you only need to give Topshelf some details and implement a ServiceControl based class. The two Dovetail applications I mentioned use the basic pattern demonstrated here but push TopShelf harder registering many Services using the IoC container and utilizing a message bus architecture MassTransit from the same smart people Dru Sellers and Chris Patterson that brought you Topshelf. This extra messaging sauce which makes it easy to produce and consume a lot of work across many threads making the most of your server hardware.

The post Dovetail Bootstrap – Now With Service appeared first on Clarify Solutions | Dovetail Software.

FubuMVC Hijacking Behaviors

Kevin Miller — Wed, 25 Apr 2012 03:24:21 +0000

Have you every wanted a convention to wrap an endpoint and conditionally transfer the original chain to a different one? No server redirect here just plain old hijack your currently scheduled response with another one. This is the FubuMVC equivalent of the classic ASP Server.Transfer. Here are a couple of examples on how to do this.

Scenario: Web Service API Exception Handling

We have a web services application where when things go wrong we want a pretty error page for the developer type who is consuming the service. You don’t want an ugly yellow and red YSOD you have too much style for that.

Answer: Wrap the Behavior Chain

In FubuMVC you can easily wrap one behavior with another.

This simple convention pulled from Dovetail Bootstrap wraps all of our API actions with the exception handling behavior. The generic argument is the type of the request you wish to redirect to when something goes wrong.

The behavior does the heavy lifting putting a try/catch around a wrapped behavior. When an exception is thrown and Asp.Net custom errors are enabled we need to hijack the behavior and return a different result. The generic argument type is used to build the target chain’s input model and invoke that behavior chain using a FubuPartial.

A Partial? What the fsck is a FubuPartial?

A FubuPartial is a really silly way of saying that we want to render the result of an input model. A partial can be rendered two ways: with or without authorization nodes. Invoke() checks the authorization nodes and InvokePartial() does not. Why wouldn’t you want to authorize? Well your behavior is likely already positioned after the authorization nodes in the behavior chain so it might not benefit anyone to do yet another authorization. Unless that is you do for this particular part of your application.

In the end a FubuPartial is a handy way to get and render the contents of any behavior chain given you have the input model. Tldr; Ok there you go this was the big take away from this post. The rest is window dressing and example code and chest thumping.

Scenario: Single Page Web App

Often with a single page application using a client side framework like Backbone.js you want to support deep linking using browser History and pushState. Let’s support transferring web service requests coming from search engines and user’s bookmarks to the site’s actual home page while leaving the URL intact so the client side framework can be in-charge of rendering the page.

Background

A single page app does not reload it uses AJAX requests for everything. Your single page FubuMVC application will likely have RESTful-ish web services returning JSON for each of the views presented to the user. Urls like:

/customer/new
/customer/1
/customer/1/edit

Your web services only return JSON and likely don’t have HTML versions of their results. If a user bookmarks one of these links they will only get back JSON. Yuck!

Enter TransferAPIRequestsTo Behavior

How about we Hijack web service requests that come from actual browsers. We do this in similar fashion as the first example:

The API requests are wrapped with a behavior that checks to see if the request’s mimetype want’s HTML. If so, just like before we get the partial for the generic argument’s input model and render it.

Conclusion

A couple of interesting scenarios that leverage the flexibility of FubuMVC in similar yet very different ways.

The post FubuMVC Hijacking Behaviors appeared first on Clarify Solutions | Dovetail Software.

Dovetail SDK C# Helper Extensions

Kevin Miller — Wed, 29 Feb 2012 01:06:52 +0000

Dovetail SDK is quite useful for doing data access with your Clarify databases. When using our ClarifyGenerics there is some ceremony to the syntax there for backwards compatibility that always bothered me. I’ve written a few helper extensions for Dovetail SDK that I use a lot these days so I thought I should share them with you.

ClarifyGeneric Extensions

I use these to help me filter generics in a fluent way. Create and Traverse to generics without needed a separate line to set which fields should be selected.

ClarifyDataRow Extensions

Row fields are plain objects but you always want to put them into a strongly typed variable. I use these to get at fields without having to do a lot of conversions and checking for DBNull.

An example of these extensions in play

Here I create a generic. Filter it in my fancy new way and project the results into many result objects.

The post Dovetail SDK C# Helper Extensions appeared first on Clarify Solutions | Dovetail Software.

Automating Deployment For Clairfy Using Rake

Kevin Miller — Mon, 06 Feb 2012 19:03:02 +0000

Gary posted recently on Automating Deployment For Clarify. A customer recently had an automation need that I answered in a similar way but rather than using NAnt I pulled from our fancy favorite new build automation tool Rake. In this post we’ll take a look at automating common database setup tasks like applying schema scripts and executing arbitrary SQL using Rake. I hope you find it useful.

Where is the Code?

I put together a Dovetail Automation gist with everything you need to solve two problems.

Applying Dovetail Schema Editor schema scripts.
Applying all SQL scripts in a directory.

Check out the readme for setup and execution instructions. Next I’ll talk a little about what each task does.

Applying Schema Scripts (apply_schemascripts)

This task looks for a child directory named schema below where the rakefile.rb is located. Every file in this directory ending in schemascript.xml will be applied to your target database.

Applying SQL Scripts (apply_sql)

This task looks for the child directory named database again below the current working directory. Every file ending in .sql is

Ordering

Both these tasks apply files to your database in the order they are found in the file system. This means you’ll want your scripts to be orthogonal to each other.

I’ll leave it as an exercise to the reader to customize these scripts to follow a conventional file ordering. If you do fork the gist and let me know.

My Quick Argument For Rake over Nant

The main difference between to Nant and Rake is that NAnt encodes its behaviors using XML and Rake uses the scripting language Ruby. I am a self proclaimed NAnt Ninja but I do everything I can in Rake these days. Why? XML is pretty terrible at expressing useful behavior compared to a top shelf language like Ruby with lots of extensibility via its deep repository of Gems.

The post Automating Deployment For Clairfy Using Rake appeared first on Clarify Solutions | Dovetail Software.

Introducing Dovetail Bootstrap

Kevin Miller — Wed, 25 Jan 2012 05:24:46 +0000

Over the years we’ve grown a lot of infrastructure for building web applications on Dovetail SDK. These infrastructure pieces get used by a lot of different Dovetail products. Today we are launching Dovetail Bootstrap which packages up many handy capabilities like dynamic case history and case creation on top of common infrastructure like: model projection, authentication, logging, and session management. The core goal of Dovetail Bootstrap is to get Dovetail SDK developers up and creating web applications quickly.

How do I get it?

The code for Bootstrap is open and available on GitHub. Feel free to watch, fork and improve the code. We’d love to work with you to add great new things to Bootstrap. I recommend reading the ReadMe as it takes you through setting up a development environment.

What Can Bootstrap Do?

Bootstrap is not a product but a starting point giving you the building blocks of your own applications. Let’s take a look at the basic features that are there now.

Authentication

Implementing authentication is a pain. Bootstrap’s “Hello Bootstrap” web application has a simple one. When a user is not authenticated they are redirected to the Sign In page.

After signing in, the user is taken to the URL they originally requested. Your code can easily get the current user of the current incoming web request. This is all pretty typical stuff for web developers but a pain to write and wire up. Below, the master layout uses a partial to display the current user.

Case Listing

The home screen, above, demonstrates showing the user’s open cases. To accomplish this we are using a Model Map to pull case details from a view (qry_case_view) to a view model object which the Spark view engine will consume.

Clicking on the case ID takes us to the case history.

Dynamic History

We have invested a lot in our dynamic history infrastructure and here we are sharing it with you. Getting good case history is a tricky problem. There is the static history on the case record which is not that useful as its length maxes out at 32K. The better alternative is to use the activity entries logged for every action taken, but hand building history via activity entry queries and collating results is a tricky and manual data access task.

All the work to build this view is done by Bootstrap’s history assembler and it works for cases, subcases, and solutions. Better yet the output of the history generated can be extensively customized:

· Add or remove which history (activity) entries are shown

· Edit history contents, including getting records related to the activity entry.

· Child object history can be mixed in e.g. case history can contain selected sub case history items.

The output of the history assembler is a history view model and is rendered by the appropriate view.

Create Case

Most everyone wants to create a case at one time or another. Bootstrap comes with an example of doing this using our Dovetail SDK

Look in the /api/cases/create handler directory. The get_handler class is in charge of showing the form for GET requests of /api/cases/create url. The post_handler class handles POSTs to the same URL and creates the case based in the input model.

Take a look a the drop downs for the case Type, Severity and Priority. The dropdown values are automatically populated by a feature of FubuMVC called HTML Conventions.

In this case we set an attribute on the output model of the create case GET. We have trained the web application to render special Html when trying to display an input for this “special” attributed property. HTML Conventions are another whole blog post. I put together a gist containing all the moving parts that make this happen.

Application List Listing (try to say that fast)

Application lists (GBSTs) are used a lot in Clarify. In fact, I just talked a bit too much about conventionally showing drop downs for them above. As an example in Bootstrap we provide a simple listing and detail view of application lists.

Take a look at how easy it is to do the ClarifyGeneric query used to generate the listing. Here is a detail view of one of the lists showing off its elements.

To show this list detail view we project a view model from the Dovetail SDK’s list cache ordered by rank highlighting the default list element.

That finishes our fly through of the Bootstrap Web example. There is an extremely exciting About page that I did not show off. I’ll let you grab the code and take it for a spin to see that gem.

What exactly is in Bootstrap?

Bootstrap is written in .Net 4.0. It currently consists of 3 libraries and a web application pulling the libraries together showing it all off.

Bootstrap – SDK session management, history builder, settings (configuration) support.
ModelMap – map view models to your database schema and easily populate them (awesome).
Fubu – FubuMVC specific infrastructure: list rendering conventions, forms and token based authentication.
Web – An example Agent authenticated web application with a user customized home page, case history rendering, create case example, application lists

Retrieving data using ModelMap

Long ago I talked about the first iteration of ModelMap called DataMapper.

“~~DataMap~~ ModelMap makes it easy to populate model objects from a Clarify/Dovetail CRM database. It is a sort of one-way Clarify specific object relational mapper tool.”

Model Map saves you from having to write a lot of data access code for querying and mapping data from query results into view models. For more details take a look at my original post and at the example map shipped with Bootstrap which is used by the home page to build a list of the current user’s open cases. If you don’t find anything else useful about Bootstrap do stop and take a look at the power of ModelMap at Dovetail we use it everywhere.

Web Application Framework

The Bootstrap web application is built on ASP.Net using the FubuMVC web framework. At Dovetail we use FubuMVC a lot. My secondary goal of Bootstrap is to make our customers more aware of “Fubu” by shipping great and useful examples of using it.

By no means are you required to write only FubuMVC apps using Bootstrap. In fact, I’ve updated our Dovetail Mobile product (based on ASP.Net MVC 2) to use Bootstrap to prove out this assertion. I may later provide guidance for using Bootstrap with ASP.Net MVC. Let me know if this interests you.

Wrap Up

That is about it for the main features and examples in the Bootstrap web application example. Bootstrap is by no means is done. We are starting to use Bootstrap as a foundation for our products at Dovetail and it has been really handy publishing the Bootstrap libraries as Nuget packages for use by us internally.

The name of the project is a blatant rip-off of Twitter Bootstrap which we use in the web application.

I apologize for glossing over a lot of FubuMVC detail there is an ever improving about of educational information on Fubu out there. Go ahead and follow them on the Twitters.

If you have any suggestions or find bugs hit me on or Twitter or create a GitHub issue. If you are stuck please do send me a line.

The post Introducing Dovetail Bootstrap appeared first on Clarify Solutions | Dovetail Software.

Compositional Architecture FTW

Kevin Miller — Thu, 09 Jun 2011 01:15:14 +0000

While doing a deployment of our Dovetail Carrier product our customer was having connectivity issues with their email server. We ran into situation where we needed prove that the product could connect and simply firing up an email client was not as good as proving that our code worked. The compositional architecture of Carrier allowed me to quickly build a little console application to detect email configuration problems.

Recompose It

Plucking the necessary pieces of code out of our application and recomposing a utility to solve the problem at hand was easy. Here is the gist of the console code.

Writing this code did not take long.

Lean on the container

Allowing the container to compose my types for me frees me from having to worry about minutia. There are other things going on behind the scenes that I don’t have to worry about. For better output I pulled in our logging infrastructure. Way better than writing to standard out.

Validation?

Not really. A minute before I posted the console app to the customer figured out the problem was on their side. But had the mystery still been afoot we’d had a tool on our belt.

The post Compositional Architecture FTW appeared first on Clarify Solutions | Dovetail Software.

Dovetail Software is Multi-Core Aware

Kevin Miller — Tue, 12 Apr 2011 21:31:08 +0000

A customer recently asked if they should provision multiple CPUs on virtual machines hosting Dovetail applications. The short answer is Yes most of our applications do a pretty good job taking advantage of multiple CPU cores. Let’s take a look at this in a bit more detail.

Our Clarify products all use our .Net Dovetail SDK. The SDK is not itself multi-threaded (each thread runs on a CPU core) but most Dovetail products using the SDK utilize multiple cores in one way or another.

Web Applications

Most of our customers use one of our web applications. Dovetail Agent, Agent Lite, and Dovetail Mobile, and Dovetail Administration are implicitly multi-threaded because the web server (IIS) that they run on uses a thread per web request being processed.

Note: There is a limit to the number of threads IIS will put into play.

Message Based Applications

Dovetail Seeker and Dovetail Carrier are message based applications which run as Windows services. They both use multiple threads to produce and consume messages. This architecture does a very good job at parallelizing the work being done. For example Dovetail Seeker is a bit CPU intensive when indexing a large amount of content. We have seen a linear increase in the amount of content that can be indexed concurrently when adding additional cores. Here is a screenshot from my development machine doing some heavy Dovetail Seeker indexing.

Seeker’s usage is not purely CPU as there is still a bit of waiting that occurs while talking to the database and file system. It doesn’t hurt to have a fast database server and local file system where the search indexes get written.

Dovetail RuleManager

Dovetail RuleManager is very much like our message base applications. It is a Windows service and uses multiple threads. We wrote this application before our experience with message based architectures. If I did major rework this application would become message based. RuleManager uses a producer consumer multi-threading model where one thread polls the database for events feeding consumer threads work to do when events are found.

Conclusion

Should you use multiple CPUs for Dovetail applications? Yes. How many? I would recommend at least 2 cores for web applications. Our message based applications can in theory use as many CPUs as you throw at them.

The post Dovetail Software is Multi-Core Aware appeared first on Clarify Solutions | Dovetail Software.

Kevin Miller – Clarify Solutions | Dovetail Software

TikaOnDotnet now supports Tika 1.7

FubuMVC using ModifyChainAttributes

The Problem

Create a Modify Chain Attribute

More On Behaviors

FubuMVC Wisdom: Bottles and StructureMap Registries

Auto Import Referenced Bottle Registries

More On Bottles If you Like

TikaOnDotNet 1.4 Released as a Nuget

From Zero to Text Extraction

Step 1 – Install the Nuget

Step 2 – New up a TextExtractor

Finale – Compare the text with the original

Original

Extracted Text

Getting a ASP.Net request URL into a log4net context

Dovetail Bootstrap – Now With Service

Topshelf

Polling The Database

Model Maps

Wrap Up

FubuMVC Hijacking Behaviors

Scenario: Web Service API Exception Handling

Answer: Wrap the Behavior Chain

A Partial? What the fsck is a FubuPartial?

Scenario: Single Page Web App

Background

Enter TransferAPIRequestsTo Behavior

Conclusion

Dovetail SDK C# Helper Extensions

ClarifyGeneric Extensions

ClarifyDataRow Extensions

An example of these extensions in play

Automating Deployment For Clairfy Using Rake

Where is the Code?

Applying Schema Scripts (apply_schemascripts)

Applying SQL Scripts (apply_sql)

Ordering

My Quick Argument For Rake over Nant

Introducing Dovetail Bootstrap

How do I get it?

What Can Bootstrap Do?

Authentication

Case Listing

Dynamic History

Create Case

Application List Listing (try to say that fast)

What exactly is in Bootstrap?

Retrieving data using ModelMap

Web Application Framework

Wrap Up

Compositional Architecture FTW

Recompose It

Lean on the container

Validation?

Dovetail Software is Multi-Core Aware

Web Applications

Message Based Applications

Dovetail RuleManager

Conclusion