Learn dotCMS

Geolocation Viewtool

Chris Falzone — Thu, 11 Apr 2013 21:58:11 +0000

Geolocation – The Options and Issues

At Aquent, we have relied on being able to determine where our visitors are coming from for some time now. This allows us to target specific information to visitors based on where they are coming from. Getting that information is becoming easier and easier as there is no shortage of GeoIP Lookup Services available; from free services to paid, client-side and server-side, and in any programming language you could ask for. A couple of these we have used to varying degrees of success. The guys over at dotCMS have actually came up with a pretty nice example implementation on the 2.2 Demo Site:

  #set($clickstream  = $session.getAttribute("clickstream"))
  
  #set($geolocationFromSession = $session.getAttribute("geolocation"))
 
  #if(!$UtilMethods.isSet($geolocationFromSession))
    #set($locationURL = "http://www.geoplugin.net/json.gp?ip=$!clickstream.remoteAddress")
    #set($geolocation = $json.fetch("$!locationURL"))
    $session.setAttribute("geolocation", $geolocation)
  #else
    #set($geolocation = $session.getAttribute("geolocation"))
  #end

This code is using the JSON tool to access geoPlugin’s free service. It even goes as far as storing the data for the user’s session. At Aquent, we have used similar approaches. However, there is one slightly annoying issue with this approach that we ran into. What happens when geoPlugin’s service is not available? Unfortunately the JSON tool is not very resilient here. It holds on to the connection and will wait for it to timeout for a very long time. No amount of configuration seems to get it to fail-over quickly. On a busy site this quickly eats up HTTP connections and even has the potential to take your site down. Anyone who has worked with the JSON or XML tools to pull in 3rd-party resources into their dotCMS implementation knows exactly what I am talking about.

MaxMind Solution

What we wanted to do at Aquent was come up with a solution that does not depend on connections to an external site, has fast and accurate data that is easy to keep up to date, and is simple for our front-end developers to use. This is where MaxMind comes in. MaxMind provides several databases that you can download that are kept up to date weekly. Their Open Source Java API is lightning fast at looking up the data and very easy to use. They also supply a way to download the database files automatically. They offer both paid versions of the databases as well as free versions for public use (which aren’t updated quite as often, and will be a little less accurate as a result).

We decided to go with the GeoLite City Database for now, but plan on upgrading to the full version of the City Lookup Database in the near future. We decided to go the City Database so that we would be able to access a wide array of information about the visitor including:

Country Name/Code
Region
City Name
Postal Code (US only)
Latitude and Longitude
DMA, Area and Metro Codes (US only)

The only problem left to solve was to provide a way for our front-end developers to access the database in Velocity. The easiest way to accomplish this is a viewtool. You can pick up the viewtool over at our GitHub repo.

Writing the plugin

Writing the viewtool plugin was not very difficult. Here is quick overview of what we needed to do:

First, we need to compile the MaxMind Java API into a JAR that we could include in the plugin. There are some instructions in their readme file on how to do this and the resulting jar file goes into the lib folder of the plugin. Every plugin needs a MANIFEST.MF and build.xml which can be copied from the example configuration plugin.

Click Here if you want to skip the java and jump into using the viewtool. Now for the core code:

public class GeoIP implements ViewTool {
	private LookupService cityLookup;
	private boolean inited = false;
	PluginAPI pluginAPI = APILocator.getPluginAPI();

Every Viewtool must implement the ViewTool Interface. Then we start off by creating a LookupService.

	
	@Override
	public void init(Object initData) {

The init method of a viewtool gets executed when the viewtool is created. This can be different depending on what scope you set your viewtool as in your toolbox.xml. For this project, I want the tool to init once when dotCMS starts up. This is why I made the tool application scoped.

		String dbFileName = "";
		try {
			dbFileName = pluginAPI.loadProperty("com.aquent.plugins.geolocation", "maxmind.dbFileName");
		} catch (Exception e) {
			Logger.error(this,"Unable to load plugin property - maxmind.dbFileName", e);
			return;
		}
	
		String dbPath = "";
		if (UtilMethods.isSet(Config.getStringProperty("ASSET_REAL_PATH"))) {
			dbPath = Config.getStringProperty("ASSET_REAL_PATH") + File.separator + dbFileName;
		} else {
			dbPath = Config.CONTEXT.getRealPath(File.separator + Config.getStringProperty("ASSET_PATH") + File.separator + dbFileName);
		}

We decided it would be best to store the database file in the assets directory. This means if you are running in a clustered environment you only need to maintain the file in one place. We then use a property in the plugin.properties file to determine the filename of the database to load.

		try {
			cityLookup = new LookupService(dbPath, LookupService.GEOIP_MEMORY_CACHE | LookupService.GEOIP_CHECK_CACHE);
		} catch (Exception e) {
			Logger.error(this,"Unable to get a LookupService",e);
			return;
		}

Now we create the LookupService and point it at our db file. We decided to use their Memory Cache option for caching the DB to make the lookups faster. You could also use the Standard or index cache options. I am not sure at this point how large the full database is in memory so we may have to move to a different cache method in the future. You can read more about the options in MaxMind’s Java API readme.

		// A flag to let the viewtool know we are good to go
		inited = true;
	}

Finally if we got to this point and there were no errors then we know the viewtool is inited. I set a boolean flag so in my methods I know if the viewtool is inited or not.

The last part threw me off a bit. I thought since the Location object contained public properties I could just return the Location Object and those properties could be accessed in Velocity. So I tried this first:

	public Location getLocation(String ip) {
		Logger.debug(this, "MaxMind GeoIP Viewtool - getLocation called with ip "+ip);

		// Get the Location from the LookupService
		Location loc = null;
		if(inited) {
			loc = cityLookup.getLocation(ip);
		} else {
			Logger.info(this,"Attempt to Call getLocation and not inited");
		}

		return loc;	
	}

Unfortunately it seems that velocity cannot just access properties on an object without getters and setters. So I needed to provide a method that returned a map with all of the properties:

	public Map getLocationMap(String ip) {
		Logger.debug(this, "MaxMind GeoIP Viewtool - getLocationMap called with ip "+ip);

		// Get the Location from the LookupService
		Location loc = null;
		Map locMap = new HashMap();
		if(inited) {
			loc = cityLookup.getLocation(ip);
			locMap.put("countryCode", loc.countryCode);
			locMap.put("countryName", loc.countryName);
			locMap.put("region", loc.region);
			locMap.put("city", loc.city);
			locMap.put("postalCode", loc.postalCode);
			locMap.put("latitude", loc.latitude);
			locMap.put("longitude", loc.longitude);
			locMap.put("dma_code", loc.dma_code);
			locMap.put("area_code", loc.area_code);
			locMap.put("metro_code", loc.metro_code);
		} else {
			Logger.info(this,"Attempt to Call getLocationMap and not inited");
		}

		return locMap;
	}

We decided to keep both methods because the Location Object has a nice distance method to compute the distance between 2 locations. We even decided to include a few helper methods so that you can pass in a latitude and longitude and get the distance as well. I know our front-end developers will enjoy this as we have lat/long in some of our structures already.

The last thing our plugin needed was the toolbox mapping. Pretty simple, give it a key that will be used in velocity to map to the viewtool, a scope, and point it at your viewtool’s class. As I stated before, for this project it made sense to make this an application scoped viewtool so that we only need to do the heavy operation of loading the database into memory once.

Using the plugin

So you are probably wondering at this point how to use the viewtool. The first thing you will need to do is grab the source from GitHub. You can either clone the repository or download it in a tar or zip form. You are going to want to then extract or clone that thing into your dotCMS’s plugin directory.

Next you will need to obtain the database file from MaxMind. Remember that you will need the City Database for this as the Country database will not work. You can either purchase the full version or download the Lite Version for free. Once you have the zip file you will want to extract this file into your shared assets directory. Finally, don’t forget to change the filename in conf/plugin.properties to match what you are save the db file as.

Now you are ready to stop dotCMS, run bin/deploy-plugins.sh (or bat), and startup dotCMS. On startup if you are tailing the log file you should see a message like: “MaxMind GeoIP Viewtool Started”. Now let’s dig into some velocity code:

Getting the User’s IP address

You have a couple options here. Depending if you have a load balancer and how it sends the IP in the header one or more of the following options might work for you:

#set($ip = $request.getRemoteAddr())
#set($ip = $session.getAttribute("clickstream").remoteAddress)
#set($ip = $request.getHeader('x-forwarded-for'))
#set($ip = $request.getHeader('x-cluster-client-ip'))

## A trick to finding the right header
#foreach($key in $request.getHeaderNames) 
  $key = $request.getHeader($key) 

#end

Getting the User’s Location

Now that you have the user’s IP address you can get the LocationMap and access the various properties:

## We might as well take a hint from the dotCMS Folks and take advantage of caching the user's location in the session for later use
#set($locFromSession = $session.getAttribute("loc"))
 
#if(!$UtilMethods.isSet($locFromSession))
  #set($loc = $geoip.getLocationMap($ip))
  $session.setAttribute("loc", $loc)
#else
  #set($loc = $session.getAttribute("loc"))
#end

## Now you have access to the following data:
 
  IP:
  ${ip}
  ${loc}
  
  Country Code:
  ${loc.countryCode}
  Country Name:
  ${loc.countryName}
  Region:
  ${loc.region}
  City:
  ${loc.city}
  Postal Code:
  ${loc.postalCode}
  Latitutde:
  ${loc.latitude}
  Longitude:
  ${loc.longitude}
  DMA Code:
  ${loc.dma_code}
  Area Code:
  ${loc.area_code}
  Metro Code:
  ${loc.metro_code}

Computing Distance

Lastly, if you want to determine the distance between 2 locations you have a couple of options:

## First get a couple Location Objects or pull some latitude and longitude data from one of your structures:
#set($locObj  = $geoip.getLocation($ip))
#set($loc2    = $geoip.getLocationMap('213.52.50.8'))
#set($loc2Obj = $geoip.getLocation('213.52.50.8'))

## If you have two Location Objects you can get the distance using the Location Objects:
Distance between ${esc.d}loc and ${esc.d}loc2 = ${locObj.distance($loc2Obj)} miles

## If you have one Location Object and a set of lat,long for a second location you can get the distance using the viewtool helper method:
Distance between ${esc.d}loc and ${esc.d}loc2 = ${geoip.distance($locObj, $loc2.latitude, $loc2.longitude)} miles

## If you have two sets of lat,long you can use the viewtool helper method as well:
Distance between ${esc.d}loc and ${esc.d}loc2 = ${geoip.distance($loc.latitude, $loc.longitude, $loc2.latitude, $loc2.longitude)} miles

Keeping your database up-to-date

MaxMind also provides a C Library that you can use to keep your database up-to-date. Setting this up is fairly simple. They have a full set of instructions over on their site. Just don’t forget to pass the -d option to tell it to save your database file in your assets folder:

Usage: geoipupdate [-hv] [-f license_file] [-d custom directory]

Compatibility

Currently I have tested the code on 1.9.5.x and 2.2.1. In the future we plan on converting the plugin to OSGI, but we wanted to maintain compatibility with the 1.9.5 folks. Let us know either on GitHub or in a comment here if it useful for you or if you have feature requests or find any bugs.

Conditionalizing Structure Fields

Michael Fienen — Fri, 26 Oct 2012 03:31:29 +0000

Imagine a situation. You have a structure with a bunch of fields. But only certain fields are relevant at certain times, based on the answers to other fields. What’s an admin to do? Well, thanks to custom fields, a lot, actually. Using a custom field, we can include a little JavaScript that will allow you to make certain fields conditional.

Featured Field Configuration

Let’s start with a structure. In this case, the News article structure, since most people will be familiar with it. Let’s say you want to add some additional functionality like making something be ‘sticky’ in WordPress. And let’s say that you want that to be for just a specific range of time. So obviously these date fields would only be important if you’ve set the news item to be featured, otherwise they’re just taking up space. Step one, let’s add the field that will control this featured state. I’ll use a radio button, and a basic boolean value.

Next, add a couple date fields, nothing special here. We’ll call them “Featured Start” and “Featured End.” The only important part is that at this point, these can’t be required (though with some extra JavaScript, you could manage that too – client side). In the end, you’ll have something similar to what’s below.

All three fields

Now, let’s talk magic sauce. Add a custom field to the structure, and just drop it at the bottom (that’s probably where it is anyway after adding it). It’s not going to show anything, so it’s placement isn’t really important. We’ll call this field “Conditional Fields.” I’m going to do this with jQuery, because that’s what I happen to be better at. Hint: you can easily add jQuery to a custom field using my Resource Loader plugin. In the value for the custom field, drop the following code in.

It’s that simple. You can easily adapt this to other field types besides radio buttons, or toggle multiple sets of fields on a structure. If you want to make the fields required based on the on/off state, you can’t use the built in required setting on the field. Instead, you’ll have to add a click handler to the submit button and run your own validation for that. Otherwise, everything else like regex validation, default value, etc should work as normal. This is a great way to simplify your content forms and provide progressive discovery on fields so users don’t feel overwhelmed the first time they see a structure.

The Resource Loader Plugin (1.0)

Michael Fienen — Tue, 23 Oct 2012 21:48:53 +0000

After some discussions with a coworker, an idea took shape for a Velocity based alternative to framework loaders such as the JavaScript based RequireJS. The idea is that you can upload common or frequently used files for your templates to your dotCMS host or shared host and pull those en masse with a simple macro. Welcome to the Resource Loader (GitHub project page).

A Web Resource file asset

First, how it works (jump down the page if you want to see an explanation of how it’s built). When you install the plugin, it does two specific things. First, it creates a macro called loadResource(). Then, it creates a File Asset structure called Web Resource (this is configurable in the plugin.properties file, if you want to change it). From there, you upload files that you want to be loader enabled using that structure, and give them a common resource ID. So, let’s say you use Foundation as the basis for your sites. You can upload the core CSS and JS files for that, and give them all the resource ID ‘foundation.’ If you use the entire Foundation library, that’s at least a couple CSS files and several JS files. Then, in your template, all you have to do is run the code.

## CALL ALL OUR LOCALLY STORED FOUNDATION LIBRARY FILES
#loadResource('foundation')

Even better, out of the box the macro supports any JS library in the Google Hosted Libraries without needing to upload a local version at all. So, want to add jQuery on a fresh site? Just drop in:

## GET JQUERY FROM GOOGLE'S CDN IF IT ISN'T AVAILABLE LOCALLY
#loadResource('jquery')

If you prefer CDN based resources, when you upload your files, just include an external link to the CDN version and set it to default to that link.

There are several other options included as well, which you can just set as Velocity variable prior to calling the macro:

$htmlDebug (bool): Prints debug info out on each library as an HTML comment in the source.
$commentBreaks (bool): Show HTML comments before and after loading the library files. Good for seeing the start and end of each library, especially if you’re loading several in a row.
$resourceCache (int): Cache time for resources, in seconds. By default, once you call a resource, it will use dotcache() to save that for a day.
$boxJs(bool): Enables the 3rd party caching proxy/CDN BoxJS. This will take all the JS files for your resource library and combine and minify them, then serve it from their Amazon based CDN as a single file. Only works when used on local files.
- $boxJsMinify (bool): Enabled by default. Set to false to disable minification if it causes issues for your file.
- $boxJsCache (bool): Enabled by default. Set to false to disable caching on the BoxJS CDN.
- $boxJsVersion (str): Allows you to create versioned packages on their CDN if you ever want to roll back for something.
- $boxJsAppend (bool): Enabled by default. Set to false to spit out the JS right where you place the macro (otherwise it’s added to the end of the page).
- $boxJsDefer (bool): Set to true if you want manual control over when, where, or how your JS is brought into the page. Requires you to manually call box.get(); in JavaScript on the page where you want the code at.
$boxCss(bool): Like BoxJS above, this enables the 3rd party caching proxy/CDN BoxCSS. This will take all the CSS files for your resource library and combine and minify them, then serve it from their Amazon based CDN as a single file. Only works when used on local files.
- $boxJsMinify (bool): Enabled by default. Set to false to disable minification if it causes issues for your file.
- $boxJsCache (bool): Enabled by default. Set to false to disable caching on the BoxCSS CDN.
- $boxJsVersion (int)Allows you to create versioned packages on their CDN if you ever want to roll back for something.

So knowing all this, a more advanced example might be:

## ENABLE DEBUGGING
#set($htmlDebug = true)
## DISABLE COMMENT BREAKS
#set($commentBreaks = false)
## TURN DOWN CACHE TIME TO ONE HOUR
#set($resourceCache = 3600)
## RUN ALL OUR CSS THROUGH BOXCSS
#set($boxCss = true)
## PULL IN OUR CUSTOM CSS LIBRARIES FOR THE SITE
#loadResource('all-the-css')

That would be a good way, if you had a dozen common CSS files you use, to create a common shared library that gets combined and cached for an hour so that the end user ends up with just a single HTTP request for all twelve files.

How It’s Built

For those of you interested in doing something like this, the basic setup for the plugin is pretty easy. The hardest part is deploying a structure programatically, and even that’s pretty easy once you’ve seen the code. There are four files specifically you need to pay attention to for the most part on this:

MANIFEST.MF
conf/plugin.properties
conf/macros-ext.vm
src/com/learndotcms/plugins/resourceloader/ResourceLoaderPluginDeployer.java

The only thing to remember for the MANIFEST.MF file is line two, where you need to define the deployer class that needs to run when the plugin is installed. In this case, the class that builds our Web Resource structure.

Manifest-Version: 1.0
Deploy-Class: com.learndotcms.plugins.resourceloader.ResourceLoaderPluginDeployer
Plugin-Name: Resource Loader Plugin
Plugin-Version: 1.0.121022
Author: Michael Fienen ( fienen@gmail.com - http://learndotcms.com/ )

For our properties, we added three user configurable values, for the shared host ID, and the structure name and Velocity variable name.

##The default for the reload.force is true.  If it is set to false the properties will only load once from the file.
##After that they will be stored and maintained in the database.  If true every time the server restart the properties
##will be cleared and reloaded.
reload.force = true

## Set the Structure and Velocity variable name for the resource loader's File Asset structure (DO NOT LEAVE BLANK)
webResourceName = Web Resource
webResourceVarName = WebResource

## Set the identifier for your shared host if used for sharing common elements between hosts (Uncomment it and give it a value if needed)
## sharedHostId =

Now, due to its size, I’m not going to source the entire macros-ext.vm file here. Instead, you’re welcome to download it or view it on GitHub. But, the basic structure holds true for it as with all macros. Also, I take into account the shared host ID if necessary. That uses a special viewtool ($pluginapitool) to get the property value from the plugin, which you can see on line 5 below.

## BASIC MACRO STRUCTURE
## BLAH, BLAH, BLAH, COMMENTS EXPLAINING MACRO AND USAGE
#macro(loadResource $library)
    ## SET UP THE SHARED HOST FOR THE FILE QUERY, IF PROVIDED
    #if($UtilMethods.isSet($pluginapitool.loadProperty("com.learndotcms.plugins.resourceloader","sharedHostId")))
        #set($sharedHost = " conhost:${pluginapitool.loadProperty('com.learndotcms.plugins.resourceloader','sharedHostId')}")
    #end

    ...do stuff using the passed variable $library when someone calls #loadResource('foobar')...
#end

Lastly, the deployer class. Like with the macro, I’m just going to sample this for you. The full code is here.

// This is all the overhead and such for this particular plugin
package com.learndotcms.plugins.resourceloader;

import com.dotmarketing.cache.FieldsCache;
import com.dotmarketing.business.DotStateException;
import com.dotmarketing.exception.DotDataException;
import com.dotmarketing.exception.DotRuntimeException;
import com.dotmarketing.plugin.business.PluginAPI;
import com.dotmarketing.plugin.business.PluginAPIImpl;
import com.dotmarketing.plugin.PluginDeployer;
import com.dotmarketing.portlets.structure.model.Field;
import com.dotmarketing.portlets.structure.model.Field.DataType;
import com.dotmarketing.portlets.structure.model.Field.FieldType;
import com.dotmarketing.portlets.structure.model.Structure;
import com.dotmarketing.portlets.structure.factories.FieldFactory;
import com.dotmarketing.portlets.structure.factories.StructureFactory;
import com.dotmarketing.util.Logger;

// Set up the deployer class name
public class ResourceLoaderPluginDeployer implements PluginDeployer {
  // This is what we do when the plugin is deployed
  public boolean deploy() {
    // Set up our Web Resource Structure. Note we're using the pluginAPI again for the property value.
    String structureName = pluginAPI.loadProperty("com.learndotcms.plugins.resourceloader", "webResourceName");
    String structureVarName = pluginAPI.loadProperty("com.learndotcms.plugins.resourceloader", "webResourceVarName");
    Structure st = new Structure();
    st.setName(structureName);
    st.setDescription("Common CSS/JS/VTL Files used for sharing and building templates.");
    st.setFixed(false);
    st.setSystem(false);
    st.setStructureType(Structure.STRUCTURE_TYPE_FILEASSET);
    st.setVelocityVarName(structureVarName);
	try {
		StructureFactory.saveStructure(st);
	} catch (Exception e) {
		Logger.error(this.getClass(), e.getMessage(), e);
		throw new DotRuntimeException(e.getMessage(), e);
	}

	Field field1 = new Field("Host or Folder", Field.FieldType.HOST_OR_FOLDER, Field.DataType.TEXT, st, true, false, true, 1, "", "", "", true, false, true);
	field1.setVelocityVarName("hostFolder");
	try {
		FieldFactory.saveField(field1);
	} catch (Exception e) {
		Logger.error(this.getClass(), e.getMessage(), e);
		throw new DotRuntimeException(e.getMessage(), e);
	}

	Field field2 = new Field("File Asset", Field.FieldType.BINARY, Field.DataType.BINARY, st, true, false, false, 2, "", "", "", true, false, false);
	field2.setVelocityVarName("fileAsset");
	try {
		FieldFactory.saveField(field2);
	} catch (Exception e) {
		Logger.error(this.getClass(), e.getMessage(), e);
		throw new DotRuntimeException(e.getMessage(), e);
	}

        // And we go on like this for the other eleven fields in the structure

        // Clear the field cache in dotCMS, just in case
	FieldsCache.clearCache();

    return true;
  }

  // We don't need to do anything when we redeploy
  public boolean redeploy(String version) {
    return true;
  }
}

That’s the basic story here. All it needs is to be uploaded and deployed then, and you’re off and running. You can deploy any of the four core structure types (content, widget, form, file) this way, and include any of the standard field types.

Commonly Creative Code

Michael Fienen — Thu, 18 Oct 2012 17:59:21 +0000

For those of you that attended dotCMS Boot Camp this year, you may have caught my presentation Creatively Common, looking at how to build your own light weight version of Flickr in dotCMS (or any kind of asset repository, really). As promised, I wanted to release that code to everyone for your own use or sharing. You’ll also find the slides at the bottom of this post, and I’ll link over to the audio once dotCMS has made that available from the conference.

[ You can download the code here. ]

There are three pieces to the zip file:

A starter.zip file you can use locally on a fresh install of dotCMS that is a copy of the default Quest Financial site, but including the built out library in the /library directory. This starter has only been tested on version 2.1.1.
The /widget code directory contains just the Velocity and HTML from the three core widgets used on the index and detail page for library
The license: all the widget code is released under a CC by 3.0 license.

Please take the code and use it to help you towards your own goals. See if you can find better ways to implement features. Think about ways to extend it: better meta data, different data types, permissioned submission, galleries, etc. There are a lot of ways this could be made to be much more than just a fancy photo gallery. Feel free to leave me any questions in the comments.

Creatively Common (#BootCamp12) from Michael Fienen

We’re migrating to GitHub!

Michael Fienen — Sat, 02 Jun 2012 19:01:49 +0000

Just a fast update today. As we move forward and start looking at current and future plugin opportunities, we’re going to take a cue from the core dotCMS team and start tracking things on GitHub. This will help us manage the coding, and also give you all the opportunity to help direct features, bugs, etc, and even contribute if you so desire.

To start things off, our IRC Support Portlet plugin (which we wrote about here) has been successfully set up and tagged in GitHub. You can follow it here. If you haven’t tried it out, download it and check it out. If you think we can make it better, feel free to log an issue on the project and we’ll check it out!

Hopefully you’ll find this a highly beneficial move, and we look forward to your feedback!

10 Things To See in dotCMS 2.0

Michael Fienen — Tue, 24 Apr 2012 16:16:00 +0000

The public release beta of dotCMS 2.0 has been out since March. Prior to that, the current stable version, 1.9.5, was released. This month, they also made the long anticipated move away from traditional SVN over to Github. Needless to say, 2012 has been busy, and the 2.0 release has been a long time coming. With that said, if you’re looking at dotCMS for the first time or wondering if it’s worth upgrading from 1.9.5, I wanted to look at a few of the features coming in 2.0 that you’ll likely want to note. These are some of the bigger singular items, but there are countless other improvements, bugfixes, and tweaks as well that cumulatively are just as important.

So, in no particular order, we’ll start with the…

1:) Multifile Uploader

Multifile Upload Popup

This is a personal victory for me. Not because I had anything to do with it, I’ve just been griping about the multifile upload tool since version 1.5. Prior to 2.0, users were limited to uploading only up to 19 files at a time (through the browser – not if you had WebDav access), and even then, it was through a form with 19 input fields. It was cludgy, slow, and not at all user friendly. This is a big step forward to much more efficiently facilitate the upload of large batches of files.

I’d like to see this tool continue to evolve, for instance emulating other systems’ ability to do things like explode a zip file and selectively upload the contents. Maybe show image thumbnails, and let you edit metadata prior to saving. A progress bar or inline visual uploading (a la Flickr) would be nice for general UX. A lot of ways you could go, but this is a big step forward in the area of file management, and I’ll take it.

2:) Files on Root

Admittedly, this is a pretty simple addition, but for everyone that’s ever created vanity URLs or URLRewriteFilter rules to handle things like favicons or robot.txt files, this will be a welcome addition. Just like it sounds, you can now have files that live at the root level of a host. There are always reasons for this, whether the aforementioned, or other things like Google Webmaster Services, that allow you to authenticate ownership of a domain by placing a simple HTML file on your server.

3:) Key/Value Pair Field

Key/Value Pair Content Field

If you’ve ever used Custom Fields in WordPress, this new type of structure field should be very familiar to you. The key/value pair field is going to provide a great way to add custom additional data to contentlets that may not always warrant an entire field, or in cases where you don’t need to care as much about the UI as the data. Or, use it as a way to include debug data or test fields with a contentlet while you develop, without needing to expose your day-to-day users to additional interface weight. Better still, if you’re an e-commerce site or someone that has to deal with products or items that have varying specs, you can use this type of field to help easily build out things like feature tables.

One other use that comes to mind that has varying usefulness depending on your organizational structure is this: if you’re in a situation where you would benefit from being able to give certain people access to sets of data that might get repurposed and reused across the site, but you don’t want to convince them to edit Velocity variables, you could set up a “Dataset” structure with a title and key/value pair field that they can populate. Then, you pull the appropriate data set in to pages across your site (financial aid info, user demographics, sports statistics, etc).

I’ll look forward to possible improvements in the future, like inline editing and autosuggest for keys and values. All in all, this is a pretty raw way of dealing with data on a contentlet, but really, sometimes that’s just what you want.

4:) Files as Content

Setting up a new File content structure

Files as content is another natural evolution of one of the internal cogs of the system, and it won’t be the last. In 1.9, hosts and forms became content. At some point beyond 2.0, I suspect we’ll see users and pages as content as well. The content engine is really the heartbeat of dotCMS. The evolution we see is taking something that used to be rigid and static in the system previously, and allowing you to extend it as a structure. This dramatically improves dotCMS’s ability to serve as a lightweight DAM solution, as well as CMS. At the same time, it can give you access to additional file metadata you wouldn’t have otherwise when you include or link to it (besides what is now extracted automatically from files and indexed along with it, like image EXIF data). Now, you can pull and query for files in dotCMS just like content.

Don’t be fooled, this is a pretty significant step forward for asset storage in dotCMS. Just the fact that now meta data is extracted as key/value pairs by itself is pretty important (imagine the possibilities for file repositories, photo galleries, etc).

5:) ElasticSearch

ElasticSearch is a somewhat more invisible change for a lot of folks, but it’s important nonetheless. Why invisible? Because it sits on top of the Lucene engine, but in such a way as to be able to wrap the old queries (similar to how the move from 1.7 to 1.9 went with the newer syntax, and 1.9 just magically let the old syntax work). So, the syntax itself doesn’t need to change for any of your content pulls and queries. Where this becomes an important change though is for larger users, especially in clusters. Lucene by itself was, frankly, not awesome at replicating the index across several hosts. ElasticSearch is designed to handle that exact issue, providing excellent scaling across multiple nodes, not just a single node with more resources. It’s fast, lightweight, and error resistant.

What I really want to look in to though is that while ElasticSearch supports traditional Lucene syntax, it also supports JSON based queries as well. I’m not sure if this is exposed in 2.0, but the implications are pretty big, including the possibility for regular expression based queries, result scoring, and more. While it may not be readily accessible in dotCMS 2.0, it is a part of ElasticSearch, so it’s conceivable that we’d see it exposed in a future release.

6:) Custom Workflows

Default Workflows in dotCMS 2.0

Workflows have always been a feature I’ve scoffed at. This is mainly because I see them as a feature that is more for marketing than actual use. It demos well, as I say. Ultimately, I’m not sure I’ve ever seen an organization implement a workflow within a CMS particularly well or consistently, whether in dotCMS or any other system. That said, the addition of custom workflows (for enterprise users) is a vast improvement over the prior implementation. Pre-2.0, workflows were little more than a system of tag for users – sending rough messages back and forth, possibly with a link to what you were supposed to do. Nothing else, and nothing about it was binding within the business logic.

Now, there is an actual rule system that allows you to create a process through which a contentlet or item must travel, and it’s tied to this workflow logic so that you can enforce the process. But, this is by no means a particularly simple system to set up, as the power brought with it the tradeoff of the complexity necessary to create all the rules and conditions content can go through at different steps.

Is it really a good enough improvement that I think people will flock to it and use them across their sites? No, probably not. That’s not because it’s a bad feature, but rather the result of the problem of meshing programmatic workflows with human workflows. But still, you should know what it does and how to leverage it, as there is a lot of upside if you can make it work within your people infrastructure. Even if it’s just forcing something like a proofreading step by a content strategist.

7:) System Host

Like being able to store files on the root of a host, this is a seemingly simple thing that just makes life happier for everyone. If you host multiple websites in one instance of dotCMS, you’ve no doubt played with how content lives on different hosts to silo it. Other folks have ran in to issues where they want to share things like VTL files across domains (which was possible pre-2.0 by doing something like a #dotParse('//someotherhost.com/common/vtl/shared.vtl')). But obviously this meant that your VTL had to live on one of your hosts, and that if you needed a shared resource pool, you had to permission accordingly – even if it didn’t make sense on the host where the pool was. Normally, where you saw the SYSTEM_HOST was in content queries. In 2.0 though, it’s an actual thing. This makes it a great tool for building a resource repository for all your other sites. You access it like any other host from the Website Browser – just click on the “Change Host” dropdown and select “shared.”

8:) Content Locking

Locking panel and content actions.

Prior to dotCMS 2.0, opening content for editing caused the content to lock. The problem was, this was little more than window dressing, as a single click allowed you to override a lock, regardless of who you were. So really, locked content was just more of a warning, than a lock. The other problem was that users could easily click in to a piece of content, but use the back button instead of the cancel button, thereby leaving a lock in place for days, weeks, or longer.

Now, you cannot edit a piece of content without taking the next step of locking the contentlet. You must do this as an actual action. Then, once locked, it is truly locked. Another using stumbling across it can’t accidentally open it while you have it open and edit and save it if you have it locked.

It’s all about helping to maintain integrity of the data in environments where multiple people might try to edit the same thing. While I think this is a good improvement, I hope the UI is improved. I can’t tell you how many times in testing I’ve clicked right on the content form out of habit expecting to update something, but instead get the red flash of the “Lock for Editing” button. I’d like to see the content form not look like a form when it isn’t locked (e.g. no borders on text fields, or a red shade to fields, or a light gray overlay – something that provides immediate, obvious visual feedback so you don’t forget). Sort of like the new category selector from 1.9, if you’re upgrading you can expect this feature to take some getting used to while users new to dotCMS in 2.0 will probably take it in stride. Personally, I don’t mind the trade off for the extra reinforcement within the data model.

9:) Spring MVC

I won’t dive too deeply into this, mainly because Will over at dotCMS has already done a good job explaining it. But the gist of it is that it allows you to create functional plugins with only the use of a single class that gets mapped to a page in the /application/spring path on your site. This is somewhat similar to what you may have chosen to do in the past with Struts, but with less overhead.

Ultimately, this is just another option for folks that want to extend dotCMS. It’s somewhat less intimidating that Struts, and needs less configuration, which is nice. However, I’ve always felt one of the biggest barriers to the community extending the system wasn’t the framework, but simple availability of open source Java devs interested in helping out a project like this. Spring has the advantage of being easy in the right places though, which may spur some folks to start taking a serious look at new plugins.

10:) Social Integration

Twitter update box from asset.

I said this was a list of ten things. Really it’s nine plus this bonus. I almost don’t want to count this because it’s a toy, in a way. When editing assets in 2.0, you’ll notice in the action sidebar where you lock, save, and publish assets, there’s now an option to “Tweet This!” It’ll generate a bit.ly link for you, and allow you to announce the publishing of your asset. The bit.ly link isn’t automatically inserted in the text field, which is a little frustrating, and it doesn’t appear to support broadcasting from multiple Twitter accounts a la Hootsuite. It’s just a simple addition to the system that is nice to see, if only to know that some thought is being given to how content in dotCMS is being spread, and how that may be facilitated as time goes on. There’s a lot of room for improvement and expansion on something like this: additional services (Facebook, Tumblr, LinkedIn, Google+, etc), using it on other elements (for instance, why isn’t this on pages in Edit Mode?), and maybe even serve as a way of getting some lightweight social analytics for the item you’re looking at (click tracking, RTs, etc).

So there you have it, ten things in dotCMS 2.0 that will likely make a few waves. Keep track of JIRA and now the Github issue tracker page for all the details regarding changes in the system, because there’s still a lot more that I haven’t talked about. Also, if you’re upgrading, make sure to take some time testing before launching live, as several of these new features will involve a lot of changes on the data model side of things. Plus, now that it’s on Github, be sure to take some time to contribute pull requests for things that you might be able to fix or improve. I think overall that’s a good move for them, as it exposes the platform to the overall open source community in a new way that SVN simply wasn’t capable of.

Phone Number Formatting Macro

Michael Fienen — Fri, 20 Apr 2012 04:14:01 +0000

If you’ve ever had multiple people entering phone numbers into a field, you’ve no doubt discovered the varied formats they can provide. Perhaps you’re already enforcing field validation on it, which is a pretty good solution for the most part. But also, maybe you need to reformat that number for a page or get old data to look the same.

Rather than building out a viewtool to handle it, which you then need to compile and deploy in a plugin, you can use something like this macro example. Just pass it a phone number string (such as “1-555-555-1234″ or “555.555.1234,” it can handle different punctuation – that’s the point) and it does the rest. It defaults to US numbers, but you could easily change that or pass a different $phoneCountry value. If you want to include the country code in the number, just set $addCountryCode to true before using the macro (it will stick until set otherwise).

## ###############################################################################
## Macro Name: Phone Number Normalizer
## Version: 1.0
## Author: Michael Fienen
## Email: fienen@gmail.com
## Description: Takes a phone number passed to it and attempts to parse it into
## a standard format. Can be used to supplement field validation on a structure.
##
## Required Parameters:
## - $phoneNum = Phone number you want to normalize (string)
##
## Optional Parameters:
## - $phoneCountry = Pass a phone number country code. Only needs to be
## set once unless country changes for different cases. Default: 1 (string)
## - $addCountryCode = Add the $phoneCountry code to the number. Default:
## false (boolean)
## ###############################################################################
#macro(normalizePhone $phoneNum)
   ## SET OPTIONAL VALUE DEFAULTS
   #if(!$UtilMethods.isSet($phoneCountry))
       #set($phoneCountry = '1')
   #end
   #if(!$UtilMethods.isSet($addCountryCode))
       #set($addCountryCode = false)
   #end

   ## CREATE THE INTERNATIONAL IDENTIFER HOLDER
   #if($addCountryCode)
       #set($countryCode = "${phoneCountry} ")
   #end

   #if($phoneCountry == '1')
       ## REMOVE ALL THE PUNCTUATION WE TEND TO SEE IN NUMBERS
       #set($strippedPhone = $phoneNum.replaceAll('\.|-|\(|\)|\s|\+',''))
       ## REMOVE LEADING +1, IF THERE
       #if($strippedPhone.length() == 11 && $strippedPhone.substring(0,1) == "1")
           #set($strippedPhone = $strippedPhone.substring(1))
       #end
       ## INCLUDE A FAILSAFE JUST IN CASE SOMETHING WENT SCREWY WITH THE NUMBER
       #if($strippedPhone.length() == 10)
           #set($normPhone = "$!{countryCode}(${strippedPhone.substring(0,3)}) ${strippedPhone.substring(3,6)}-${strippedPhone.substring(6)}")
       #else
           #set($normPhone = $phoneNum)
       #end
   ## ADDITIONAL COUNTRY FORMATS CAN FOLLOW HERE
   #else
       #set($normPhone = $phoneNum)
   #end

   ## OUTPUT THE NUMBER
   $!{normPhone}

   ## RESET OPTIONAL VALUES
   #set($countryCode = '')
#end

This will output the number directly, so you’d use the macro inline where you want the number. For example:

#set($addCountryCode = true)
#if($URLMapContent)

    
        Address: $!{URLMapContent.address}
        Phone: #normalizePhone($URLMapContent.phone)
    
  
#end

This particular case is set up to work with US phone numbers, but starting on line 47 you could easily add in additional country codes and patterns. Just use the US entry as an example. I would also highly recommend adding field validation to your structure field if you’re concerned about the general integrity of the data. Feel free to use and modify this code to fit your needs.

Photo Credit: Some rights reserved by herzogbr

Creating a Color Picker Field

Michael Fienen — Thu, 12 Apr 2012 01:29:28 +0000

As we rapidly approach the launch of dotCMS 2.0, we’ve seen the addition of a number of new structure fields to the system since its launch. And there’s still more to come too – personally, I see a lot of potential in the generic Key/Value Pair field, which is very similar to custom fields in WordPress. However, you’ll almost certainly run into situations where you need a field that isn’t available. That’s where custom fields can be very helpful. In this tutorial, I’ll show you how to add a color picker field to a structure. Why a color picker? Why not? It’s a fast and easy way to allow customization of a theme, template or tool in the system, and can be helpful when trying to give users more options when they may not know how to find hex values.

Before I start, I should mention that Dojo has a color picker widget who’s library is included in dotCMS. But see, there’s two issues. One – I know jQuery better, so that’s what I’m using in this example, so just be aware that there is the extra script to load as a result. Two – I did try the Dojo color picker out, and got it mostly working, but the documentation for it isn’t very good and I simply couldn’t handle some of the use cases that I wanted to properly. So, jQuery all the way, and specifically, I’m using this color picker plugin. If anyone does it with Dojo and gets it working the same as here, I’d love to see it, since I’d prefer not load the extra JavaScript for jQuery if it’s avoidable.

Colorpicker Custom Field

In theory, by this point you caught that we’re going to use a custom field for this, so on your structure, go ahead and set one up. In this case, we’ll call it “Background Color.” We’ll then pull in jQuery, run noConflict() so that it won’t collide with Dojo in the back end, include the JS and CSS for the color picker, and apply it to a div. Additionally, you’ll see in the code I include a field called colorPickerField. This is so that you can manually control the color – and more specifically – unset the color if you don’t want it. Most of the color pickers don’t accept a null value for color, so if you rely just on the picker, then you will always have a value even if you don’t want one. If that’s not necessary in your case, by all means leave it out.

Since this is fairly straightforward, here’s all the code in one drop. The comments should pretty clearly explain what everything is doing. I also readily admit I’m not the world’s most awesome JavaScript ninja, so there may be some better ways to handle some of the stuff below. I’ll update code as people make recommendations.

Top: Plain, unmodified usage. Bottom: Modified CSS to improve UI.

A couple other notes. I tweaked the CSS a little here to adjust things so that it made more sense in the context of our fields. So, you’ll note stuff like on lines 4 and 5 where I add some spacing and hide a couple tools. Also, in the screenshot up above the code, you’ll notice my colorpicker is light, while the normal one is dark. In the download, there’s a custom_background.png image that’s for lighter themes that you can switch to. I’ve also added in extra validation on our colorPickerField input so that we don’t send invalid color values to the picker.

Usage is simple. Either type in the hex value in the text box, which will update the color picker accordingly, or select a color in the picker, and it will set the value in the text box automatically. In this case, the extra validation will cause the text box to take on a red background if you type in a value that isn’t a valid hex color, and it won’t update the picker until you type one in correctly (I also apply the same regex to the field validation on the structure entry as well). In the screenshot to the right, you can see I even adjusted the CSS further so that unlike the example above, my text box shows up inside the color picker, so it all feels like one nice, concise tool.

Hopefully this will give you some insight into just how easy it is to create unique types of custom fields that don’t already exist within dotCMS. If you have any questions, feel free to leave a comment below.

Photo credit: Some rights reserved by PhOtOnQuAnTiQuE

dotCMS and Mobile Techniques: Notes from Boot Camp 2011

Michael Fienen — Thu, 18 Aug 2011 15:56:36 +0000

At dotCMS Boot Camp 2011, I was invited to come and talk about mobile implementation techniques using dotCMS. Luckily, this isn’t as much as issue with trying to pull off clever dotCMS tricks as it is just knowing good mobile techniques. Below are a number of links and examples from my talk. This isn’t so much about a full blown tutorial, as much as it is just a quick reference for those that attended my session. A more complete tutorial will come at a later date.

Mobile Detection Using the Clickstream

By accessing the user’s clickstream data, and pulling out the isMobile() flag, we can create a conditional environment to do redirects, apply different stylesheets, change layout, etc.

## GET CLICKSTREAM SO WE CAN CHECK OUT THE DEVICE
#set($clickstream  = $session.getAttribute('clickstream'))
## IS THIS A MOBILE DEVICE?
#if($clickstream.mobileDevice && $clickstream.mobileDevice == 'true')
  ## GET THE REDIRECT COOKIE
  #set($goToMobile = $cookietool.get('goToMobile').value)
  #if($goToMobile == 'true')
    ## SEND THE USER TO THE MOBILE SITE, THIS MUST BE DONE BEFORE
    ## ANY DATA IS SENT TO THE BROWSER
    #$response.sendRedirect('http://m.yoursite.com/')
  #else
    ## IF IT'S MOBILE, BUT THEY WANT THE NORMAL SITE, SEE IF IT
    ## SHOULD BE DISPLAYED LIKE MOBILE OR LIKE NORMAL
    #set($displayMobile = $cookietool.get('displayMobile').value)
  #end
#end

By setting a cookie at the same time as the check, you can prevent users from having to go through the selection each time if they prefer the normal site over the mobile site. You could set the cookie with some simple JavaScript.

Creating a Mobile Template

Creating a mobile template is no different from any other template. If you’re using a framework like jQuery Mobile, you can build all the necessary parts right into the template.

## Container: General Template Includes
#parseContainer('e97bda22-33e8-4b9d-9508-0aefb4c899d6')



  
  
  
  

  $!{pageTitle}

  
  
  
  

  
  




  
    $!{pageTitle}
  


  
    ##Body Container 1
    #parseContainer('93cf710e-fbbb-4d32-a5fb-974dcac38c96')
  


  
    Copyright ©2011

jQuery Mobile and the #navigation() Macro

If you want to use the #navigation() macro and take advantage of jQuery Mobile tooling, you just need to add a little extra JavaScript into the mix to ensure the list is picked up and styled correctly. A single line of JavaScript will get you rolling.


  #navigation(1 1)

By using the .attr() method, we can add in any of the data-roles or other attributes that can be used to control how the list navigation looks and behaves as a result of jQuery Mobile. (Note: as seen in the above example, when writing code inline on the page and referencing ID’s, I replace all #s with ${esc.h}. This is the Velocity tool for printing out a hash, and is good for preventing accidental collisions in normal velocity code.)

Building a News Page

In this example, I pull in a list of recent news headlines and spit them out readily formatted for my jQuery Mobile stylesheet (Note: My code uses 1.7 syntax. However the principle remains unchanged for 1.9.)

#set($lastArticleDate = '0')
#set($today = $date.format('MM/dd/yyyy HH:mm:ss',$date.getDate()))
## FILTER DOWN TO A SPECIFIC SET OF NEWS CATEGORIES
#set($newsCatList = $categories.getChildrenCategoriesByKey('sections'))
#set($newsCats = "+(")
#foreach($cat in $newsCatList)
  #set($newsCats = "${newsCats}c${cat.inode}c:on ")
#end
#set($newsCats = "${newsCats})")
#set($query = "+structureInode:5 ${newsCats} +date1:[1/1/2010 00:00:00 TO ${today}] +languageId:1* +deleted:false +live:true")
#pullContent("${query}" '10' "date1 desc")


  The latest campus news from public relations.



#foreach($article in $list)
  ## SET DATE FOR COMPARISON
  #set($thisArticleDate = $date.format('EEEE, MMMM d, yyyy',$article.NewsPublishDate))
  ## IF CURRENT DATE IS DIFFERENT THAN THE LAST ARTICLE, PRINT OUT A DATE DIVIDER
  #if($thisArticleDate != $lastArticleDate)
	${thisArticleDate}
  #end
	${article.NewsHeadline}
  #set($lastArticleDate = $date.format('EEEE, MMMM d, yyyy',$article.NewsPublishDate))
#end

Links, Tools, and Resources

I mentioned several sites and tools when talking about mobile, here are many of them:

jQuery Mobile – A JavaScript framework that extends jQuery and helps add a number of mobile features to your site.
PhoneGap – Utility for compiling a web app into a wrapper so that you can deploy it as a native app for a number of mobile platforms.
Android SDK – Free SDK for Android that comes with a mobile phone simulator, useful for testing.
CSS Media Queries & Using Available Space – Guide to how to implement CSS media queries for changing CSS rules based on display conditions reported by the browser.
jQuery Mobile Google Maps Examples – Nice list of ways to build on jQuery Mobile with Google Maps (i.e. pulling content from a buildings structure to populate a map)

Legacy Form Submission Security Patch

Michael Fienen — Mon, 27 Jun 2011 13:00:35 +0000

A while back, it was exposed that if using the legacy form submission struts path (e.g. /sendEmail, /submitWebForm, etc), a spammer could remotely invoke the tool and use it to send spam emails to people. While this wasn’t exactly a bug in the strictest sense, it could prove to be a headache for users should their server be targeted.

Enter costarica.com. Thanks to their effort, a patch has been made available (for 1.9) that will force the server check the referring HTTP request against a list of approved server names and IPs. As a result, attempts to access the tool directly will be denied – the tool has to be invoked by a request originating from your site. This has been set up as a plugin for easy installation.

Instructions:

Edit conf/plugin.properties to include your list of approved hostname (you’ll have to do this each time you add or remove hosts in dotCMS)
Copy the plugin to the plugins/ folder of your dotCMS installation
Shut down dotCMS
Do an ant clean-plugins deploy-plugins
Start up dotCMS

[ DOWNLOAD Legacy Form Submission Security Patch 1.0 for dotCMS 1.9 ]