boulder dan

Thread safety with no threads in sight

noreply@blogger.com (Dan) — Sun, 06 May 2012 15:16:00 +0000

When people discuss thread safety in code it is almost entirely within the context of threading in a single process. The problems that come up are races conditions, data races, and unseen updates with solutions like locking, synchronized updates and immutability.

This definition of thread safety is too focused. Much of the problems normally attributed to thread safety can be seen in programs that have no access to OS threads or are running on multiple servers.

My primary work is in Ruby, where we normally labor under the illusion of thread safety by primarily deploying to a single threaded VM. The problem is that often this single thread hinders scaling. The standard solution is to take long running tasks and allow them to finish asynchronous in a queued process.

The main web process and the backgrounded process don't share any data, so thread safety in the traditional sense isn't an issue, but if you step back to look at the application as a whole they *do* share a database. This database is often mutable and is therefore a big source of thread safety bugs. Data races are the most obvious, but lost updates and stale reads also need to be considered.

How do you fix these problems? With the same concepts you would at the application layer but translated to whatever database you have.

If you're using an ACID compliant RDBMS then you're probably used to transactions which correspond to locking and mutex constructs in most programming languages. You should also consider improving your schema so that the database can enforce as much of your business logic as possible. This way your database can reject data that would create an inconsistent state even if your application deems it okay (remember your application won't know about the other processes running simultaneously).

To pick on Rails for a moment: most application built on Rails have their database constraints in the application. This creates tough scenarios like the following.

1. user = User.find_by_email("danielhopkins@gmail.com")
2. if !user
3. User.create!(:email => "danielhopkins@gmail.com")
4. end

An execution might look like this:
Process #1 | Process #2
|
ln 1 user = nil |
ln 2 true | ln 1 user = nil
ln 3 create! | ln 2 true
ln 4 done. | ln 3 create!
| ln 4 done.

A worst case scenario of process execution has created identical users. Even with no threads involved we have a clear threading bug.

Of course there are ways to prevent this from happening in Rails, this is illustrating how threading issues can manifest in a single threaded application. The same thing would apply to Java or any of the functional languages that pride themselves on their threading capabilities.

Remember that thread safety doesn't always involve threads.

It's a broader application problem that requires diligent understanding of how your application interacts with shared state. None of us are immune; especially in the world of internet applications.

Epic google error message fail

noreply@blogger.com (Dan) — Thu, 24 Feb 2011 02:22:00 +0000

I just spent a day without being able to send email because I enabled two factor authentication on my work account. That and a complicated gmail setup that apparently appeals to no one else ;)

I forward all my mail from my work, google apps account into my personal gmail account so that all my email is consolidated for reading, filtering, etc. I then use the “Send mail as” functionality to email from my work account. Gmail even allows you to configure alternate SMTP information so that when I send from my work account people don't get a "Sent on behalf of..." in their Outlook.

Well, when I turned on the new two factor authentication for my work email account the smtp password stopped working and this all broke...silently.

No errors when I sent the email, no errors on the SMTP configuration page and worst: all my emails made it successfully into the "sent” folder!

I finally just guessed what must be happening and fixed it.

Nice.

Connecting to an untrusted certificate in java

noreply@blogger.com (Dan) — Sun, 20 Feb 2011 22:38:00 +0000

I was getting errors(unable to find valid certification path to requested target) in Alfresco trying to connect to LDAP over ssl using an untrusted certificate and found a great hint on the interwebs that I thought I'd share.

Most of the hits on google mention this post: http://blogs.sun.com/gc/entry/unable_to_find_valid_certification

My only criticism of that article is that it only minimally addresses how to actually use the fixed keystore. I downloaded their source and hacked it up a little change the name of the certificate store from "jssecacerts" into "cacert" which is the default certificate store for all java programs. My goal was to fix the certificate store for the entire machine.

To install an untrusted certificate into your keystore the process is like this:

Help improve the Kindle reading ecosystem.

noreply@blogger.com (Dan) — Thu, 06 Jan 2011 16:12:00 +0000

I sent this as an email to kindle-feedback but thought I would share it with the interwebs as well. Let me be clear that I am absolutely in love with my Kindle. I used to not particularly enjoy reading but the kindle has changed my opinion in two short weeks. With that said my major sadness about it was access to personal content. Hence this letter:

Dear Amazon,

Please improve the reading ecosystem represented by the kindle device, and the plethora of applications that also represent "kindle devices" e.g. the iphone, ipad, web, etc. Right now these other devices are treated like second class citizens because there is no way to get personal content onto them and to synchronize the reading of the content.

This hurts users who want to use their kindles as "reading devices" vs. "buying from amazon's bookstores" device.

One idea on how to improve this is to give access to a certain amount of disk space at Amazon, maybe 1 gb (per google docs) for free users and more for people that have bought kindles or have purchased books (maybe there is even a reward program, the more money you spend with Amazon, the more personal disk space you get). This space could be used to store personal documents that then appear in the "archived" section of people's kindles. Then you could add synchronization for: latest location, bookmarks, annotation. Now I can read documents i'm currently working on, review design specifications, etc. All on my kindle.

This changes my kindle from a device I use for an hour a day at home before bed into my preferred reading device at work, at home and on the road.

Thanks for a fantastic device and feel free to contact me,

--dan

Remap Caps lock on a Macbook Pro

noreply@blogger.com (Dan) — Tue, 14 Dec 2010 01:29:00 +0000

I was fortunate enough to receive a CR-48 from Google last week. My impressions of it are more or less in line with what MG Siegler over at TechCrunch has reported: good battery, nice screen, light weight laptop. The only problem is that I have needed to adjust my workflow when I'm working on my regular computer, a Macbook Pro to facilitate sharing documents between the two machines. For example, I have migrated away from OmniOutliner to a comparable online version, Workflowy.

The other major change is that my keyboard habits are changing. With only a weekend usage of ChromeOS I already prefer having the caps lock key replaced with a button to open up new tabs in my browser.

When I got into work this morning I immediately missed my new favorite key. I decided to try and figure out a way to replace this missing behavior.

A recent article outlines how to turn the caps lock key into a control, option or splat. One benefit of an Apple approved method is that the keyboard caps lock light disables itself when you change the functionality. A never ending flickering green light would have been a constant annoyance for me.

This approach isn't enough, I wanted more - to be able to open a new tab in Chrome. I found the PCKeyboardHack preference pane/kext. Using the instructions provided I remapped the caps lock key to F14, or keycode 107 on the slim apple keyboard.

Now that it was remapped to a real keyboard code it wasn't difficult to change the new tab shortcut in chrome to respond to F14.

These changes work on the laptop's keyboard in addition to the external keyboard and I'm very pleased with the results.

Notes about Alfresco's devcon

noreply@blogger.com (Dan) — Mon, 08 Nov 2010 18:22:00 +0000

Nate McMinn has a brief wrap up of the Alfresco developers convention. I was unable to attend and it sounds like I missed a lot of interesting news. I was excited to here:

One upcoming project that was discussed at DevCon is putting together a third-party components catalog for Alfresco. Right now there is nothing like this available. Alfresco community projects are scattered all over the place. Some are in Alfresco Forge, some are on Google Code, still others are on developers' blogs (mine included). I'm sure I'm forgetting a few locations, but you get the idea. Rolling all of this up in one queryable repository would be a fantastic addition to the Alfresco community.

I wonder how this will be deployed and who will maintain it. The wiki idea for documentation seems to be barely moving along.

Creating custom JMX MBeans for reporting in Alfresco

noreply@blogger.com (Dan) — Mon, 08 Nov 2010 15:10:00 +0000

JMX rocks. When configuring a server it is a boon to developers. Especially when combined with the Alfresco subsystem architecture. You can interate on changes to the LDAP sync without having to restart the server. JMX also gives savvy system administrators a way to manage and monitor what’s going on within the repository.

If you’re still unfamiliar with the basics of JMX, especially within the context of Alfresco, Jarred Ottley over at Alfresco has written a number of excellent tutorials. I’ve added some additional articles and come up with the list below.

Some links:

JMX basics in Alfresco
Some JMX console implementation - I prefer Java VisualVM, which comes with the JDK
JMX from the commandline
Tunnelling JMX - for use with Amazon EC2 or any firewalled computer. 3.2 sp2 and newer should review this update .

With the basics out of the way it is often interesting to create your own MBean that can report custom statistics or expose custom methods. This tutorial creates a new MBean that shows the number of Asynchronous jobs being run. Alfresco exports beans using standard Spring practices. This makes keeps everything well documented. The list of things to create is small:

Context file to register new MBean
Annotated Java class

Context File

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd">
<beans>
  <bean id="whySlow" class="com.zia.jmx.WhySlow"/>
  <bean id="ziaExporter"
    class="org.springframework.jmx.export.MBeanExporter">
    <property name="assembler" ref="assembler"/>
    <property name="beans">
      <map>
        <entry key="Zia:name=WhySlow" value-ref="whySlow"/>
      </map>
    </property>
  </bean>
  <bean id="jmxAttributeSource"
    class="org.springframework.jmx.export.annotation.AnnotationJmxAttributeSource"/>
  <!-- will create management interface using annotation metadata -->
  <bean id="assembler"
    class="org.springframework.jmx.export.assembler.MetadataMBeanInfoAssembler">
    <property name="attributeSource" ref="jmxAttributeSource"/>
  </bean>
</beans>

Java Class

  @ManagedResource
  public class WhySlow {
    @ManagedAttribute( description = "Asynchronous actions left to run" )
    public long getAsyncActions() {
      AsynchronousActionExecutionQueueImpl aaeq = ( AsynchronousActionExecutionQueueImpl ) AlfUtil.getSpringBean( "defaultAsynchronousActionExecutionQueue" );
      long ret = -1;
      try {
        Class<?> c = aaeq.getClass();
        Field[] props = c.getDeclaredFields();
        Field tpeField = c.getDeclaredField( "threadPoolExecutor" );
        tpeField.setAccessible( true );
        ThreadPoolExecutor tpe = ( ThreadPoolExecutor ) tpeField.get( aaeq );
        ret = tpe.getTaskCount() - tpe.getCompletedTaskCount();
      } catch( NoSuchFieldException nsfe ) {
        nsfe.printStackTrace();
      } catch( IllegalArgumentException e ) {
        e.printStackTrace();
      } catch( IllegalAccessException e ) {
        e.printStackTrace();
      }
      return ret;
    }
  }

The annotations are important for documentation in the console. There are some reflection shenanigans that allows access to private fields. Your implementation will not need much of this code, except for the annotations.

When this code is deployed the WhySlow mbean will appear at the top level, next to Alfresco node. This is controlled by the key of the map passed into the “beans” property(Zia:name=WhySlow) and is explained in the Spring docs.

Wrap up

A bean created under JMX tends to keep the separation of concerns better than many of the alternatives. We have created “consoles” in webscripts, but it seems to be difficult to train the sys admins to go to multiple places for configuration. Once the repository is started it is consistent to point users at JMX for all administration and reporting.

Using Alfresco for authentication in Spring Surf

noreply@blogger.com (Dan) — Tue, 02 Nov 2010 14:37:00 +0000

Kevin Roast’s blog had a great post about Authentication User’s against an Alfresco Repository. When we first explored Surf, when it starting to get traction in 3.2 or 3.3 our major complaint was that it seemed difficult to actually connect to the repository.

I’m glad to see that this has changed.

Or that it is now better documented.

Technorati Tags: Alfresco

Rapid development in Alfresco Share - the refresh webscript

noreply@blogger.com (Dan) — Mon, 01 Nov 2010 19:26:00 +0000

Zia Consulting develops in Eclipse using a setup different than described by Alfresco. We run an embedded version of Jetty with the Alfresco default WARs. At runtime we mix in changes using extra classpath entries and web overlays. This has many benefits: most importantly that we develop functionality without having to restart the repository. When there is a reason to restart the server we make efforts to fix the problem through programming or configuration.

Share has caches that make the webscripts and surf run faster. These caches have documentation on how to turn them off but this documentation changes with every release of share or surf. This is difficult to keep up with.

If you're interested in the documented process I have a list of articles to read:

Old Surf intro - refreshing the Surf page cache. I don't believe this is needed in the most recent versions of share.

Official wiki documentation - For Alfresco 3.0 ➔ 3.3, before Surf joined the Spring project.

Official developer guide - The section for debugging didn’t get finished. Likely because they moved over to Spring.

I’m sure there are more. Feel free to add them to the comments.

We have given up on following the documentation and developed a custom Web Script that invalidates these caches whenever a source file is changed. The code for this Web Script is from the service index refresh that is part of “org.springframework.extensions.webscripts.bean.IndexUpdate”. We paid particular attention to the reset method of “org.springframework.extensions.webscripts.AbstractRuntimeContainer”. The reset method gets run when you goto http://localhost/share/service/index and click the reset button. The important code is trivial:

  /* public void reset(){ ...*/
  scriptProcessorRegistry.reset();
  templateProcessorRegistry.reset();
  getRegistry().reset();
  configService.reset();
  /* } */

Running all four of these resets takes a couple seconds and is impractical to do after every save. Taking just the script and template reset and putting them in a Web Script keeps it pretty lean. Running time drops from a 2 - 3 seconds to a few hundred ms. It is fast enough that the refresh will happen before you can cmd + tab to your web browser and refresh.

We created a refresh Web Script using the above code and then setup an Eclipse builder to run when the source is updated. To do this you need to create a few things.

Refresh Web Script. It’s best to use “none” authentication.
Ant script that will call the new Web Script. Shown below.
In Eclipse turn the directory containing javascript and freemarker templates (the config directory if your projects look like exploded amp files) into a source directory so it triggers the auto build.
Create a builder according to this tutorial with one change:
- In step #11 they recommend NOTcreating an auto build. Ignore this direction - auto building is really handy for our purposes. It doesn’t cause performance issues, so turn it on and configure it to run “run-refresh-webscript”.

The ant file that we use:

<?xml version="1.0" encoding="UTF-8"?>
<project name="project" default="run-refresh-webscript">
  <target name="run-refresh-webscript">
    <get src="http://localhost:8080/share/service/zia/util/refresh"
      dest="refresh-results.dat" />
  </target>
</project>

Update:

I found this article by Kevin Roast that has some great tips on Share development. The “mode” selection is of great interest.

Creating calculated datalist columns in Alfresco Share 3.4b

noreply@blogger.com (Dan) — Thu, 28 Oct 2010 14:17:00 +0000

This is a short one since not much changed. If you didn't have a chance to read the 3.3 version of this please review it now since I'm not going to post the entire tutorial again. We only need to change the filter and specifically how the dynamic columns are added to the form.

public class ProjectStatusFilter extends AbstractFilter<Object, NodeRef> {
  @Override public void afterGenerate(Object item, List fields,
  List forcedFields, Form form, Map context) {
    boolean typeDefCorrect = (item instanceof TypeDefinition && ((TypeDefinition) item)
      .getName().equals(PmoModel.TYPE_PROJECT_STATUS_DL));
    boolean nodeRefCorrect = (item instanceof NodeRef && AlfUtil.services()
      .getNodeService().getType((NodeRef) item).equals(
      PmoModel.TYPE_PROJECT_STATUS_DL));
    if (typeDefCorrect || nodeRefCorrect) {
      double varianceValue = 0d;
      double percentComplete = 0d;
      double percentExpended = 0d;
      // If this is a node then do the calculations
      if (nodeRefCorrect) {
        NodeRef itemNode = (NodeRef) item;
        /* Some calculations that aren't interesting */
      }
      /* Here are the changes for 3.4b */
      form.addField(FieldUtils.makePropertyField(PmoModel.variancePropDef,
      varianceValue,AlfUtil.services().getNamespaceService()));
    }
  }
  /* Left the other 3 fuctions for completeness */
  @Override
  public void afterPersist(Object item, FormData data, NodeRef persistedObject)
  {}
  @Override
  public void beforeGenerate(Object item, List fields,
    List forcedFields, Form form, Map context) {}
  @Override
  public void beforePersist(Object item, FormData data) {}
}

JBPM Workflow in Alfresco Share 3.4a community release

noreply@blogger.com (Dan) — Tue, 26 Oct 2010 16:15:00 +0000

Dick Weisinger over at FormTek posted a great article on adding new workflows to the Share 3.4 using the newly updated form service. I'm currently developing similar functionality for 3.4b and found the same results - jbmp workflows just seems to work in the share interface.

Article

Creating calculated datalist columns in Alfresco Share 3.3

noreply@blogger.com (Dan) — Mon, 25 Oct 2010 13:55:00 +0000

Datalists out-of-the-box don't have a whole lot of functionality outside of simply capturing data. A simple way to help users is to add calculated columns that transform other columns or related data from the repository. In Alfresco 3.3 this functionality is straightforward and allows creating powerful datalists that even an Excel user could love.

For this demo we will create a part of a project management datalist that takes allows users to enter the following: Estimated to completion, estimated at completion, actuals and budget. From these inputs we will generate the variance, percent complete and percent expended.

Overview

Create a node and type form filter
Create a couple of new static fields
Create a new context file to instantiate the filters
Tell Share to display the fields in the forms

Create a form filter

Form filters derive from an AbstractFilter and must implement the four abstract functions in that class. Filters are called whenever the form service runs and will be pass either a node instance or a typedef. I created this implementation when I didn't understand the difference so I implemented both. I also liked that I could show zeros before the node was created.

  public class ProjectStatusFilter extends AbstractFilter<Object, NodeRef> {
  @Override public void afterGenerate(Object item, List fields,
  List forcedFields, Form form, Map context) {
    boolean typeDefCorrect = (item instanceof TypeDefinition && ((TypeDefinition) item)
      .getName().equals(PmoModel.TYPE_PROJECT_STATUS_DL));
    boolean nodeRefCorrect = (item instanceof NodeRef && AlfUtil.services()
      .getNodeService().getType((NodeRef) item).equals(
      PmoModel.TYPE_PROJECT_STATUS_DL));
    if (typeDefCorrect || nodeRefCorrect) {
      double varianceValue = 0d;
      double percentComplete = 0d;
      double percentExpended = 0d;
      // If this is a node then do the calculations
      if (nodeRefCorrect) {
        NodeRef itemNode = (NodeRef) item;
        /* Some calculations that aren't interesting */
      }
      ContentModelFormProcessor.generatePropertyField(
        PmoModel.variancePropDef, form, varianceValue, AlfUtil
        .services().getNamespaceService());
      ContentModelFormProcessor.generatePropertyField(
        PmoModel.percentCompleteDef, form, percentComplete, AlfUtil
        .services().getNamespaceService());
      ContentModelFormProcessor.generatePropertyField(
        PmoModel.percentExpendedDef, form, percentExpended, AlfUtil
        .services().getNamespaceService());
    }
  }
  /* Left the other 3 fuctions for completeness */
  @Override
    public void afterPersist(Object item, FormData data, NodeRef persistedObject) {}
  @Override
    public void beforeGenerate(Object item, List fields,
    List forcedFields, Form form, Map context) {}
  @Override
    public void beforePersist(Object item, FormData data) {}
}

The key to this is the ContentModelProcessor.generatePropertyField. In order for this to work we need to create a field to be passed in. I created these fields as anonymous classes and put them into the model interface.

Create new field definitions

I'll give one example, the others are simple variations. This is the class that is built when you do any content modeling, and the documentation and possible values are the same. Check the alfresco wiki for more information.

static final PropertyDefinition percentExpendedDef = new PropertyDefinition() {
    @Override
    public boolean isStoredInIndex() {
        return false;
    }
    @Override
    public boolean isProtected() {
        return false;
    }
    @Override
    public boolean isOverride() {
        return false;
    }
    @Override
    public boolean isMultiValued() {
        return false;
    }
    @Override
    public boolean isMandatoryEnforced() {
        return false;
    }
    @Override
    public boolean isMandatory() {
        return false;
    }
    @Override
    public boolean isIndexedAtomically() {
        return false;
    }
    @Override
    public boolean isIndexed() {
        return false;
    }
    @Override
    public String getTitle() {
        return "Percent Expended";
    }
    @Override
    public QName getName() {
        return PROP_CALC_PERCENT_EXPENDED;
    }
    @Override
    public ModelDefinition getModel() {
        /* This is the name of a model that this type will belong to,
        it's at the top of a model file for example
        <model name="pmodl:risksDataListModel" xmlns="http://www.alfresco.org/model/dictionary/1.0">
        */
        return AlfUtil.services().getDictionaryService().getModel(
                MODEL_RISKS_DATA_LIST);
    }
    @Override
    public IndexTokenisationMode getIndexTokenisationMode() {
        return null;
    }
    @Override
    public String getDescription() {
        return null;
    }
    @Override
    public String getDefaultValue() {
        return null;
    }
    @Override
    public DataTypeDefinition getDataType() {
        return AlfUtil.services().getDictionaryService().getDataType(
                DataTypeDefinition.INT);
    }
    @Override
    public ClassDefinition getContainerClass() {
        return null;
    }
    @Override
    public List getConstraints() {
        return null;
    }
};

Create a new context file to instantiate the filters

In the usual places (e.g. WEB-INF/alfresco/extension/custom-form-context.xml) create a new context file and instantiate the filter.

  <?xml version='1.0' encoding='UTF-8'?>
  <!DOCTYPE beans PUBLIC '-//SPRING//DTD BEAN//EN' 
      'http://www.springframework.org/dtd/spring-beans.dtd'>
  <beans>
      <bean id="projectStatusFilterNode" 
          class="com.ziaconsulting.forms.ProjectStatusFilter" 
          parent="baseFormFilter">
          <property name="filterRegistry" ref="nodeFilterRegistry" />
      </bean>
      <bean id="projectStatusFilterType" 
          class="com.ziaconsulting.forms.ProjectStatusFilter" 
          parent="baseFormFilter">
          <property name="filterRegistry" ref="typeFilterRegistry" />
      </bean>
  </beans>

Tell Share to display the fields in the forms

Using the standard method to modify a form in share using the form service tell the node and type forms to display. This means creating a share-config-custom.xml and adding a few extra elements to your currently configured data list.

  <?xml version="1.0"?>
  <alfresco-config>
    <config evaluator="model-type" condition="pmodl:projectStatus">
      <forms>
        <!-- Data Grid view -->
        <form id="datagrid">
          <field-visibility>
            <show id="pmodl:projectName"/>
            <show id="pmodl:projectStatus"/>
            <show id="pmodl:budget"/>
            <show id="pmodl:actuals"/>
            <show id="pmodl:etc"/>
            <show id="pmodl:eac"/>
            <!-- New Calculated column -->
            <show id="pmodl:pecentExpendedCalculated"/>
          </field-visibility>
        </form>
      </forms>
    </config>
    <config evaluator="node-type" condition="pmodl:projectStatus">
      <forms>
        <!-- Data Grid view -->
        <form id="datagrid">
          <field-visibility>
            <show id="pmodl:projectName"/>
            <show id="pmodl:projectStatus"/>
            <show id="pmodl:budget"/>
            <show id="pmodl:actuals"/>
            <show id="pmodl:etc"/>
            <show id="pmodl:eac"/>
            <!-- New Calculated column -->
            <show id="pmodl:pecentExpendedCalculated"/>
          </field-visibility>
        </form>
      </forms>
    </config>
  </alfresco-config>

Next Steps, upgrading to 3.4b

The upgrade to 3.4 isn't too bad. Alfresco changed the API and didn't bother to leave any hints about how to upgrade, but fortunately I was able to re-figure out how to do this. I'll publish the updates later this week.

Attaching to a running Alfresco instance - running code in Alfresco

noreply@blogger.com (Dan) — Mon, 18 Oct 2010 21:54:00 +0000

A coworker and I have been scheming for a number of months on ways to run migration scripts and develop workflow in Alfresco. The common thread is that we have a repository that's running and we want run arbitrary code against it. We have had a number of ideas that have taken us in two different directions. I'd like to talk about a few of them, outline the issues we've run into and then outline our merged solution.

Migration

One approach (the merits depend on the project) to migration is load a bunch of data into the repository and then run some code on the repository to translate the data, extract metadata or whatever. This can be run with an external suite of tools (e.g. an ETL), but this is often overkill. The code to accomplish the same thing in Alfresco is often easier than 3rd party tools, involves less tools or languages and can be cheaper. There are a number of ways to run javascript or java code.

Command Servlet

There is an old (from the age of the documentation) servlet that can run arbitrary javascript that seems to fit the bill except that it will only run code stored as nodes in the repository and only in "insecure" mode, which means no access to native java and you have to develop over cifs/webdav or directly in a wysiwig editor in Alfresco. None if this is ideal.

Command Servlet

Secure webscripts

As a webscript

Creating a web script gives you access to the real Rhino engine, i.e. "secure mode" or could write the script in Java giving you access to the entire API. This is kludgy because you're executing your migration script by refreshing a webpage. Also, the "standard out" is either logging messages or the free marker template. The last nail in the coffin is that you must run your script in a massive transaction. This seems to really slow down the system and if you are processing more than 1000 nodes you will get cache error messages. Turning off the caching in the web script's descriptor file means no more access to the company home.

Workflow development

The slowest part of workflow development is stepping through a long workflow. To speed up the development process my coworker has developed a series of junit tests that setup the workflow and run it to a particular step that he's working on. This allows all of the previously developed steps to be skipped over so that he can develop and debug the current step, it also provides a certain level of integration/regression testing. He also spends a lot of time developing the java parts of workflow from within a breakpoint, allowing the jvm to swap the class files and then stepping through the code.

There are couple issue with this approach. The first is the startup and memory usage. The junit test spins up an entire alfresco repository, which is slow and very memory intensive. Since it is sharing the same alf_data as the real repository it tends to corrupt the lucene indexes and sometimes the entire repository. He ends up dropping and reloading the entire schema multiple times per day and reindexing the indexes. Even with these drawbacks the development time improvements, due to faster iterations, are substantial.

Java vm attachment - a new approach

Along with Java 6 came a new feature that I only recently discovered. You can now attach to a running virtual machine and run code loaded from a jar file and pushed into the running virtual machine. Conceptually I can write a program (using a Java main), bundle it up in a jar, load it onto a server with a running Alfresco repository and then execute the code which will run in the same virtual machine and therefore have access to the applicationContext and any managed beans. It works like JMX, for developers.

If attaching to the VM is all you're interested in feel free to stop here and read through this tutorial of the process. If you're interest in a more Alfresco specific, or at least a more web container specific implementation then stay with me.

I took what I learned in that tutorial and created a program and agent main function. Both are bundled into a jar file. The jar has a manifest that makes it executable and takes a classpath directory that is added to the running container and a Runnable class to be run in a new thread from within the container. The relevant code in the program class is:

String classPathToRunnable = args[0];
String runnableClassname = args[1];
String agentArgs = classPathToRunnable + "," + runnableClassname;
List<VirtualMachineDescriptor> listAfter = null;
try {
  listAfter = VirtualMachine.list();
  boolean connected = false;
  for (VirtualMachineDescriptor vmd : listAfter) {
    if (vmd.displayName().contains("Main")) {
      vm = VirtualMachine.attach(vmd);
      vm.loadAgent(agentJarPath, agentArgs);
      connected = true;
    }
  }
  if (!connected) {
    System.err.println("Couldn't connect, never found the server.");
  }
} catch (Exception e) {
  e.printStackTrace();
}

ClassPathToRunnable is a directory, e.g. "bin", when run in a typical java environment. The runnable classname is a fully qualified classname, such as "com.ziaconsulting.MyRunnable". The classpath directory is going to be added to the classpath of the container and then we'll use reflection to call the class that is passed in. The way the attach process works is to run on a jar file that lists a class to run in the manifest file. I have a simple ant file that manages the build and the manifest file:

<project name="Attach" default="jar">
  <target name="jar">
    <jar destfile="agent.jar">
      <fileset dir="./bin">
        <include name="**"/>
      </fileset>
      <manifest>
        <attribute name="Main-Class"
          value="com.ziaconsulting.Attach" />
        <attribute name="Agent-Class"
          value="com.ziaconsulting.Agent" />
      </manifest>
    </jar>
  </target>
</project>

Pay special attention to the "Agent-Class" manifest attributes.

The next step is to work with the agent main class. This proved to be a very tricky part because of the complexity inherit with java containers, specifically the number of classloaders. I developed code specific to Jetty's container which will probably not work with tomcat. Though the ideas should be the same:

public static void agentmain(String agentArgs, Instrumentation inst) {
 // Execute the class that was passed in
 try {
   // Split the arguments - should be a classpath and a class
   String[] args = agentArgs.split(",");
   if (args.length != 2) {
     throw new IllegalArgumentException(
     "Not enought arguments passed");
   }
 
   // Begin the jetty specific stuff, we are trying to get the
   // classloader from the Alfresco web app
   Server s = Main.server;
   WebAppDeployer wap = s.getBeans(WebAppDeployer.class).get(0);
 
   HandlerCollection h = wap.getContexts();
   Handler[] handlers = h.getHandlers();
   for (Handler handler : handlers) {
     if (handler.toString().contains("alfresco")) {
       if (handler instanceof WebAppContext) {
         WebAppContext wac = (WebAppContext) handler;
 
         WebAppClassLoader cl = (WebAppClassLoader) wac
         .getClassLoader();
 
         // Check and see if this classpath is already added, if
         // not then add it
         boolean classPathAlreadyExists = false;
         for (String entry : System.getProperty(
           "java.class.path").split(
           System.getProperty("path.separator"))) {
             if (entry.equals(args[0])) {
             classPathAlreadyExists = true;
             break;
           }
         }if (!classPathAlreadyExists) {
           cl.addClassPath(args[0]);
         }
         // Run the passed in class
         final Class runnableClass = wac.getClassLoader()
         .loadClass(args[1]);
         final Thread thread = new Thread(
         (Runnable) runnableClass.newInstance());
         thread.setContextClassLoader(cl);
         thread.start();
 
       }
     }
   }
 } catch (Exception e) {
   e.printStackTrace();
 }

This may seem like beating around the bush a little, my goal was for all the agent/attaching code to be abstracted so that all users have to do is create their own runnable class that knows nothing about how to get into the VM. This is demonstrated in the simplicity of my runnable class:

public class MyRunnable implements Runnable {
  public void run() {
    try {
      ApplicationContext ac = AlfUtil.applicationContext;
      System.out.println(ac.toString());
    } catch (Exception e) {
      e.printStackTrace();
    }
  }
}

There isn't much more to show. I haven't had time to expand this to anything more than a proof of concept, but even at this stage it's pretty compelling.

The output of this code is directed to the stdout of the container e.g. catalina.out or alfresco.log.

Building Alfresco enterprise from source v3.4 beta

noreply@blogger.com (Dan) — Mon, 11 Oct 2010 21:54:00 +0000

Our development environment runs off an exploded a war file in an embedded version of jetty, all running in Eclipse. We typically build the war from source pulled from Alfresco Enterprise SVN. Knowing how to do this means we can checkout from any of the development branches and get a feeling for how different features are progressing without waiting for the official build to be distributed. It also give me a warm fuzzy to know how the beast is built.

Up until 3.4b it was an easy process to run "ant incremental" within the enterpriseprojects directory and then unzip the wars that were produced. The build was changed in 3.4 so that this process now creates a Circular Dependency and fails to run immediately.

I posted this as a regression bug to JIRA and got a response today from Paul Holmes-Higgin to try ant -Dbuild.script=enterpriseprojects/build.xml -f continuous.xml distribute. This still causes a failure unless you setup extra some folder structure to support building all of the distribution packages (see this discussion) but at least you get the war file in the assemble directory, which is good enough for me.

One thing he didn't mention is that you need to add version numbers as arguments to ant so that some of the jars that are created have valid filenames.

I settled on this:

$ ant -Dbuild.script=enterpriseprojects/build.xml
-Dversion.major=0 -Dversion.minor=0 -Dversion.revision=0
-f continuous.xml distribute

Again, this build will fail unless you setup the folder structure described in the above discussion.

Check for war files:

$ ls -l build/assemble/web-server/webapps/
total 239080
-rw-r--r--  1 dhopkins  staff    97M Oct 13 07:52 alfresco.war
-rw-r--r--  1 dhopkins  staff    20M Oct 13 07:52 share.war

Which leads eventually to...

How sweet it is!

Using the patch service to run arbitrary code on AMP install

noreply@blogger.com (Dan) — Sun, 10 Oct 2010 21:49:00 +0000

There seems to be many times that the Alfresco repository needs to be modified when an AMP is first installed e.g. migration of current nodes. While Alfresco provides a data bootstrap mechanism it appears it is inadequate for anything except the most basic tasks. This post was predicated on finding a use of the patch service in the WCM installation script.

I haven’t seen many docs on the proper or improper use of the patch service. At first blush it is used as a Repository internal tool to update the repository during major releases and hotfixes. As I’ve mentioned, the inspiration for this came from the WCM "installation" during which you insert the wcm-bootstrap-context.xml into the repositories classpath and then start it up. In that file is a new bean that uses the patch service to create a couple of folders.

Creating my own patch

To create a new patch we need a couple of things

A context file to create the spring beans
A properties file with description information
A java class to instantiate that will run the patch

The context file can exist any where in the alfresco/extension package. It should contain a bean that inherits from the basePatch abstract bean.

<bean id="patch.pmoUsers" class="com.ziaconsulting.bootstrap.FreshProjectData"
  parent="basePatch">
  <property name="id">
    <value>patch.pmoUsers</value>
  </property>
  <property name="description">
    <value>patch.pmoUsers.description</value>
  </property>
  <property name="fixesFromSchema">
    <value>0</value>
  </property>
  <property name="fixesToSchema">
    <value>${version.schema}</value>
  </property>
  <property name="targetSchema">
    <value>10000</value>
  </property>
  <property name="services" ref="ServiceRegistry"/>
  <property name="groupAuthorityName" value="freshprojectadmins"/>
  <property name="groupDisplayName" value="Fresh Project Admins"/>
</bean>

This is a simple bean that inherits from basePatch. The settings that are the most important are the three versions that are passed in (fixesFromSchema, fixesToSchema and targetSchema). This is in contrast to the way that Alfresco typically uses this service, where the patch is meant to be applied to one specific schema.

The "id" is written to the patch table in the database and the description is a reference to a key in a properties file that has been loaded by the i18n Resource Bundle Bootstrap Component.

The properties file looks like any normal properties and is loaded by the I18N service. The bean that we will use is:

<bean id="patch.pmoUsers.resourceBundles"
class="org.alfresco.i18n.ResourceBundleBootstrapComponent">
    <property name="resourceBundles">
    <list>
      <value>alfresco.extension.pmoPatches</value>
    </list>
  </property>
</bean>

Nothing fancy here. Just make sure that the ID is unique so you don't stomp the original message bundle bean.

The last thing to create is the java class that runs when the patch is installed. It's pretty easy to find loads of examples in the repository of other patches and copy the ideas. Here is one for completeness.

public class FreshProjectData extends AbstractPatch {
...... La la la, details ......
@Override
protected String applyInternal() throws Exception {
    // Create the new user group
    String freshProjectAdmingroup = services.getAuthorityService()
      .createAuthority(AuthorityType.GROUP, groupAuthorityName);
    services.getAuthorityService().setAuthorityDisplayName(
      freshProjectAdmingroup, groupDisplayName);
    return "Created " + freshProjectAdmingroup;
  }
}

One thing to note is that the return string is placed into the "report" field in the database (see next section).

Changes to the database

The database stores a record of the applied patch in the alf_applied_patch table. Which you shouldn't need to check unless you're curious.

Using Jar deployment to solve conflicts with multiple customizations in Alfresco Share

noreply@blogger.com (Dan) — Thu, 02 Sep 2010 21:33:00 +0000

Installing customizations using AMPs into a new Alfresco repository is pretty easy, there isn’t much room for configuration overwriting other configuration. In older systems that have been installed by a client or in systems that have multiple SI’s customizations installed it maybe the case, especially in Share, where configuration from one amp overwrites the configuration of another amp. You can get into difficult to detect problems depending on the order in which amps are applied to Alfresco (probably alphanumeric, last one wins).

With the release of 3.3 Alfresco has created a new way to deploy customizations to Share using Jar files and some trickery with Servlets to handle loading web resources e.g. Css, and client-side javascript. The major benefit with this approach is that it allows developers to have some config files loaded from the jar file, where they won’t be overwritten. Not all config, it seems that only the alfresco config i.e. One that end with -custom can be loaded this way. You can see the list in the file slingshot-application-context.xml:

<list>

 <value>classpath:org/springframework/extensions/webscripts/spring-webscripts-config.xml</value>

 <value>classpath:META-INF/spring-webscripts-config-custom.xml</value>

 <value>jar:*!/META-INF/spring-webscripts-config-custom.xml</value>

The files that can be deployed this way in Share are:

spring-webscripts-config-custom.xml
spring-surf-config-custom.xml
share-config-custom.xml
spring-surf-config-custom.xml
spring-webscripts-config-custom.xml

It turns out you can also do this in DM Explorer for web-client-config-custom.xml.

The idea here is not to stop using amps to deploy the code but instead to include the config within the jar with the other java code, instead of bundling it within the amp.

Amp structure before:

/amp

/lib

myAmp.jar ** Showing the contents

/com

/ziaconsulting

/MyClass.class

/config

share-config-custom.xml

   /web

And then after:

/amp

/lib

myAmp.jar ** Showing the contents

/META-INF

share-config-custom.xml

/com

/ziaconsulting

/MyClass.class

/config

/web

Using this idea it’s going to be very unlikely that another customization will override the share-config-custom.xml.

Creating the jar

Using this information we can create a project to deploy to share and include a share-config-custom.xml without fear that another amp will overwrite our configuration e.g. Records Management.

If you use ant to build the amp file your jar task might look like this.

<jar destfile="${work.dir}/lib/myAmp.jar">

<fileset dir="${work.dir}/bin" includes="com/**" />

<metainf dir="config-share-metainf">

<patternset>

<include name="**"/>

</patternset>

</metainf>

</jar>

Deploying the finished product

Assuming that your build is already creating an amp file the amp is deployed just like usual. If you want to double check that the new source file is being picked up then turn on debug logging for the UrlConfigSource. Open up the log4j.properties in the share webapp (typically tomcat/webapps/share/WEB-INF/classes/log4j.properties) and add this to bottom of the file.

Log4j.logger.org.springframework.extensions.config.source.UrlConfigSource=debug

If all has gone to plan you should see a log message from that class.

13:28:36,038  INFO  [config.source.UrlConfigSource] Found META-INF/share-config-custom.xml in

file:/Users/dhopkins/Public/work/test/alfresco-enterprise-tomcat-3.3.1/tomcat/webapps/share/WEB-INF/lib

/pmoShare.jar

That’s what we want to see.

boulder dan

Thread safety with no threads in sight

Epic google error message fail

Connecting to an untrusted certificate in java

Help improve the Kindle reading ecosystem.

Remap Caps lock on a Macbook Pro

Notes about Alfresco's devcon

Creating custom JMX MBeans for reporting in Alfresco

Context File

Java Class

Wrap up

Using Alfresco for authentication in Spring Surf

Rapid development in Alfresco Share - the refresh webscript

Creating calculated datalist columns in Alfresco Share 3.4b

JBPM Workflow in Alfresco Share 3.4a community release

Creating calculated datalist columns in Alfresco Share 3.3

Overview

Create a form filter

Create new field definitions

﻿Create a new context file to instantiate the filters

﻿Tell Share to display the fields in the forms

Next Steps, upgrading to 3.4b

Attaching to a running Alfresco instance - running code in Alfresco

Migration

Command Servlet

As a webscript

Workflow development

Java vm attachment - a new approach

Building Alfresco enterprise from source v3.4 beta

Using the patch service to run arbitrary code on AMP install

Creating my own patch

Changes to the database

Using Jar deployment to solve conflicts with multiple customizations in Alfresco Share

Creating the jar

Deploying the finished product

Create a new context file to instantiate the filters

Tell Share to display the fields in the forms