Cristian's Blog

Publishing custom artifacts with SBT

2015-09-30T13:46:00.000-07:00

In today's world the code is packaged and re-packaged in different ways. The code is typically packaged in a jar file. Along with that there is another jar file for the sources, and another for the javadoc. But then an application will use more than one jar along with other resources which results in the need for a "distribution" which is typically a zip file with a bunch of resources copied in the appropriate structure to be run. But then there is Docker which yet another packaging solution for delivering the application. With docker you would re-package the distribution in a different format. All these files need to be published to some repository as part of the build or release process.
I guess typically, the UI is last piece that with the most dependencies in a large application. In my Company for instance we develop the back-end code using plain Java and Maven and UIs using Play Framework and SBT. There is always the possibility of Mavenizing the Play project but somehow I never liked to complicate things when there is no absolute need for it.
When it comes to dependency management, the Maven builds will publish their artifacts in a repository (Artifactory for instance) and the UI will use the artifacts via managed dependencies.
SBT then compiles and packages the UI code and the resulting artifacts can be published to Artifactory via the simple "publish" command.
There are however some gaps here, and they all can be solved with a few lines of code in your build.sbt. :

Play will generate multiple jar files but not all are published by default. For instance, try creating a Play module that is not just code, but is a full Play app complete with images, CSS and other assets. Play will package the assets separately from the project jar. It all works fine if you use a distribution created via "dist" because Play knows to add all the jars. But if you try to use the application as a dependency in another, you will need the assets jar which is not typically published. Fortunately it does not take too much code to convince SBT to do so. So once you encounter the problem you will ask Google for a solution and you will find that adding this to your build.sbt will result in publishing the assets jar too:
```
packagedArtifacts in publish := {
  val artifacts: Map[sbt.Artifact, java.io.File] = (packagedArtifacts in publish).value
  val assets: java.io.File = (playPackageAssets in Compile).value
  artifacts + (Artifact(moduleName.value, "jar", "jar", "assets") -> assets)
}
```
Then you can add both the regular artifact and the asset jar as a dependency (note the "assets" classifier):
```
libraryDependencies ++= Seq(
  "com.acme" %% "foo" % "1.0",  "com.acme" %% "foo" % "1.0" classifier "assets")
```
Play/SBT do a nice job of generating a distribution. The distribution step, however is not included in the release process (see sbt-release plugin). This can be fixed easily by customizing the release process:
```
releaseProcess := Seq[ReleaseStep](
  checkSnapshotDependencies,  inquireVersions,  // runTest,           // do not run tests - assume everything is perfect  setReleaseVersion,  commitReleaseVersion,  tagRelease,  releaseStepTask(dist in myapp), // create distribution  publishArtifacts,  setNextVersion,  commitNextVersion,  pushChanges)
```
In the above snippet I am removing the test phase because we are automatically running tests as part of each commit so when it gets to release, everything is in order. This is just to speed up things.
Also, I am adding a step for creating the distribution. This step need to be run before the version is changed or the zip file name will be stamped with the next version. You'll see later why I do it before publishing artifacts as well.
Now the zip file is created but it is not included in the list of artifacts being published. Oh well, the trick at point 1) can be applied here too.
I don't know about others but I do not like to mix jars with distributions so I really wanted to publish the distribution to a different repository. This also is fixable and you can find the solution on stackoverflow. The idea is that you can create a custom task that extends the "publish" task and overrides the "publishTo" setting.

Finally, since I am working on a relatively large project, there are in fact multiple submodule/applications that are being built and they do not always need to be deployed together. As such, I am creating multiple distributions that package different components. So now I need to put it all together - publish multiple distributions to a different repository during the release process.

lazy val publishDist = taskKey[Unit]("Publish distributions - Custom Task")
publishDist := {
  println("Publishing distributions ...")
  val extracted = Project.extract(state.value)
  Project.runTask(publish, extracted.append(List(
    publishTo := Some("distributions" at distRepo),    publishMavenStyle := true,    publishArtifact in Compile := false,    publishArtifact in Test := false,    publishArtifact in Universal := false,    packagedArtifacts in publish := {
    val artifacts: Map[sbt.Artifact, java.io.File] = 


(packagedArtifacts in publish).value

artifacts +
          (Artifact("root", "zip", "zip") ->
            baseDirectory.value / "target" / "universal" / ("root-" + version.value + ".zip")) +
          (Artifact("submodule1", "zip", "zip") ->
            baseDirectory.value / "modules/submodule1/target" / "universal" / ("submodule1-" + version.value + ".zip")) +
          (Artifact("submodule1", "zip", "zip") ->
            baseDirectory.value / "modules/submodule2/target" / "universal" / ("submodule2-" + version.value + ".zip")) +
    }
  ), state.value), true)
}

Here I am adding to SBT's default list of artifacts my files - the zips along with customizations on where and how to publish.
This task can be invoked manually in the activator console, but I created it so that I can invoke it as part of the release process. It is important that the task is invoked after the distributions are created:

releaseProcess := Seq[ReleaseStep](
  checkSnapshotDependencies,  inquireVersions,  // runTest,           // do not run tests - assume everything is perfect  setReleaseVersion,  commitReleaseVersion,  tagRelease,  releaseStepTask(dist in root),
  releaseStepTask(dist in submodule1),
  releaseStepTask(dist in submodule2),
  publishArtifacts,  releaseStepTask(publishDist), // publish the distributions to Artifactory  setNextVersion,  commitNextVersion,  pushChanges)

As it's always the case with SBT it looks easy and obvious when done, but it's not as obvious when you are not very clear on its concepts.

Play 2.2 subprojects and their assets

2014-02-04T19:16:00.003-08:00

Yes, I still like Play! very much. But sometimes it is frustrating in its lackluster documentation. I spent literally hours trying to understand how to work with sub-projects. The fact that I could not find a step-by-step tutorial did not help (note to self - create one!)
I followed the documentation and created the top project, then subprojects. Then created dependencies. Btw, if you want to generate the eclipse projects from console make sure you do not aggregate the projects. For some reason, whenever I add aggregate(subproj1, subproj2) to the top level poject in build.sbt it will omit some projects.
All good, projects compile, and run. But now it is important to take a look at the packages - make sure you refactor your code and move all controllers in subproj1 into their own package (ie. controllers.subproj1). Then, do the same for the scala.html files in views. You should end up with something like this:

multiproj
 build.sbt
 app
   controllers
     Application.java
   views
     index.scala.html
     main.scala.html
   conf
     application.conf
     routes
   public
     images
        ...
     stylesheets
        ...
     javascripts
        ...
   modules
     core
       build.sbt
       app
         controllers.core
           Application.java
         views.core
           index.scala.html
           main.scala.html
       conf
         application.conf
         core.routes
       public
          ...
     test
       build.sbt
       app
         controllers.test
           Application.java
         views.test
           index.scala.html
           main.scala.html
       conf
         application.conf
         core.routes
       public
          ...

and the top build.sbt looks like this:

name := "multiproj"

version := "1.0-SNAPSHOT"

libraryDependencies ++= Seq(
 javaCore, cache
)     

play.Project.playJavaSettings

lazy val root = project.in(file(".")).dependsOn(core, test)

lazy val core = project.in(file("modules/core"))

lazy val test = project.in(file("modules/test")).dependsOn(core)

and the top routes file:

GET     /                           controllers.Application.index()
-> /core core.Routes
-> /test test.Routes
# Map static resources from the /public folder to the /assets URL path
GET     /assets/*file               controllers.Assets.at(path="/public", file)

Obviously, the core and test routes files must use proper packages when referring to the controller classes. So far, it is not that bad. But what about the resources? Notice how the same top level public folder is invoked above. Well, there is an extra step - create an Assets class in the controllers.core and controllers.test that will help scope the lookup to the project directory. Play documentation nicely provides the contents of this class:

package controllers.test;

import play.api.mvc.Action;
import play.api.mvc.AnyContent;

public class Assets {
    public static Action at(String path, String file) {
        return controllers.Assets.at(path, file);
    }
}

and now update the route in the test.routes accordingly:

GET     /assets/*file               controllers.test.Assets.at(path="/public", file)

Make change in the stylesheet from top level project and you will see it applied. Now change the color in the stylesheet from test project and nothing happens. WTF?!?!

Well, as it turns out, Play uses the same classloader to load the files, so if the file name from top level and subproject is the same, it is not going to work. Thus, it is important that the assets from subprojects (public folders from subprojects) have unique names. You can follow the same scheme as for routes file and have public/main.css in top level project and public/test.main.css in the test subprojects. Then you will need to make sure you use the proper reverse routing in the template. Example: @controllers.test.routes.Assets.at("stylesheets/test.main.css")

Bottom line, yes you can split your application in sub-projects and be organized, but I am a bit disappointed in the number of hoops I have to jump to make it go. It should be simpler, Play!. Please make it more straight-forward! I would suggest that the console should know about sub-projects and pre-create the right structure and naming... Can't be that hard to make the console do all the renaming steps above.

Links:

http://www.playframework.com/documentation/2.2.x/SBTSubProjects
https://github.com/playframework/playframework/issues/1181

Play and JAAS

2013-03-14T08:15:00.000-07:00

I blogged in the past about Play and JAAS. I have actually created a Play module in GitHub (https://github.com/cduicu/play-jaas) that can do JAAS style authentication in a Play application. Recently I have updated the module for Play 2.1 as well as fixing one important multi-threaded bug.
You see, I was lazy and I was passing the Http.Context via a static in order to have access to it in the callback and the login module. But this is wrong as it makes the login module thread unsafe! This was fix in the last commit and the context is now passed via callback handler to callback and then to auth module.

Play 2.0.x Module

2013-02-07T07:26:00.000-08:00

During development one always finds themselves in the position of creating reusable code. For Play, I have a project containing a lot of the boilerplate code that I want to share among many concrete Play applications. One solution is to create a regular Play project (I'll refer to it as "playcommons") and set-up project dependency between your application and the playcommons project. See here for more details. However, managing the versions of the playcommons is not straight-forward.
Another solution is to make the playcommons project a Play module. This way, the project can be compiled and published and therefore the versions will be managed via the repository. I did not find some clear documentation on Play website on creating modules but here is a good article on the subject: http://www.objectify.be/wordpress/?p=363.
In the end, you can either get the packaged jar of the module and include it in the Play application (this way you don't have to manage a repository) or publish the module in a repository and then manage the dependency via Ivy in Build.scala.

The Big Data Landscape

2013-01-31T06:14:00.004-08:00

I did not make this, but I do find it interesting.

DataTables Fixed Height

2013-01-30T06:59:00.002-08:00

I've already confessed that I am a big fan of Twitter Bootstrap and jQuery. In my apps I often deal with tables and grids. In the past I was using the wonderful ExtJS grid. But ExtJS is rather heavy and it does not play well with Bootstrap. So I was looking at alternatives, and there is a nice (and free!) jQuery Plugin for this: DataTables.
While DataTables gives you a lot of flexibility I could not find a nice way of having the table's height fixed. There is a discussion on their forum for this topic but the solution did not work that well for me so I came up with a simpler alternative that seems to work well. The following JS snippet refers to the examples on DataTables website.

$table = $('#example');
$table.dataTable( {
  "sDom": 'tif', 
  "bScrollInfinite": true, 
  "bScrollCollapse": true, 
  "sScrollY": "200px",
  "oLanguage": { 
    "sLengthMenu": "_MENU_ / page",
 "sInfo": "(_START_ to _END_) of _TOTAL_",
 "sSearch": ""
  }
  ,"fnDrawCallback": function() {
        $table.dataTable()._fnScrollDraw();        
        $table.closest(".dataTables_scrollBody").height(200);
   }  
} );

From the snippet above, only the fnDrawCallback is important. It will fix the table's height to 200px as instructed in the scroller configuration. The trick is to let the component do its work and then simply adjust the height of the scroller element.
A better version would not duplicate the height and would reuse the value configured in sScrollY property, but I did not find yet how to access it.

Inline editable fields for an HTML table

2013-01-23T13:28:00.000-08:00

I am a fan of Twitter's Bootstrap. It is simply fantastic. But you can make it more powerful if you use jQuery plugins and Bootstrap extensions. So here is Bootsnipp - a site that lists cool Bootstrap extensions.
Traditionally on the web, when you want to capture information from the user you have to create a form. This is great, but sometimes building a form is not that natural. Sometimes we want to edit a value in the middle of a text, or edit the value from a table. There are a lot of solutions out there, and this is just one of them using Bootstrap and a neat extension named X-editable.
While the X-editable site shows a lot of examples, I could not find a way to make my table handle more like a form. What this means is that I want to edit the fields in the table without posting the changes and have a "Save" and "Reset" buttons to apply the changes or revert to the original values.
The "Save" button is nicely demo-ed on X-editable site but the reset functionality was not to my liking as the example is not truly reverting the values to their original state, but instead it is setting them to null (which is fine for that example as it deals with creating a new entry).
Then I realized that the library does not save the original value and it is lost. So I plugged in my handlers for save and reset in order to add my desired functionality.The trick is that I save the original value in the element itself on the "save" event and then make sure I clean it up when the processing completes (either successfully or the reset is invoked). My example also extends the original example and looks for next editable item on the rows as well as columns.

// this is to automatically make the next item in the table editable
$('.edit').on('save', function(e, params){
    var that = this;
    // persist the old value in the element to be restored when clicking reset
    var oldItemValue = $(that)[0].innerHTML;
    if (!$(that).attr('oldValue')) {
     $(that).attr('oldValue', oldItemValue);
    }
    setTimeout(function() {
        // first search the row
        var item = $(that).closest('td').next().find('.edit');
        console.log(item);
        if (item.length == 0) {
            // check the next row
            item = $(that).closest('tr').next().find('.edit');
        }
        item.editable('show');
    }, 200);
});

The "Reset" button handler:

$('#resetbtn').click(function() {
    $('.edit').each(function() {
        var o = $(this);
        o.editable('setValue', o.attr('oldValue')) //clear values
         .editable('option', 'pk', o.attr('pk')) //clear pk
         .removeClass('editable-unsaved')
      .removeAttr('oldValue');
    });
});

And the "Save" button handler:

$('#savebtn').click(function() {
   $('.edit').editable('submit', { 
       url: '/post', 
       //ajaxOptions: { dataType: 'json' },           
       success: function(data, config) {
           $(this).removeClass('editable-unsaved') //remove unsaved class
               .removeAttr('oldValue'); // clear oldValue
       },
       error: function(errors) {
           console.log('error');
           var msg = '';
           if(errors && errors.responseText) { //ajax error, errors = xhr object
               msg = errors.responseText;
           } else { //validation error (client-side or server-side)
               $.each(errors, function(k, v) { msg += k+": "+v+"
"; });
           } 
       }
   });
});

You can find the code and a demo here.

Play Framework 2.0.3 to 2.1 RC2 upgrade

2013-01-15T12:53:00.002-08:00

If you were expecting that upgrading your Play project from 2.0.x to 2.1 will be a simple replacement of the play libraries you will be disappointed. But don't worry, as it is not that bad. Here is a mini guide for the upgrade.

Download Play Framework 2.1RC2.
Edit [yourApp]/project/plugins.sbt file and set SBT version to "2.1-RC2"
Edit [yourApp]/project/build.properties file and set SBT version to “0.12.2-RC2”
The [yourApp]/project/Build.scala file needs to be updated to remove a warning about “using a deprecated version of Play's SBT Project”. To do this you will need to:

Replace “import PlayProject._” with “import play.Project._”
Update appDependencies object value and make sure the sequence contains "javaCore, javaJdbc, javaEbean".
Example:
val appDependencies = Seq(javaCore, javaJdbc, javaEbean)
Replace
“val main = PlayProject(appName, appVersion, appDependencies, mainLang = JAVA)” with
“val main = play.Project(appName, appVersion, appDependencies)”

1. It is recommended to run “play clean-all” before compiling anything. This will ensure a cleanup of sbt.
A change in the API that will likely impact every application out there is in how you access the form. Every reference to Controller.form() will error as that is no longer available. Instead you should change your code to use play.data.Form.form().
Launch the play console and try and compile the application. You might get new warnings regarding your routes. They are just warnings but you should have a look at them.
If your code compiles and you are using Eclipse as your IDE of choice you should re-eclipsify your project as Play 2.1 will have a different classpath and a different set of libraries. Note that the old "eclipsify" command is no longer available as it was replaced by "eclipse" command.

If you are wondering about differences regarding classpath, tasks, commands between 2.0.3 and 2.1 RCs you can have a look here.

Thread Deadlock

2013-01-11T11:58:00.003-08:00

Instead of long articles, here is a picture that describes the thread deadlock nice and clearly:

Can't get simpler than that!

Variables in Scala files for Play 2.x

2012-09-28T06:29:00.001-07:00

The scala html files for Play 2.x will eventually be compiled in classes/methods. Each one has a signature to allow type-safe parameter passing from controller to the view. But sometimes we need to include a script in the page and that is when the need for local variables may occur.
For instance I want to make sure I add HTML only if the object is not null. In JAVA this would be equivalent to:

Feature f = features.getFeature(featureName);
if (f != null) {
.... use the feature object ....
}

or if I need to iterate:

for (Feature f : features) {
if (f == null) continue;
.... use the feature object ....
}

Note that I test the object for null and use "continue" instead of enclosing everything in the if block. It is a matter of preference - I think it makes the code more readable.

Two problems arise when trying to do the same in Scala in the html pages:

declaring local variables in Play scala pages is quite cumbersome
there is no "continue" in Scala.

Here are a few workarounds for the issues mentioned above:

Use "Option". Option is a replacement for checking for null. The map is created only if not null. In this case a map of one element is created and is assigned to “feature”.

@Option(features.getFeature(featureName)).map {feature=>
........}

What if there are more that one?

@Option(features.getFeature(featureName)).map {feature=> @Option(1).map (f2 => }}

You can also use for loops.

@for(feature1 <- Option(features.getFeature(featureName)); f2 <- Option(1)) {
}

If List is used instead of Option, the for loop will be executed once for each item in the List.

Upgrade from Play 2.0.1 to 2.0.2

2012-06-28T10:29:00.000-07:00

Say you have a project in Play 2.0.1, but guess what? There is a new (minor) version out there that might just fix a bug you care about. How do you go about upgrading?
Well, first you need to download the new Play 2.0.2 framework and replace the old one. Then, you need to edit a couple files in your project in order to make it compatible:

edit [appName]/project/build.properties and change sbt version from 0.11.2 to 0.11.3. Like this:
sbt.version=0.11.3
edit [appName]/project/plugins.sbt and change the version of play from 2.0.1 to 2.0.2. Like this:
addSbtPlugin("play" % "sbt-plugin" % "2.0.2")

That's it! Now launch play again and you should see that it updates a bunch of libraries and then its' ready.

Additionally if you are using Eclipse IDE, you might need to re-eclipsify your app in order to fix the Eclipse project classpath to point to the new 2.0.2 directory.

Play2 Port Change

2012-05-25T07:07:00.000-07:00

Play 2.0.1 defaults the port to 9000. But if you want to develop more that one app at a time you need to be able to change the port. Googling this you'll find the recommendation for typing in command line:
play "run 9001"
But this does not work in Windows. Until they fix this you need to do this in two steps. First enter the Play console with a simple:
play
then while in the Play console you can run the application with the port of your choice:
[appName] $ run 9001

Play2 Project Dependency

2012-05-25T06:26:00.001-07:00

I have recently started to use Play framework, and I quite like it. It may have its issues, but all in all it is a neat framework. Like many other developers out there I want to use it from within my favourite IDE - Eclipse. And Play makes that happen via it "eclipsify" command.
All is good, when you are working on a single application. But what happens when you want to create more than one application and you realize that you don't want to copy the boiler plate code in each of the project. Take the security feature for example. Each app will have to implement it and Play does provide the basics but there is still some code to be written. And I don't want to copy the file(s) each time a I create a new app.
Now Eclipse does have a neat feature where you can create project dependencies which means that all the classes from the project you depend on are added to the classpath of your current project. Great! That works fine for regular Java projects, but Play does its own building behind the scenes which has nothing to do with Eclipse's build.
There is however a way out. And it is quite simple. You see, each Play 2.x project will create the .class files in the [project_dir]/target/scala-2.9.1/classes. (I suspect that the location will change later, when the scala version is updated).
Now let's say you have a project called "common" where you add all the utilities and boiler plate code. Then you have "myApp" where you want to use the code from "common" without copying it. In Eclipse, you go to Project Properties for "myApp", "Java Build Path", select "Project" tab and add the "common" project in the list of required projects. That takes care of the Eclipse side.
To make it work in Play, there is one other thing you need to do. Locate "Build.scala" file from "myApp" project (this will be in myApp/project/ directory). This is the file that controls the Play build. you want to modify it and add the classes generated by "common" project to both runtime and compile classpaths. This is how you can do it:

val wsdir = (new java.io.File("./..")).getCanonicalPath // workspace dir
val appDependencies = Seq(
// Add your project dependencies here,
)

val main = PlayProject(appName, appVersion, appDependencies, mainLang = JAVA)
.settings(
dependencyClasspath in Runtime ++= Seq(file(wsdir+"/common/target/scala-2.9.1/classes"))
)
.settings(
dependencyClasspath in Compile ++= Seq(file(wsdir+"/common/target/scala-2.9.1/classes"))
)

Note that this code snippet assumes that projects "myApp" and "common" share a common parent directory.

And that's it for development. Creating the final build for production is another story and that depends on how you want to deploy your projects.

Single Sign On with SAML2

2012-05-24T12:49:00.000-07:00

This is a complicated topic with a lot of information published in the form of articles all over the internet. While SSO can be done in various ways and with different technologies, it seems like the standard that has the most traction is the Web Browser SSO Profile in SAML2.
I find it odd that I could not find a good SSO or SAML book anywhere. Which makes it difficult for a beginner to tackle. There is a pretty good description of the process on Wikipedia though.
Well one has to start somewhere so, being in the Java camp I looked for the options available to the Java programmer. As it turns out, if you want to go the SAML way there is an open source solution called Shibboleth. This is based on OpenSAML
There is even a test Identity Provider (IdP) available on the internet: TestShib. However, since I was working on the company network behind firewalls I could not use it. As part of the sequence of events, the IdP needs to redirect to the Service Provider (SP) and that part did not work for my setup.
As such I downloaded and installed Shibboleth IdP in a Tomcat instance and gave it a go. I must say that while there isn't much code involved (or that complicated), what proved to be the most difficult part was the configuration of Shibboleth IdP and the metadata file. Once you get that right as well as the certificates, it is not much to it.
So while there are examples out there (perhaps better written or more complete) I created my own and put it in GitHub. Check it out here: https://github.com/cduicu/SAML2Authn.

Just started!

2008-07-10T09:50:00.000-07:00

I've nothing to say now ... just created the blog :).