Ironic programmer

Scala Stock Charts - part 6 some GUI things

2011-10-19T11:02:00.001+02:00

I got a bit caught up in all the programming and finished up a few diagrams, then some refactoring and was able to remove a couple of hundred lines of code. The program now looks like this:

As you can see a lot have happened since the last pictures of the application. So here is some things.

Main part

The applications is build up around a main frame that contains the bar chart and some overlays. As you can see there is also some text fields and buttons. The upper part is to search stocks/indexes and also handle the overlays in the bar char. These fields and buttons are:

Search field to enter a search criteria to filter the instruments to search for.
Search button, to apply the search
A combo box, to show the filtered search result, and also choose the stock to view. This combo box listen to changes and updates the bar chart with the selected stock
Next there is a box to limit the range of the data. I have not introduced recalculation for splits and emissions so to have a possibility to handle to not show before these events, I added a box to filter away prices before this date part.
Next there is the intervals for the moving averages in the chart, default they are set to 5,20,50 but can be changed but have as most 3 different, but also as few as one. Default they are Simple Moving Average (SMA),
to change to Exponential Moving Average (EMA) there is a checkbox
to update changed settings for Moving averages there is a button "Update averages"
finally on the top there is a check box to switch on/off viewing of "Bollinger Bands"

In the bottom part there is buttons for additional chart types. these are the ones I picked as most interesting for me and they show some different aspect of the prices.

Stocastics (show how a stocks price is doing in relation to past movements)
RSI (shows how strong a stock is moving in the current direction)
Aroon (shows if a stock is trending or oscillating)
MACD (and also MACD-histogram) (a momentum oscillator)

As I mentioned the main part is a Bar chart diagram. It contains some overlays like volume, moving average and optional Bollinger bands. These help to get a basic overview of a stocks movement and trend. It is made up of a specialized panel for the bar chart to handle the drawing of the chart.

The main application implements a method named top that is used in wrapped Swing features used in Scala. I have an object representing my starting point of the application, the first lines looks like:

object ScaBorsApp extends SimpleSwingApplication with Logging{
    def top = new MainFrame {
      title = "Technical Analysis for Stocks"
      val COL_GREEN = new Color(0,150,0)
      val searchField = new TextField {
          text = "a"
          columns = 10
        }

As you can see I provide a title and also here is the search field where to put a search string. The most important thing to note here is the inheritance of SimpleSwingApplication which makes this object a Swing application

The next part looks like this

/*
       * Search button, will get the pricelist and do filter and sort to present in combo box
       */
      val searchBtn = new Button("Search") {
          action = Action("Search"){ 
            val pricelists = PriceService.getPriceLists map { kv => kv.instruments } flatMap { 
                case i: Instrument => i 
                case x => x
              } filter {
                i: Instrument => i.name.toLowerCase().startsWith(searchField.text.toLowerCase)
              }
            
            val items = pricelists.toList sortWith {(i1,i2) => i1.name.toLowerCase < i2.name.toLowerCase}
            combo.peer.setModel(
              ComboBox.newConstantModel(
                  for{p <- items} yield p.ticker + ": " + p.name
              )
            )
          }
        }
        
      /*
       * ComboBox to present filtered search for stocks, see searchButton action code how search is performed
       */
      import event._
      val combo = new ComboBox(List("-- Search to be able to select --")) {
        preferredSize = new Dimension(300,20)
        listenTo(selection)
        reactions += {
          case SelectionChanged(c) => 
            curInstrument = new Instrument(selection.item.split(":")(0),selection.item.split(":")(1).trim,"")
            curPriceList = PriceService.getInstrumentData(curInstrument).filter(p => p.date.substring(0,fromTextBox.text.length).toInt >= fromTextBox.text.toInt)
            barchartPanel.updateList(curPriceList)
          case e => info(e.toString)
        }
      }
...
val chkBollinger = new CheckBox {
        text = "Bollinger bands"
        reactions += {
          case x => barchartPanel.setBollinger(this.selected) 
        }
      }

Here you can see some on the fly declaration and reactions to events on the components. The following code shows how to add these to the contents of other panels. you can also see the creation at the click of the button, creation of a generic diagram for "stochastics" mixed in with a DrawListener to make it drawable. Also how the series of stochastics is added to a generic diagram can be seen here.

 
     /*
       * Top menu panel.
       */
      val menu = new FlowPanel {
        contents += new Label("Stock")
        contents += searchField
        contents += searchBtn
        contents += combo
        contents += fromTextBox
        contents += avgSettings
        contents += chkExpAverage
        contents += btnUpdateAvg
        contents += chkBollinger
      }
....
val bottomPanel = new FlowPanel{
        contents += new Button(){
          text = "Stochastics"
          reactions += {
            case bc: ButtonClicked => val stD = new Frame {
                title="Stochastics " + curInstrument.name
              }
              stD.open()
              val panel = new GenericDiagram with DrawListener
              val scrPan = new ScrollPane{
                preferredSize = new Dimension(1180,220)
                contents = panel
              }
              stD.contents = scrPan
              val stoList = PriceUtil.stochastics(curPriceList, 14)
              panel.addSerie(ChartSerie(stoList,Color.blue))
              panel.addSerie(ChartSerie(PriceUtil.slidingAvg(stoList, 3),Color.red))
              panel.addDateSerie(curPriceList.map{p=>p.date})
              panel.setLabelLevels(List(20d,50d,80d))
              // need to force set max, it is not always 100 ??
              panel.maxVal = 100d
          }
        }
....
contents = new BorderPanel() {
        add(menu, BorderPanel.Position.North) 
        add(scrollPane, BorderPanel.Position.Center)
        add(bottomPanel, BorderPanel.Position.South)
      }

This last part of the code adds the actual panels containing the fields, bar chart and buttons to the Swing application.

Additional panels

To add some extra check to the main part there is some diagrams that first was their own specialized panel, but after some refactoring they now all except MACD uses a generalized panel called GenericDiagram and here is a part of that code:

And here is addition of some of the buttons into menu panel (top panel) followed by some example of the bottom panel with addition of button and reactions to its event. You see here that here is where the panel frame is set up inside the reaction to show the GenericDiagram with "stochastic" data.

    val chkBollinger = new CheckBox {
        text = "Bollinger bands"
        reactions += {
          case x => barchartPanel.setBollinger(this.selected) 
        }
      }

...

trait BasicChartProps {
  val PAD_TOP,PAD_BOTTOM = 20
  val PREF_H = 200
  val PREF_W = 1180
  var maxVal = Double.MinValue
  var minVal = Double.MaxValue
  var startX = 50
}
/**
 * Generic panel for simple series
 */
class GenericDiagram extends Panel with BasicChartProps with Logging{
  private var series = List[ChartSerie]()
  private var dateSerie = List[String]()
  
...

Not so much to mention really except that as you see I extend with a trait with basic chart properties. this is intended to have common properties for this and other diagrams. The var's is to be able to add series and date series to the diagram. So this panel is only responsible to draw the things that are sent to it.

Ability to draw

No bar chart is complete if it is not possible to draw some help lines. This is managed by adding a trait for this, that can be mixed in wherever I want it. The code for that trais is:

...
/**
 * Trait to use for Panels that should have line drawin possibilities
 */
trait DrawListener extends Panel {
  val pointZero = new Point(0,0)
  
  var curStart = pointZero//new Point(0,0)
  var curEnd = new Point(0,0)
  listenTo(mouse.clicks, mouse.moves, keys)
  reactions += {
    case e: MousePressed  => 
                curStart = e.point
         //moveTo(e.point)
         requestFocusInWindow()
       case e: MouseDragged  => curEnd = e.point; repaint() //lineTo(e.point)
       case e: MouseReleased => moveTo(curStart);lineTo(e.point);curStart=pointZero;curEnd=pointZero
       case KeyTyped(_,'c',_,_) => 
         path = new geom.GeneralPath
         repaint()
       case _ => repaint()
  }
  /* records the dragging */
  var path = new geom.GeneralPath

  def lineTo(p: Point) { path.lineTo(p.x, p.y); repaint() }
  def moveTo(p: Point) { path.moveTo(p.x, p.y); repaint() }
  
  override def paintComponent(g: Graphics2D) = {
    super.paintComponent(g)
    g.setColor(Color.black)
    if(curStart.x != 0 && curEnd.y != 0){
      g.drawLine(curStart.x,curStart.y,curEnd.x,curEnd.y)
    }
    g.draw(path)
  }
}
...

An example of this functionality is to draw help lines in a MACD diagram. (or any really)

This can be mixed in like in this case:

val panel = new MACDPanel(Nil) with DrawListener

And then it is possible to draw on that panel.

Scala stocks part 5 - Refactoring, big time

2011-10-01T18:39:00.000+02:00

There is soooo much stuff that I would like to look in to. Alfresco, Liferay, GWT, Spring, Scrum, Agile etc. Right now though, I continue with my Scala project doing a Technical analysis tool for stocks using a Swing gui, also that with Scalas Swing libraries.

After working with the GUI, I started to get annoyed on myself mixing things up here and there, so I decided to refactor a bit. Actually, since I wrote the last post, I got a bit further then just plain refactoring. Simultaneously with the refactoring I worked on the GUI and also restructured code to finish up the loading of prices, enhancement of that loading and in that work I also managed to introduce some "Actors".

I will now also start to explain a little bit more what the code does.

I will split the refactoring and explanation in more than this post, otherwise I may get too long.

Refactoring of the FileHandler.
The new file is now much shorter, and includes only stuff to handle files. The extraction and transformation to the model is removed from the file handling. that messy code can be seen here http://ironicprogrammer.blogspot.com/2011/08/scala-stock-charts-part-3-saving-quotes.html if you have not already read it.

FileHandler.scala

/*
 @author Peter Johansson / Ironic programmer 
 */

package me.ironic.scabors.file
import me.ironic.scabors.model._
import me.ironic.scabors.util.Logging

object FileHandler{
  def apply() = { new FileHandler() }
}

class FileHandler extends Logging {
  import java.io.{File,BufferedWriter,FileWriter,IOException}
  import scala.collection.mutable.{ HashMap => MutableHashMap}
  import scala.collection.immutable.{ HashMap }
  import scala.io.Source
  
  val rootDir = { scala.util.Properties.userDir }
  val filesDir = { rootDir + "/files"}
  val logFile = { rootDir + "/" + "sBors.log"}
  val pricelistsFile = { rootDir + "/files/pricelists.txt"}
      
  def checkDir(dirName: String){
     val f = new File(dirName)
     if(!f.exists){
       f.mkdirs
       info("!! Created dir at : " + dirName)
     }
  }
  
  def writeToFile(fileName: String, append: Boolean, rows: List[String]){
    // Check that dir exista or creat if not
    checkDir(filesDir)
    
    val bw = new BufferedWriter(new FileWriter(fileName,append))
    
    rows foreach { row =>
      bw.write(row + "\n")
    }
    
    bw.close()
  }
  
  def priceFile(ticker: String) = {
    rootDir + "/files/" + ticker.replace(" ","_") + ".txt"
  }
  
  def loadFile(fileName: String)  = {
    val fc = Source.fromFile(fileName)
    fc.getLines.toList
  }
  
}

In the start of the file, you see:

object FileHandler{
  def apply() = { new FileHandler() }
}

This is a companion object I use it to have a factory method. Maybe not really nescessary but at least I get away with not having to write
new FileHandler()
every time I need it and can use the shorter version
FileHandler()

The rest of the FileHandler should be quite easy to figure out. I have only one write-method where I send in the filename, if I want to append and a list of strings to write to the file. With this I can do all the saving I need.

There is also only one method for getting the file content (loadFile), this returns a list of strings which I later can traverse and to all the logic with.

I also have a method to have a format for how a price file is named. I do not want any blanks in the filename so this method is replacing that with "_" instead.

As you can see the FileHandler extends Logging. I do not need that here since no logging is going on in this class as it is now, but I can add it if I want. More about the Logging later.

Introducing PriceDAO.
To hide FileHandler from the rest of the system, and also to the mapping between files and model that was before done in the FileHandler, I introduced a DAO object to do the work. Doing this I can always change the way I persist later without too much impact on the code.

There is still some refactoring to do in this one since I have not changed the fact that I use mutable lists.

PriceDAO.scala

/*
 @author Peter Johansson / Ironic programmer 
 */
package me.ironic.scabors.dao

/**
 * Provides service methods and communicate with underlaying storage
 */
import me.ironic.scabors.util.Logging
import me.ironic.scabors.model._
import me.ironic.scabors.file.FileHandler
import scala.collection.mutable.{ HashMap => MutableHashMap}
import scala.collection.immutable.{ HashMap }
  
object PriceDAO extends Logging{
  
  def saveDailyPrices(prices: List[PriceItem]) = {
    info("## Saving daily prices")
       
    var lists = new MutableHashMap[String,List[Price]]
    var curList = "?"
    
    prices foreach { price =>
      price match { 
        case q: Price => { 
                            savePriceToFile(q);
                            lists(curList) = List(q) ::: lists(curList)
                          }
        case qList: PriceList => {
                                    curList = qList.name;
                                    lists += qList.name -> List()
                                 }
        case _ => warn(price.toString)
      }
    }
    
    // Save current lists to file...
    saveStockPriceList(lists)
  }
  
  /**
   * Method for saving a price to file
   */
  protected def savePriceToFile(price: Price) = {
    val fileHandler = FileHandler()
    val fName = fileHandler.priceFile(price.ticker)
    fileHandler.writeToFile(fName, true, List(price.toString))
  }
  
  /**
   * Method for saving an updated pricelist map to file.
   */
  protected def saveStockPriceList(pricelist: MutableHashMap[String,List[Price]]){
    info("## Saving pricelists")
    
    val sb = new StringBuffer
    sb.append("{ \"pricelists\": [\n")
    for{ (listname,prices) <- pricelist } {
      sb.append("\t{ \n\t\t\"listname\": \"" + listname + "\",\n")
      sb.append("\t\t\"stocks\": [\n")
      for{ price <- prices } {
        sb.append("\t\t\t{ \"ticker\": \""+price.ticker + "\", \"aktie\": \"" + price.aktie + "\"},\n")
      }
      sb.append("\t\t],\n")
      sb.append("\t},\n")
    }
    sb.append("}\n")
    
    val fileHandler = FileHandler()
    val fName = fileHandler.pricelistsFile
    fileHandler.writeToFile(fName, false, List(sb.toString))
  }
  
  /**
   * Method for loading price lists to a HashMap, 
   * each entry in the map is a list of Instruments
   */
  def loadPriceLists  : MutableHashMap[String, List[Instrument]] = {
    info("## Loading pricelists")
    var pricelists = new MutableHashMap[String, List[Instrument]]
    val ListExtractorRE = """.*"listname": "(.+)",.*\s*""".r
    val InstExtractorRE = """.*"ticker": "(.+)", "aktie": "(.+)".*\s*""".r
   
    /*
     * When hitting a PriceList, a new list begins
     */
    var curList = "?"
    val fileHandler = FileHandler()
    val fName = fileHandler.pricelistsFile
    val rows = fileHandler.loadFile(fName)
    rows foreach { line =>
      line match {
        case ListExtractorRE(listname) => val pList = new PriceList(listname)
          curList = pList.name
          pricelists += pList.name -> List()
        case InstExtractorRE(ticker, aktie) => val inst = new Instrument(ticker, aktie) 
          pricelists(curList) = List(inst) ::: pricelists(curList)
        case _ => None
      }
      
    } 
    pricelists
  }
  
  /**
   * Method for loading all prices for a specific instrument.
   */
  def loadInstrumentData(inst: Instrument) : List[Price] = {
    val fileHandler = FileHandler()
    info("Loading instrument " + inst)
    val fName = fileHandler.priceFile(inst.ticker)
    
    val PriceExtractorRE = """Price\(([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*).*\)""".r
    val pricelines = fileHandler.loadFile(fName)
    val prices = pricelines map { line =>
      line match {
        case PriceExtractorRE(date,ticker,aktie,diffKr,diffPerc,buy,sell,last,high,low,returnNo,returnKr) => 
          Price(date,ticker,aktie,diffKr.toDouble,diffPerc.toDouble,buy.toDouble,sell.toDouble,last.toDouble,high.toDouble,low.toDouble,returnNo.toDouble,returnKr.toDouble)
        
      } 
    } 
    prices.toList
  }
  
}

Some explanation maybe?

Well, I am not going into every line. I choose some parts that I think is interesting parts of Scala.

   
def saveDailyPrices(prices: List[PriceItem]) = {
/*....*/
     var lists = new MutableHashMap[String,List[Price]]
    var curList = "?"

      prices foreach { price =>
      price match { 
        case q: Price => { 
                            savePriceToFile(q);
                            lists(curList) = List(q) ::: lists(curList)
                          }
        case qList: PriceList => {
                                    curList = qList.name;
                                    lists += qList.name -> List()
                                 }
        case _ => warn(price.toString)
      }
    }

This code it possibly a good candidate to do some further refactoring on to try to make it immutable, but this is what is going on.

lists is mutable, because I need to add things in it, and I assume that I do not know what lists are coming in. (Even if they are pretty known from the file)
prices is a list of type List[PriceItem], this means that everything in that list is of something under type PriceItem but right now I am only interested in the types Price and PriceList
If it is a PriceList I know that a new price list is starting, I add this as a map key with an empty list as the map value.
The flow goes on and for every Price that comes up this is added in the current list.

The next chunk of interest...

    val ListExtractorRE = """.*"listname": "(.+)",.*\s*""".r
    val InstExtractorRE = """.*"ticker": "(.+)", "aktie": "(.+)".*\s*""".r
    /*....*/
    val rows = fileHandler.loadFile(fName)
    rows foreach { line =>
      line match {
        case ListExtractorRE(listname) => val pList = new PriceList(listname)
          curList = pList.name
          pricelists += pList.name -> List()
        case InstExtractorRE(ticker, aktie) => val inst = new Instrument(ticker, aktie) 
          pricelists(curList) = List(inst) ::: pricelists(curList)
        case _ => None
      }
      
    }

Here is the loading of the price lists. I will sow you part of the file so you know what we are dealing with.

{ "pricelists": [
        { 
                "listname": "OMX Stockholm Mid Cap",
                "stocks": [
                        { "ticker": "ORES", "aktie": "resund"},
                        { "ticker": "AF B", "aktie": "F B"},
                        { "ticker": "VNIL SDB", "aktie": "Vostok Nafta Investment"},
                        { "ticker": "WIHL", "aktie": "Wihlborgs"},
                        { "ticker": "WALL B", "aktie": "Wallenstam B"},

That is how it is saved. So seeinging that makes it easier to explain what the above code does.

first there is 2 "extractors", these are regular expressions. In Scala you can use the 3-qouted strings to make a regular expression just by putting .r after it. This expressions can then be used as types cases in pattern matching and is very powerful way to extract thing.
(""".*"listname": "(.+)",.*\s*""".r) is for extracting the name of a list. This matches on the line in the file beginning with any character and then "listname":
the next "extractor" is for matching an instrument. for convenience I write the file with a "ticker" on a new line to avoid complex regexps. So I only need to parse the file and match on these.
When matching a new list I put a new map entry with an empty list, and when hitting a "ticker" I just add that instrument in the current list. Similar to what is going on when saving them.

Major restruction of PriceLoad.

I more or less totally reworked the loading of the prices from the site they are available from

One thing I wanted to do was to use actors. Here is were I got loose on doing that. Well, thats a bit exaggerating since there is only one Actor involved here (and another one in the Logging) so not so big fuzz. Here is the code, and I get into some details after it.

PriceLoad.scala

/*
 @author Peter Johansson / Ironic programmer 
 */

package me.ironic.scabors.dao
import me.ironic.scabors.model._
import me.ironic.scabors.util._
// Singleton to handle loading of Prices
object PriceFetcher extends Logging {
  import java.util.Calendar
  
  import java.text.SimpleDateFormat
    val fmt = new SimpleDateFormat("yyyyMMdd")
      
    def stringDateList(c: Calendar,bDate:String) : List[String] ={
      if(fmt.format(c.getTime()) == bDate) {
        List(fmt.format(c.getTime()))
      } else {
        val c2 = c.clone().asInstanceOf[Calendar]
        c2.add(Calendar.DATE,1)
        fmt.format(c.getTime()) :: stringDateList(c2,bDate)
      }
    }
      
   def main(args: Array[String]) = {
      if(args.length < 1  || args.length > 2){
        println("Usage: PriceFetcher YYYYMMDD # for single day")
        println("Usage: PriceFetcher YYYYMMDD YYYYMMDD # for batch mode")
        sys.exit(1)
      }
          
      val list = {
        if(args.length == 1){
          List(args(0))
        } else {
          require(args(1).compareTo(args(0)) > 0)
          val dFrom = fmt.parse(args(0))
          val cal = Calendar.getInstance()
          cal.setTime(dFrom)
          stringDateList(cal, args(1))
        }
      }
      
      info("Will run fetching for list " + list)
      val fetcher = new PriceFetchActor
      fetcher.start
      list foreach { s =>
        fetcher ! Daily(s)
      }
      fetcher ! Stop
      // Need to stop the logger, otherwise there is a hanging thread
      import scala.actors.Actor.State
      while(fetcher.getState != State.Terminated){
        // to wait until all work is done
      }
      stopLogger
      sys.exit(0)    
   }
}

import scala.actors.Actor
import scala.actors.Actor._
import me.ironic.scabors.util.Logging
class PriceFetchActor extends Actor with Logging{
  import scala.io.Source
      
  val baseUrl = "http://thesite/"
  def act(){
    loop{
      react {
        case d: Daily => 
          info("Fetching prices for " + d)
          val uri = baseUrl + d.date + "k.txt"
          try{
            val lines = readUrl(uri)
            val handledLines = processLines(lines,d.date)
            info("Did lines : " + handledLines.size)
            PriceDAO.saveDailyPrices(handledLines)
          } catch {
            case x => warn(x.toString)
          }
        case Stop => 
          info("Stopping PriceFetchActor")
          exit()
        case _ =>
      }
    }
  }
  
  def readUrl(url: String) : List[String] = {
    val fc = Source.fromURL(url).getLines().toList 
    fc
  }
  
  val PriceExtractorRE = """([[^Ticker]^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t.*""".r
  val ListExtractorRE = """(OMX[^\t]+)\t+""".r
  
  // Should go through the lines from the file and make a list with jSon-like format. 
  def processLines(lines: List[String],date: String) : List[PriceItem] = {
    info("-> Processing lines")
    val instruments = lines takeWhile(s=> ! s.startsWith("Externa listan")) 
    instruments map { inst =>  
       inst match {
        case ListExtractorRE(list) => PriceList(list)//"{\"instrumentList\": \"" + list + "\"}"
        case PriceExtractorRE(ticker,aktie,diffKr,diffPerc,buy,sell,last,high,low,returnNo,returnKr) 
          => Price(date,ticker,aktie,diffKr.replace(",",".").toDouble,diffPerc.replace(",",".").toDouble,buy.replace(",",".").toDouble,sell.replace(",",".").toDouble,last.replace(",",".").toDouble,high.replace(",",".").toDouble,low.replace(",",".").toDouble,returnNo.replace(",",".").toDouble,returnKr.replace(",",".").toDouble) 
        case _ => InvalidPrice(date,inst.replace("\t","[tab]"))
      } 
    }
  } 
}

This is an application object with a main method so it is available to run from the command line. What it does is to check if you enter with one or two arguments. Entering with one argument will load for one day. Entering with two arguments will run for a date span. I do not really car if there is days that does not have any prices, the only thing that will happen is that a FileNotFoundException will be logged.

The date span wil be recursively calculated to a list of strings with the method stringDateList.

After getting the list with dates, whether it is one item or more, I am traversing that list and sending a Daily object to an actor to do the loading and saving of that day. When all dates are handled I send a Stop message so the actor can exit itself.

Since I am sending to actors the flow of the program just passes to the end of the main method so I have to check the state of the actor before I do the exit. (There is a hang otherwise, because there is a thread that is still running somewhere.) and I do not want to do a sys.exit() before since that kills the thread that is loading.

The messages that is sent is case object/class that is defined like:

case object Stop
case class Daily(date: String)

Other than the actor logic, the load and save is more or less the same as before. Just that now it is threaded and runs by an actor by sending messages to it.

Well, that was enough for this time. The GUI is more evolved also, but that is another story, another day. But a teaser to show how it looks so far:

Scala stock charts, part 4 - Woking with the GUI - 1

2011-09-24T23:49:00.003+02:00

I have been busy the last couple of weeks, or better saying, too lazy to sit home with my Scala Technical Analysis tool. Anyway, this weekend I decided to continue and this time to start from the GUI and start using the functionality done so far from a GUI.

I have also done som restructuring in the packages which now is:

package me.ironic.scabors.app
package me.ironic.scabors.service
package me.ironic.scabors.model
package me.ironic.scabors.file

even if I have not reworked the messy file-package. I will do that later. however, I finished up with loading the price list, and that code now look like this:

def loadPriceLists : MutableHashMap[String, List[Instrument]] = {
    log("## Loading pricelists")
    var pricelists = new MutableHashMap[String, List[Instrument]]
    val fc = Source.fromFile(pricelistsFile)
    val ListExtractorRE = """.*"listname": "(.+)",.*\s*""".r
    val InstExtractorRE = """.*"ticker": "(.+)", "aktie": "(.+)".*\s*""".r
   
    /*
     * When hitting a PriceList, a new list begins
     */
    var curList = "?"
    fc.getLines foreach { line =>
      line match {
        case ListExtractorRE(listname) => val pList = new PriceList(listname)
          curList = pList.name
          pricelists += pList.name -> List()
        case InstExtractorRE(ticker, aktie) => val inst = new Instrument(ticker, aktie) 
          pricelists(curList) = List(inst) ::: pricelists(curList)
        case _ => None
      }
      
    } 
    return pricelists
  }

I also broke out the model classes from the file package, and put it in another file called PriceModel.scala, you can also see that I am now using "Price" which is a better name than "Quote" so all references to Quote is now replaced with Price. The content of that file is:

PriceModel.scala

package me.ironic.scabors.model

// To be able to handle Iterator[PriceItem]
trait PriceItem 

case class InvalidPrice(date: String, content:String) extends PriceItem
case class PriceList(name:String) extends PriceItem
case class Price(date: String, ticker :String, aktie : String,
    diffKr : Double, diffPerc : Double, buy : Double,
    sell : Double, last : Double, high : Double,
    low : Double, returnNo : Double, returnKr : Double
  ) extends PriceItem
case class Instrument(ticker:String, aktie: String)

I also made a Service object so that the GUI does not need to know or depend on anything from the file package. So far as I have gotten with the GUI it just contains two methods, one to get all price lists and one to get the data for a chosen stock. Here is the code:

PriceService.scala

package me.ironic.scabors.service

object PriceService {
  import me.ironic.scabors.file.FileHandler
  val pricelists = {
    println("# loading pricelists")
    FileHandler.loadPriceLists
  }
  
  def getPriceLists = {
     // using already loaded pricelists, i.e loading is done in val...
     pricelists
  }
  
  def getInstrument(inst: String) = {
    FileHandler.loadInstrument(inst)
  }
}

GUI programming done in Scala

I decided to leave the idea of using Java FX that I had from the beginning and that I wrote in the beginning post. Instead I decided to use Scala Swing. Not so easy as it turned out, because first of all, it has been a couple of years since I last did any Java Swing programming and second of all, it was not very similar and not easy to transform Java Swing thinking to Scala Swing thinking. Well, it is less code, but surly, at this stage it looks really messy. Maybe it takes a while for my brain to melt Scala Swing programming and the fact that google got warm from all searching and few good examples does not help.

I am to tired to explain most of the code, maybe in a later post, but here is what the purpose if the GUI is so far.

Provide a text field to filter search for stocks "Instrument"
Provide a button to perform the search
Provide a ComboBox to present the filtered result from the search and to select stock to draw
Provide Panel for drawing a Bar Chart of selected stock

I have implemented and it works as I wand, except that the tedious job to provide the functionality to draw a bar chart is not done. Thea is a later exercise and a future post to describe. I let you see the messy code for this.

ScaBorsApp.scala

package me.ironic.scabors.app

import swing._
import java.awt.Dimension
import me.ironic.scabors.service._
import me.ironic.scabors.model._
object ScaBorsApp extends SimpleSwingApplication{
      
    def top = new MainFrame {
      title = "Technical Analysis for Stocks"
      val searchField = new TextField {
          text = "a"
          columns = 20
        }
       /*
       * Search button, will get the pricelist and do filter and sort to present in combo box
       */
      val searchBtn = new Button("Search") {
          action = Action("Search"){ 
            val pricelists = PriceService.getPriceLists map { kv => kv._2 } flatMap { 
                case i: Instrument => i 
                case x => x
              } filter {
                i: Instrument => i.aktie.toLowerCase().startsWith(searchField.text.toLowerCase)
              }
            
            val items = pricelists.toList sortWith {(i1,i2) => i1.aktie.toLowerCase < i2.aktie.toLowerCase}
            combo.peer.setModel(
              ComboBox.newConstantModel(
                  for{p <- items} yield p.ticker + ": " + p.aktie
              )
            )
          }
        }
    
      /*
       * ComboBox to present filtered search for stocks, see searchButton action code how search is performed
       */
      import event._
      val combo = new ComboBox(List("-- Search to be able to select --")) {
        preferredSize = new Dimension(300,20)
        listenTo(selection)
        reactions += {
          case SelectionChanged(c) =>
            curInstrument = new Instrument(selection.item.split(":")(0),selection.item.split(":")(1).trim)
            curPriceList = PriceService.getInstrument(curInstrument.ticker)
            barchartPanel.updateList(curPriceList)

        }
      }
      
      var curInstrument: Instrument = Instrument("---","Not choosen")
      var curPriceList: List[PriceItem] = Nil
      
      /*
       * Top menu panel.
       */
      val menu = new FlowPanel {
        contents += new Label("Stock")
        contents += searchField
        contents += searchBtn
        contents += combo
      }
      
    val barchartPanel = new BarChartPanel(curPriceList)
      val scrollPane = new ScrollPane {
        preferredSize = new Dimension(1180,700)
        contents = barchartPanel
      }
      

      contents = new BorderPanel() {
        add(menu, BorderPanel.Position.North) 
        add(scrollPane, BorderPanel.Position.Center)
      }
     
      preferredSize = new Dimension(1200,800)
    }
}

class BarChartPanel(var list: List[PriceItem]) extends Panel {
  import java.awt.Color
  background = Color.white
  preferredSize = new Dimension(1180,680)
  override protected def paintComponent(g: Graphics2D) {
    super.paintComponent(g)
    
  } 
  
  def updateList(newList: List[PriceItem]) = {
    list = newList
    println("Updating list : " + list)
  }
}

Yeah, thats right, I also gave up SBT, I am now using gradle to build. here is the code for that. build.gradle

build.gradle

apply plugin: 'scala'

repositories {
    mavenCentral()
}

dependencies {
    // Libraries needed to run the scala tools
    scalaTools 'org.scala-lang:scala-compiler:2.9.1'
    scalaTools 'org.scala-lang:scala-library:2.9.1'

    // Libraries needed for scala api
    compile 'org.scala-lang:scala-library:2.9.1'

    // Libraries needed for scala.swing
//    scalaTools 'org.scala-lang:scala-swing:2.9.1'
    compile 'org.scala-lang:scala-swing:2.9.1'
}


task createSourceDirs << {
  println "# Creating source dirs"
  ['src/main/scala','src/test/scala','src/main/java','src/test/java'].each { 
    println " - $it"
    new File(it).mkdirs()
  }
}

Until later..... ciao!

Web Fragment using Scala

2011-09-13T21:41:00.000+02:00

Do you want to use Scala in a Java web environment, or write web fragment using scala. Or maybe you would ask: why do it at all? The answer is, because you can!

Well my little spike succeeded without any severe headache, actually so well that I did not got any at all and there was never any risk to get one.

I used SBT to build my Scala code, probably I could use it to make the full application, but I used Netbeans to create a web projet for my spike.

Setting up for SBT.

create folder for the project, for example:
>mkdir webfrag
enter the directory and create and edit build.sbt for servlet programming
>touch build.sbt

Put the following in the build-sbt file

name := "WebFrag"
 
version := "1.0"
 
organization := "me.ironic"
 
scalaVersion := "2.9.1"
 
scalaSource in Compile <<= baseDirectory(_ / "src")
 
scalaSource in Test <<= baseDirectory(_ / "test")
 
libraryDependencies += "javax" % "javaee-api" % "6.0" % "provided"
 
libraryDependencies += "log4j" % "log4j" % "1.2.16" % "provided"
 
libraryDependencies += "junit" % "junit" % "4.8.2" % "test"

Create a Servlet in Scala in this way.
create src/ folder and create and edit a file for the servlet, for example src/WebFragHello.scala

You can write what you want, but to make a form of HelloWorld-like example fill it with for example:

package me.ironic

import javax.servlet.annotation._
import java.io._
import javax.servlet._
import http._

@WebServlet(name = "WebFragHelloServlet", urlPatterns = {Array("/hello")})
class WebFragHelloServlet extends HttpServlet {
        override def doGet(req: HttpServletRequest, res: HttpServletResponse) : Unit = {
           val user = req.getParameter("user") match {
                case s: String => s
                case _ => "John Doe"
                }
           val out = res.getWriter()
           out.println(toHtml(user).toString())
        }
        def toHtml(user: String) = 
        <html>
        <body>
          <h1>Hello {user}&lt/h1>
        </body>
        </html>

}

This code also "catches" the case that user parameter is not sent.A special feature seams to be that the scala code does not understand urlPattern as used in Java in the annotation, that is why I use {Array("/hello")} instead of {"/hello"}

Compile and package the code with SBT
>sbt compile
>sbt package

To make it work as a web-fragment: Put the compiled jar in WEB-INF/lib folder in a Servlet 3.0 server, you also need to put scala-library.jar in the server to be able to use scala objects. In netbeans you just need to add that jar-file to "libraries" folder in the test web project

Surf to .../context-root/hello or /context-root/hello?user=Ironic

Neet isn't it?

Scala stock charts, part 3 - Saving the quotes

2011-08-22T22:51:00.001+02:00

Going further in programming the functionality behind the technical analysis tool for stocks made in Scala we are going to save the daily quotes to separate file, we are also going to supply functionality to load the data for a stock, and save and read a price list divided in 3 lists.

The format of the files is a bitt different, the Quotes I will save and read as they are, but the price list will have a different format that will look as jSon and is saved in a way that will be easy to read with pattern matching. For the matter of fact also the Stock-quotes for a single instrument will be read and transformed to a list of Quote's with pattern matching.

FileHandler.scala

/*
 @author Peter Johansson / Ironic programmer 
 */

package scala.ironic.file

object FileHandler {
  import java.io.{File,BufferedWriter,FileWriter,IOException}
  import scala.collection.mutable.{ HashMap => MutableHashMap}
  import scala.collection.immutable.{ HashMap }
  import scala.io.Source
  
  val rootDir = { scala.util.Properties.userDir }
  val filesDir = { rootDir + "/files"}
  val logFile = { rootDir + "/" + "sBors.log"}
  val pricelistsFile = { rootDir + "/files/pricelists.txt"}
    
  def saveDailyQuotes(quotes: Iterator[QuoteItem]) = {
    log("## Saving daily qoutes")
    log("--- rootDir : " + rootDir)
    log("--- filesDir : " + filesDir)
    // Check that dir exista or creat if not
    checkDir(filesDir)
    
    var lists = new MutableHashMap[String,List[Quote]]
    var curList = "?"
    
    quotes foreach { quote =>
      quote match { 
        case q: Quote => { 
                            saveQuoteToFile(q);
                            lists(curList) = List(q) ::: lists(curList)
                          }
        case qList: QuoteList => {
                                    curList = qList.name;
                                    lists += qList.name -> List()
                                 }
        case _ => log("! " + quote)
      }
    }
    
    // Save current lists to file...
    saveStockPriceList(lists)
    
  }
  
  def checkDir(dirName: String){
     val f = new File(dirName)
     if(!f.exists){
       f.mkdirs
       log("!! Created dir at : " + dirName)
     }
  }
  
  def saveQuoteToFile(quote: Quote) = {
    val fileName = scala.util.Properties.userDir + "/files/" + quote.ticker.replace(" ","_") + ".txt"
    val bw = new BufferedWriter(new FileWriter(fileName,true))
    bw.write(quote + "\n")
    bw.close()
  }
  
  def saveStockPriceList(pricelist: MutableHashMap[String,List[Quote]]){
    log("## Saving pricelists")
    val bw = new BufferedWriter(new FileWriter(pricelistsFile,false))
    val sb = new StringBuffer
    sb.append("{ \"pricelists\": [\n")
    for{ (listname,quotes) <- pricelist } {
      sb.append("\t{ \n\t\t\"listname\": \"" + listname + "\",\n")
      sb.append("\t\t\"stocks\": [\n")
      for{ quote <- quotes } {
        sb.append("\t\t\t{ \"ticker\": \""+quote.ticker + "\", \"aktie\": \"" + quote.aktie + "\"},\n")
      }
      sb.append("\t\t],\n")
      sb.append("\t},\n")
    }
    sb.append("}\n")
    
    bw.write(sb.toString)
    bw.close()
  }
  
  def log(msg: String){
    println(msg)
    val bw = new BufferedWriter(new FileWriter(logFile,true))
    bw.write(msg + "\n")
    bw.close()
    
  }
  
  def loadPriceLists : HashMap[String, List[Instrument]] = {
    log("## Loading pricelists")
    val pricelists = new HashMap[String, List[Instrument]]
    val fc = Source.fromFile(pricelistsFile)
    val ListExtractorRE = """.*"listname": "(.+)",.*\s*""".r
    val InstExtractorRE = """.*"ticker": "(.+)", "aktie": "(.+)".*\s*""".r
   
    fc.getLines foreach { line =>
      line match {
        case ListExtractorRE(listname) => println("#### got listname : " + new QuoteList(listname))
        case InstExtractorRE(ticker, aktie) => 
          println("# got instrument : " + new Instrument(ticker, aktie))
        case _ => 
      }
      
    }
    return pricelists
  }
  
  def loadInstrument(ticker: String): List[QuoteItem] = {
    val fn = rootDir + "/files/" + ticker.replace(" ","_") + ".txt"
    val fc = Source.fromFile(fn)
    val QuoteExtractorRE = """Quote\(([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*).*\)""".r
    println(QuoteExtractorRE)
    val quotelines = fc.getLines// filter( s=> ! s.startsWith("Quote")) 
    val quotes = quotelines map { line =>
      line match {
        case QuoteExtractorRE(date,ticker,aktie,diffKr,diffPerc,buy,sell,last,high,low,returnNo,returnKr) => 
          Quote(date,ticker,aktie,diffKr.toDouble,diffPerc.toDouble,buy.toDouble,sell.toDouble,last.toDouble,high.toDouble,low.toDouble,returnNo.toDouble,returnKr.toDouble)
        case _ => InvalidQuote("?", line)
      } 
    } 
    return quotes.toList
  }
  
  
}

That is a lot of code that I wrote on the fly, but not as much as it would have been to do the same thing in Java. the nice thing is the pattern matching, that saves a lot pain that Java code would bring, for example reading every line in the file would not be more code, but also creating and getting the values to objects would give a lot of more code.

[coming... Java code to do it as a comparison]

checkDir: is a method to create directory structure if it does not exist. Actually the first time loading the quotes there is no /files dir if it is not created manually, to avoid crashing, and open upp for the possibility to later use some config to have the data files in a directory of choice this is to ensure that the supplied configured dir will be created.

saveDailyQuotes: traverse the content of a daily quote file and calls to save single quote and collect the price lists and calls to save these.

saveStockPriceList: saves the current instruments divided in their lists to a file.

saveQuoteToFile: append a single quote to its data file.

Finally the methods for loading price lists and a single instrument

loadPriceLists: loads the pice lists back to a map with a list name and a list of Instruments. (Actually it is note yet finished, since I want to think out a way to avoid having a mutable HashMap, now it only prints out what it gets.)

loadInstrument: loads the file of a single stock to a list of Quote's

I am not completely happy with it, there is some duplication of code, for example I create new Writers in every method that save things. There is certainly things that can improve here. However, refactoring is a later issue, and this does what I want it to do, so for now it will have to do.

Now the interesting part is beginning to appear, next thing is to make functionality to take car of the technical stuff for the charts to be drawn.

Scala stock charts, part 2 - Getting the quotes

2011-08-19T16:46:00.001+02:00

I am Swedish so I am for the moment only interested in the Swedish stock market. I have found a site where to get the quotes, it is in text format and I intend to read this information and save it in in other files, one file for each instrument (I call on single stock instrument since it is a form of financial instrument).

The format of the daily file is: (There is an example file to work with at : http://sites.google.com/site/ironicprogrammer/files/20110817k.txt)

Slutkurser från OMX 2011-08-17 Slutkurser från Burgundy 2011-08-17
Ticker Aktie +/- +/-% Köp Sälj Senast Högst Lägst Oms (antal) Oms (SEK) Senast Högst Lägst Oms (antal) Oms (SEK)
OMX Stockholm Large Cap
ABB ABB Ltd -2,30 -1,66 136,40 136,50 136,40 139,40 136,40 3299500 453921900 136,6 139,4 136,6 183000 25240000

It is hard to see in that little box, however. first row is title information of the file.
Slutkurser från OMX 2011-08-17[6 tabs]Slutkurser från Burgundy 2011-08-17

That is not really necessary and can be skipped as the next line, but the next line is interesting to see the structure of the quotes:

Ticker Aktie +/- +/-% Köp Sälj Senast Högst Lägst Oms (antal) Oms (SEK)[6 tabs]Senast Högst Lägst Oms (antal) Oms (SEK)

When reading the file I will only be interested in the quotes from OMX, the Burgundy I do not want so when saving the quotes to different files I will not consider these, so it is the blue marked fileds that would be used.

The third row

OMX Stockholm Large Cap

is the name of the first list of quotes. It continues like this in the file;
Listname
Ticker Aktie ...
...
Listname ...

Ticker Aktie ...
...

I am only interested in the first three lists, the forth is named "Externa listan", and I will use this name to break the read.

So lets start write som code (finally!!)

I use the REPL for a little bit trial before I put in in classes/objects.

To test to read the file:

scala> val url = "http://sites.google.com/site/ironicprogrammer/files/"
url: java.lang.String = http://sites.google.com/site/ironicprogrammer/files/

scala> val file = "20110817k.txt"
file: java.lang.String = 20110817k.txt

scala> import scala.io.Source
import scala.io.Source

scala> val fc = Source.fromURL(url+file)
fc: scala.io.BufferedSource = non-empty iterator

scala> fc.getLines foreach { println }
Slutkurser fr?n OMX 2011-08-17 Slutkurser fr?n Burgundy 2011-08-17
Ticker Aktie +/- +/-% K?p S?lj Senast H?gst L?gst Oms (antal) Oms (SEK) Senast H?gst L?gsOms (antal) Oms (SEK)
OMX Stockholm Large Cap
ABB ABB Ltd -2,30 -1,66 136,40 136,50 136,40 139,40 136,40 3299500 453921900 136,6 139,4 136,6 183000 25240000
ALFA Alfa Laval -0,80 -0,68 117,50 117,60 117,50 119,10 114,80 1121500 131304100 117,6 118,9 114,49223 5760000

Ok, everything seams to work so this need to be taken car of in some other way.

Lets put it in a bit more organized way:

I put a file in src/main/scala/ironic that is named QuoteLoad.scala with the following content:

QuoteLoad.scala

/*
 @author Peter Johansson / Ironic programmer 
 */

package scala.ironic

// To be able to handle Iterator[QuoteItem]
trait QuoteItem 

case class InvalidQuote(date: String, content:String) extends QuoteItem
case class QuoteList(name:String) extends QuoteItem
case class Quote(date: String, ticker :String, aktie : String,
    diffKr : Double, diffPerc : Double, buy : Double,
    sell : Double, last : Double, high : Double,
    low : Double, returnNo : Double, returnKr : Double
  ) extends QuoteItem

// Singleton to handle loading of quotes
object QuoteLoad {
   def main(args: Array[String]) = {
      
      loadQuotes("http://sites.google.com/site/ironicprogrammer/files/","20110817")
   }
   
   val QuoteExtractorRE = """([[^Ticker]^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t.*""".r
   val ListExtractorRE = """(OMX[^\t]+)\t+""".r
  
   def loadQuotes(url: String, date: String){
      import scala.io.Source
      
      val uri = url + date + "k.txt"
      val fc = Source.fromURL(uri) 

      val instruments = processLines(fc.getLines(),date)
      println (instruments)
      instruments foreach { inst => inst match {
        case ilist: QuoteList => 
            println(ilist)
        case q: Quote =>
            println(q)
        case _ => println(inst)//
        }
      }
   }
  
    // Should go through the lines from the file and make a iterable with QuoteItem's. 
    def processLines(lines: Iterator[String],date: String) : Iterator[QuoteItem] = {
      val instruments = lines takeWhile(s=> ! s.startsWith("Externa listan")) 
      instruments map { inst =>  
         inst match {
          case ListExtractorRE(list) => QuoteList(list)//"{\"instrumentList\": \"" + list + "\"}"
	  case QuoteExtractorRE(ticker,aktie,diffKr,diffPerc,buy,sell,last,high,low,returnNo,returnKr) 
            => Quote(date,ticker,aktie,diffKr.replace(",",".").toDouble,diffPerc.replace(",",".").toDouble,buy.replace(",",".").toDouble,sell.replace(",",".").toDouble,last.replace(",",".").toDouble,high.replace(",",".").toDouble,low.replace(",",".").toDouble,returnNo.replace(",",".").toDouble,returnKr.replace(",",".").toDouble) 
          case _ => InvalidQuote(date,inst.replace("\t","[tab]"))
        } 
      } 
    }
    
}

Ok there may be some features to explain here.
First of all, I make use of an singleton object here so I will be able to call from a terminal window. So it has a main method, but I also extract the functionality so the future client will be able to use the same methods to load quotes.

There is a trait so I can avoid getting Iterator[Any] and instead getting Iterator[QuoteItem] to be able to handle the case class objects only, when I later will save the quotes to different files.

QuoteList: is for holding the name of a list (there is different list of quotes according to its category, that is more or less the amount of trading that is done and the size of the companies)
Quote: is for holding the quote for the day for a specific share
InvalidQuote: is for holding (and logging for error checking) an invalid row from the file.

There is two regexp's that is handled later in pattern matching to extract the lists and quotes from the file. They look for the structure in the file for valid quotes and the simpler one is for extracting the list name that is only one text.

def loadQuotes(url: String, date: String): handles the load from the URL and calls a helper method to extract the valid quotes and lists.

This will have to do for a first version, later I will save these quotes to different files.

Scala stock charts, part 1 - Thoughts about structure

2011-07-31T14:41:00.000+02:00

Having read through almost halfway, and not even coming to the functional part, in the online book "Programming Scala" by Alex Payne (http://ofps.oreilly.com/titles/9780596155957), I am starting to get some ideas of what I want to try out in my new Technical Analysis program for stocks. For sure I would like to use SBT (https://github.com/harrah/xsbt/wiki) as a building tool and try to use "Specs" for tests (http://ofps.oreilly.com/titles/9780596155957/ScalaToolsLibs.html#ScalaSpecs).

To start with I want to build my program using either common classes or case classes, haven´t clearly figured that out, at least I want to use companion objects. Anyway, I want to get in touch with different features in Scala so there probably will be some file handling with regexp in Scalas pattern matching. At the same way as I want to get in touch with as many features as possible, I also want to keep it as simple as possible so there probably will be som rewrites during the course of my little project as I learn more. If possible I will use trait also, but probably after a rework. I will start more "java-like" with more OO-structure and then try to "functionalize" it.

I haven´t really figured out if I want o use JavaFX either. So I will try to build a library in Scala that ccan be used by client of choice, may be JavaFX but maybe also Flex or Swing. That I will figure out later.

More about this later....
and then probably with some code :)

Long time no see

2011-07-11T14:36:00.001+02:00

It has been a while since last post. Sorry about that! It has been a period of organizational problems (not as fun as programming problems that needs to be solved) that could have been written about, but to sit in the middle of a stormy change takes the energy away.

Anyway, this is what I have been doing:
I started looking at Scala. For me it looks really interesting and I think it is something that is worth looking in to since it joins well with Java and runs on the JVM. You also need to write less code to do the same thing as with Java.
To learn Scala I have small idea that I should take my old Java-Swing tool for technical analysis of stocks on the Swedish stock market, set it up to use Scala with a JavaFX-client for the GUI. In this way I intend to get into Scala and at the same time JavaFX a little bit. this little tool of mine I have had as a project to learn new programming languages for over 10 years now. From the first version in an Excel spreadsheet it has taken the way over Turbo-pascal, Visual C++, maybe I also had a C#-client for a while until I returned to the Java Swing client.

I also started to read a lot of agile books about modelling and developing and also a bit about Neuro Linguistic Programming but that is a completely different subject but may be good if you want to take a leading role and/or how to handle the users and other team members and just get better at asking questions as one area of self development.

I will go on, so maybe there will soon be a posting about Scala or JavaFX or both. Or something completely different.

Ciao

Problem with Grails on Glassfish v3?

2010-10-06T12:59:00.001+02:00

I almost gave up trying to run grails applications on Glassfish v3. Today I stumbled on a solution that was so easy that I should be ashamed I did not investigate a bit further. The problem is with some OSGi headers and is documented here.

Others have had the problem and the solution I found on nabble was pretty simple:
Add the following line in BuildConfig.groovy in your project that should run on a Glassfish v3.

grails.project.war.osgi.headers=false

This helps with the problem not being able to deploy a grails app on Glassfish v3 when you experience the following hints in the log on GF:

SEVERE: Module type not recognized
SEVERE: There is no installed container capable of handling this application com.sun.enterprise.deploy.shared.FileArchive@68237947

Mountain climbing as analogy to programming.

2010-08-20T12:21:00.002+02:00

I just had vacation and I do not know why, but suddenly I got to think that walking/climbing a mountain (as that is one of my favorite things to do at vacation) could be used as an analogy for programming and learning programming.

As a starting point we have a geek that have no physical ability to talk about (because geeks seldom do sports or go outside the house).

Start with easy tracks

To be able to climb a mountain you need to be fit. If you start with a very steep path without being fit, you will get exhausted or fall down and die.

Also keep the distance short so you do not overuse your muscles. you may want to take a hike to morrow as well and with aching limbs it will not be that easy or fun.

That is to programming: start learning an easy programming language, do a little bit of learning every day, and eventually you will get it. Doing to much makes your brain tired and you may be so bored with the stuff that you may quit and do something else.

Increase the difficulty gradually

Having started the tracking experience, it may be boring to do the same path every time and after a while your fitness will tend to stagnate and adapt to what you are doing. That is, walk longer paths, and also more steep paths, the body tends to adapt to be able to handle the demands, but not more than necessary. We need to increase the work load to be able to handle more and more.When we are fitter, we can handle more difficult paths. However, even if it is good to challenge yourself a bit, do not overdo it (you may fall down and die). When you climb the mountain higher you will get a more panoramic view, it is beautiful and you can also start to see the path you took, and maybe also see different paths you can take to get to the same place.

That is to programming: Do not just learn the basics, they are good to have but can not help you in the long run, specially not if you want to have it as a profession. do the basics, repeat them, but when you feel comfortable with them, increase the amount of code you write, and also the difficulty of the code: that is, start doing design work, object oriented programming etc. Knowing the basics may help you write simple routines, but climbing higher will give you a better view of the surroundings and starting to put things together or how to do different designs to get to the same result.

Walk different mountains

No we may not have made it to the top of the first one yet, but does that really matter? Not really, the last part of getting to the top of a mountain may be easy, but it is usually not. Take Matterhorn, or Mount Everest for example. A lot of people climb them, but not all of them go for the top. you can go to the mountain and go as far as you have the time to do, or as high as you have the strength to do. this does not mean that you do not have a meaningful experience or doing a bad work. You doing as much as you can do or are willing to do, and even if its it not to the top, you may reach much higher than most people will do in their lifetime.

If you think that you did a mountain as good as you can or that you did it long enough to get bored. Try another one, it will give you another view, different tracks. It may have paths that is easier then the first mountain, but also paths that is harder. Maybe it will get you higher quicker and let you see more, maybe the paths are harder but the view is more rewarding than the first one.

That is to programming: Learn different programming languages, the first one you learn may not be a very good one to use professionally, but it may be a good language for learning basic programming. The new language may give you more opportunities to solve things quicker than the first one you learned. It may also be harder to do stuff in it, but when they are done, it may be better, quicker to do what you want it to do. for example Java may be easier to learn and use than C++ but performance wise, some C++ programs do better. On the other hand Java is more platform independent than C++, so it may be a better option for some companies.
Different language can solve things easier than in other languages, or have better performance. the choice of language depends of the application you want to make. Learn more than one to widen your options and you will be better at your work.

So when do I get to the top?

Ok, lets be realistic. You may never get to the top. It depends on the mountain you decide to climb. Take Mount everest for example, it is the highest mountain we have, only a few made it to the top, and even fewer without injuries.

Take most mountains that is not a walk in the park, you may never reach the top, but you can get far enough to be satisfied with the experience and the rewards it give.

For programmers: not all programmers learn absolutely everything about one single programming language. they leave the top part for experts, that are fit to do the advanced stuff.

Anyway programming is not really about doing the most advanced stuff or the most dangerous stuff. It is about doing stuff that other can follow. someone is coming after you and may have to maintain stuff you did. If it is to difficult they may not be able to go all the way, or understand how the heck you were able to get up to that cliff. programming is about leading the way for other to follow, you take a path that is fairly easy, sometimes not so easy but still manageable, you draw a map so that other can take the same path and continue from there.

So what are you waiting for, go out and climb a new mountain today!

Struts2 on Glassfish v3

2010-05-04T19:05:00.002+02:00

OMG. It was not that trivial to get it going, at least not using Netbeans.

First trying to use maven since I need to have some EJBs, but I gave up after not being able to get the archetype for Struts2 do any project for me, first complaining about no commons-collections-3.2.1, downloading it and got a little bit longer, but then could not find anything for struts2.1.6.

So I went on creating a Struts2 project and got it to work by pointing out the jar files needed in Netbeans JavaEE webproject for Glassfish, YES there is a lot of jarfiles depending on each other since I also want to use TILES. Anyway, when I wanted the use the EJBs I had to create a EAR-project to hold everything together, what happens then?

Struts2 stops working, complaining of a lot of classes not loaded or cannot find. I spent almost all the afternoon before finally finding the problem.

Netbeans ant-script wants to copy every f...ing jar file that is not a TLD to the lib of the EAR, NOT into the Web applications WEB-INF/lib, which then makes sence for the errors in the log: when loading from that EAR/lib folder, it can not find struts-core classes since that jar file is in the Web-app because it includes TLDs,

So finally solved, hack the build-impl.xml and copy ALL jar files to the web-app so it works.Why Netbeans do this way I do not know, maybe someone can inform me?

Tutorials

2010-03-26T16:01:00.000+01:00

I have started a Java tutorial for starting up with Java. It can be found at my Google-site. The tutorial is basic Java.

http://sites.google.com/site/ironicprogrammer/tutorials/ironic-programmer-java-tutorial

I will continuosly add more stuff to the site. Even if the first lessons may be a bit text to read. I will try to do programming by examples.

Scary, but maybe true

2010-03-07T16:04:00.003+01:00

A week ago I read the blogpost "The Non-Programming Programmer" by Jeff Atwood at Coding horror about that many applicants for programming work can not program at all.

During my 10 years in the business I experienced a few myself. I wish I then had the examples to give and exclude some that can not program.

Some examples:

What you can see as a teacher

At my first job as a teacher at university, part of my job was to lecture Pascal and C. Included was laborations in the language learned. Here I could see some serious laziness or lack of logic. One of the first tasks used to be to print out a matrix of the numbers 1..100 in a table of size 10x10. Of course this task is to learn some basic structures in the language learned like iterations as for-loop and conditionals like if. This should be a fairly simple task, but one guy actually just did a

writeln("1 2 3 4 5 6 7 8 9 10");
writeln("11 12 13 14 15 16 17 18 19 20");

and so on.
Another example is the laziness of students. Some did not want to put in some work to learn the language, they just wanted to pass the class. There is different ways to do this. One is to try to get a group to add your name to the report, another way is to copy someone elses work. Now the stupid part of this, a teacher notice more than you may think. Someone that is not able to manage simple task and then all of a sudden turn up with a solution that looks pretty similar to the brightest student or that of someone that the teacher helped and put in some pattern of text that was put there as an example but is failry recognizable since it was a value you do not just put in.

About applying programmers

Here we have programmers(? or not) that actually have a bigger mouth then ability to program. They do good at an interview and get the job. Later it turn out.

that they do not at all know anything about Java, and the excuse is that they hoped that they soon shuould catch up and learn.
the one I heard from the boss that applied (this was early 2000 when the market was hot). That they heard that the company was looking and maybe this person could be a candidate because he did som Basic 10 years ago and was thinking maybe he could do some programming.
I have noticed, at an interview, think really carefull when people start to talk about project they have been in and saying: "In this project WE used this and that language/database." or "The application I worked is build in this and that technique and this database". The problem with this is that some uses the knowledge of others in their CV, just because they were in a project that had a very nice arhitechture does not mean they did anything of it.

Programmers at work

One consultant I met at a task where we came in to look at why the clients web build in asp was so slow. Well the consultant they had there that have been working on the web did not know any SQL so in 4 different frames in a framset did a full lookup on the resultset of a table looped until the key was found and then used anothere key to do the same on another table.
A programmer I worked with got the task to do some of the work when we were dividing an existing Session EJB into 3-4 smaller ones and at the same time change the medthods to return Collection of objects instead of a ResultSet. He actually did that, that is moved the code from the old one to the new one. What he missed was that the work is actually not finishe just because you move code, there is actually placed were you use the code, like a client that ALSO needs to be changed if you change the contract and move the methods it need. WHOOPS!
Another man I worked with said he knew Java EE after he went throug a book, downloading the examples and compiled and run throug them once. But, when he did some actual coding, he did not reuse the methods I already had done, he made new ones with other names.
Some programers seam to have the philosophy. "If it compiles and run it must be right". NO it is not. Making the compiler happy is easy, making the application work is not the same as keeping the compiler happy, it is logic not syntax.

So that post was very welcomed and will be used in the future.

While we are at it, try to solve this one with recursion:

You have a class called Org to represent a organization. The root organization is the starting point and every instance of the class holds its own suborganizations and a reference to its parent.

Org.java

public class Org{
    private long id;
    private Org parent;
    private List children;
    private String name;

    public Org(long id, String name){
     ...
    }

    // getters / setters
    ....
}

Your task is to print the organization tree starting from the root.

Please share your own stories in the comments.

File upload in JSF2 using base class

2010-03-05T18:48:00.008+01:00

Surfing the net for JSF2 resources can be a tedious task. Actually for file upload I actually found a couple of blogs describing how to do it the firs I found actually only (FileUpload with JSF 2 and Servlet 3.0) made a taglib based on the second one. And it includes maven which I do not use.

I had to remake the example a bit since I got question to provide a sample application. When extracting the code I noticed it did not work anymore. The reason is probably the one mentioned at this blog: http://mcatr.blogspot.com/2010/01/multipart-requests-and-jsf.html.To make it work again, I did a solution that I am not happy with but it works. It takes the idea from this blog, joined with some from the other blogs mentioned. The former blog states that a multipart request to JSF 2 do not work, because it can not extract parameters from that kind of request.

The problem is that for what I actually wanted it was not right and one was a little bit overkill. So as usual I took another approach. I got some helf from Uploading files with JSF 2.0 and Servlet 3.0 but that solution includes making a custom component and renderer, and is dependant on classes found in Uploading files in Servlet 3.0. I actually copied all the code and tried it...but I got a failure when loading the page. So I wanted something with less code. and simpler to use, and I have a case when I am going to store vCards in a database so a solution to save the file on disk means I have to modify code anyway.

I took the following approach:

Use standard html <input type="file" />
Use @ManagedBean
I first had a base class to handle the file and extended the Managed bean with this one, but since this stopped working I had to add:

A Wrapper to wrap multipart requests
A Filter to catch multipart request and transform them to the Wrapper type.

I also have a FacesUtil class to simplify getting request and parameters

Here is the code:

In the view (vCardForm.xhtml)

<h:form enctype="multipart/form-data">
 <input type="file" name="vcardFile"/>
 <h:commandButton action="#{vcardBean.uploadVCard}" value="Upload"/>
</h:form>

Class to represent the uploaded file (UploadFile.java)

I created a class to store some information about the file being upload, I can later choose if I want to store to disk or in a database, for example sending this to an EJB and the EJB descides what to do.

package theironicprogrammer.model;

public class UploadFile {
 private String fileName;
 private String contentType;
 privte byte[] fileData;
 
 public UploadFile(String fileName, String contentType, byte[] fileData){
  this.fileName = fileName;
  this.contentType = contentType;
  this.fileData = fileData;
 }
 
 public String getFileName(){
  return fileName;
 }
 public String getContentType(){
  return contentType;
 }
 public byte[] getFileData(){
  return fileData;
 }
 
 public void setFileName(String fileName){
  this.fileName = fileName;
 }
 public void getContentType(String contentType){
  this.contentType = contentType;
 }
 public void setFileData(byte[] fileData){
  this.fileData = fileData;
 }
}

The managed bean (VCardBean.java)

The interesting thing going on here is that i initialize the base class with the request for use of the request. Probably I could have done this in the constructor of the baseclass, but something you have to have to refactor.

package theironicprogrammer.model;

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.Serializable;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.annotation.PostConstruct;
import javax.ejb.EJB;
import javax.faces.bean.ManagedBean;
import javax.faces.bean.RequestScoped;
import javax.servlet.ServletOutputStream;
import theironicprogrammer.session.PersonUpdaterLocal;
import theironicprogrammer.http.MultiPart;

/**
 *
 * @author theironicprogrammer@gmail.com
 */
@ManagedBean(name="vcardBean")
@RequestScoped
public class VCardBean implements Serializable{
    @EJB PersonBean personBean;

    public String uploadVCard(){
        System.out.println("uploadVCard...");
        HttpServletRequest req = FacesUtil.getRequest();
        if(req instanceof MultipartRequestWrapper){
            MultipartRequestWrapper multi = (MultipartRequestWrapper)req;

            UploadFile uf = multi.findFile("vcardFile");
            if(uf != null){
                String fileExt = uf.getFileName().substring(uf.getFileName().lastIndexOf("."));
                System.out.println("Extension" + fileExt);
                // should actually check the extension, but no big deal for example
                personBean.saveVCard(uf, "testuser");

                FacesUtil.getFlash().put("messageCode", "vcard.upload.ok");
                FacesUtil.getFlash().put("fileName", uf.getFileName());
            }

        }
        return "vCardForm";
    }

    public String findVCard(){
        UploadFile uf = personBean.findVCard("testuser");
        if(uf == null) {
            FacesUtil.getFlash().put("messageCode", "vcard.upload.err.load");
            FacesUtil.getFlash().put("messageType", "errors");

        } else {
            ServletOutputStream output = null;
            try {
                output = FacesUtil.getResponse().getOutputStream();
                FacesUtil.getResponse().setHeader("Content-disposition", "attachment; filename=" + uf.getFileName());
                FacesUtil.getResponse().setContentType(uf.getContentType());
                ByteArrayOutputStream baos = new ByteArrayOutputStream();
                baos.write(uf.getFileData());
                FacesUtil.getResponse().setCharacterEncoding("UTF-8");
                FacesUtil.getResponse().setContentLength(baos.size());
                baos.writeTo(output);
                output.flush();
            } catch (IOException ex) {
                FacesUtil.getFlash().put("messageCode", "vcard.upload.err.load");
                FacesUtil.getFlash().put("messageType", "errors");
                Logger.getLogger(VCardBean.class.getName()).log(Level.SEVERE, null, ex);
            } finally {
                try {
                    output.close();
                } catch (IOException ex) {
                    FacesUtil.getFlash().put("messageCode", "vcard.upload.err.load");
                    FacesUtil.getFlash().put("messageType", "errors");
                    Logger.getLogger(VCardBean.class.getName()).log(Level.SEVERE, null, ex);
                }
            }
        }
        return "vCardForm";
    }

    public String removeVCard(){
        personBean.removeVCard("testuser");
        FacesUtil.getFlash().put("messageCode", "vcard.remove.ok");
        FacesUtil.getFlash().put("messageType", "messages");
        return "vCardForm";
    }

}

MultipartRequestWrapper

package ironicprogrammer.http;

import ironicprogrammer.model.UploadFile;
import java.io.IOException;
import java.util.Enumeration;
import java.util.Hashtable;
import java.util.Map;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletRequestWrapper;
import javax.servlet.http.Part;

/**
 *
 * @author theironicprogrammer@gmail.com
 */
public class MultipartRequestWrapper extends HttpServletRequestWrapper {
    private static final String CONTENT_DISPOSITION = "content-disposition";
    private static final String CONTENT_DISPOSITION_FILENAME = "filename";

    private Hashtable params = new Hashtable();

    public MultipartRequestWrapper(HttpServletRequest request) {
        super(request);
        System.out.println("Created multipart wrapper....");
        try {
            System.out.println("Looping parts");            
            for (Part p : request.getParts()) {
                System.out.println(String.format("Part name: %1$s, contentType : %2$s", p.getName(), p.getContentType()));
                for(String header : p.getHeaderNames()){
                    System.out.println("Header name : " + header + ", value : " + p.getHeader(header));
                }
                byte[] b = new byte[(int) p.getSize()];
                p.getInputStream().read(b);
                params.put(p.getName(), new String[]{new String(b)});
            }
        } catch (IOException ex) {
            Logger.getLogger(MultipartRequestWrapper.class.getName()).log(Level.SEVERE, null, ex);
        } catch (ServletException ex) {
            Logger.getLogger(MultipartRequestWrapper.class.getName()).log(Level.SEVERE, null, ex);
        }

    }

    public UploadFile findFile(String attrName){
        UploadFile uf = null;
        try {

            Part p = findPart(attrName);
            if(p != null){
                String fileName = getFilename(p);
                System.out.println("Filename : " + fileName + ", contentType " + p.getContentType());
                byte[] b = new byte[(int) p.getSize()];
                p.getInputStream().read(b);
                params.put(p.getName(), new String[]{new String(b)});
                uf = new UploadFile(fileName, p.getContentType(), b);
            }

        } catch (IOException ex) {
            Logger.getLogger(MultiPart.class.getName()).log(Level.SEVERE, null, ex);
        }
        return uf;
    }

    private Part findPart(String name){
        HttpServletRequest request = (HttpServletRequest)getRequest();
        Part p = null;
        try {
            for (Part part : request.getParts()) {
                if(part.getName().equals(name)){
                    p = part;
                    break;
                }
            }
        } catch (IOException ex) {
            Logger.getLogger(MultiPart.class.getName()).log(Level.SEVERE, null, ex);
        } catch (ServletException ex) {
            Logger.getLogger(MultiPart.class.getName()).log(Level.SEVERE, null, ex);
        }

        return p;
    }

    private String getFilename(Part part) {
        for (String cd : part.getHeader(CONTENT_DISPOSITION).split(";")) {
            if (cd.trim().startsWith(CONTENT_DISPOSITION_FILENAME)) {
                return cd.substring(cd.indexOf('=') + 1).trim().replace("\"", "");
            }
        }
        return null;
    }

    @Override
    public String getParameter(String name) {
      String [] values = getParameterValues(name);
      if(values == null || values.length == 0) {
        return null;
      }

      return values[0];
    }

    @Override
    public Map getParameterMap() {
      return params;
    }

    @Override
    public Enumeration getParameterNames() {
      return params.keys();
    }

    @Override
    public String[] getParameterValues(String name) {
      return params.get(name);
    }

}

MultipartRequestFilter

package ironicprogrammer.http;

import java.io.IOException;
import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.annotation.WebFilter;
import javax.servlet.http.HttpServletRequest;

/**
 *
 * @author theironicprogrammer@gmail.com
 */
@WebFilter(urlPatterns="/*")
public class MultipartRequestFilter implements  Filter{
    private static final String REQUEST_METHOD_POST = "POST";
    private static final String CONTENT_TYPE_MULTIPART = "multipart/";


    public void init(FilterConfig filterConfig) throws ServletException {
        
    }

    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
        HttpServletRequest httpRequest = (HttpServletRequest) request;
        if(isMultipartRequest(httpRequest)){
            System.out.println("Is multipart request.... wrapping it.");
            request = new MultipartRequestWrapper(httpRequest);
        }
        chain.doFilter(request, response);
    }

    public void destroy() {
        
    }

    public static final boolean isMultipartRequest(HttpServletRequest request) {
        return REQUEST_METHOD_POST.equalsIgnoreCase(request.getMethod())
            && request.getContentType() != null
            && request.getContentType().toLowerCase().startsWith(CONTENT_TYPE_MULTIPART);
    }

}

Session bean for handling the storage.

For the example this only stores it in the session.

package ironicprogrammer.session;

import ironicprogrammer.model.UploadFile;
import ironicprogrammer.utils.FacesUtil;
import java.util.HashMap;
import java.util.Map;
import javax.ejb.Stateless;

/**
 *
* @author theironicprogrammer@gmail.com
 */
@Stateless
public class PersonBean {

    /**
     * Saves a vcard for the person with supplied username
     */
    public void saveVCard(UploadFile file, String username){
        // Make use of session for example. No need to complicate with connection to filesystem or database
        System.out.println("Saving vcard for " + username);
        getVcards().put(username, file);
    }


    public UploadFile findVCard(String username){
        System.out.println("Getting vcard for" + username);
        return getVcards().get(username);
    }

    public void removeVCard(String username){
        System.out.println("Removing vcard for" + username);
        getVcards().remove(username);
    }

    private Map getVcards(){
        Map vcards = (Map)FacesUtil.getRequest().getSession().getAttribute("vcards");
        if(vcards == null){
            vcards = new HashMap();
            FacesUtil.getRequest().getSession().setAttribute("vcards", vcards);
        }
        return vcards;
    }
 
}

Finally the class to inherit to make it all work

This is now obsolete, but I keep it in case JSF will handle the mutlipart form in the future. But even then it may not be needed.

package theironicprogrammer.http;

import java.io.BufferedInputStream;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.Part;
import theironicprogrammer.model.UploadFile;

/**
 *
 * @author theironicprogrammer@gmain.com
 */
public class MultiPart {
    private static final String CONTENT_DISPOSITION = "content-disposition";
    private static final String CONTENT_DISPOSITION_FILENAME = "filename";
   
    private HttpServletRequest request;
    
    protected MultiPart(){
        
    }

    public UploadFile findFile(String attrName){
        UploadFile uf = null;
        try {
            
            Part p = findPart(attrName);
            if(p != null){
                String fileName = getFilename(p);
                System.out.println("Filename : " + fileName + ", contentType " + p.getContentType());
                byte[] b = new byte[(int) p.getSize()];
                p.getInputStream().read(b);
                uf = new UploadFile(fileName, p.getContentType(), b);
            }
            
        } catch (IOException ex) {
            Logger.getLogger(MultiPart.class.getName()).log(Level.SEVERE, null, ex);
        }
        return uf;
    }


    public HttpServletRequest getRequest() {
        return request;
    }

    public void setRequest(HttpServletRequest request) {
        this.request = request;
    }

    private Part findPart(String name){
        Part p = null;
        try {
            for (Part part : request.getParts()) {
                if(part.getName().equals(name)){
                    p = part;
                    break;
                }
            }
        } catch (IOException ex) {
            Logger.getLogger(MultiPart.class.getName()).log(Level.SEVERE, null, ex);
        } catch (ServletException ex) {
            Logger.getLogger(MultiPart.class.getName()).log(Level.SEVERE, null, ex);
        }

        return p;
    }

    private String getFilename(Part part) {
        for (String cd : part.getHeader(CONTENT_DISPOSITION).split(";")) {
            if (cd.trim().startsWith(CONTENT_DISPOSITION_FILENAME)) {
                return cd.substring(cd.indexOf('=') + 1).trim().replace("\"", "");
            }
        }
        return null;
    }
    
}

The only thing needed in the Managed bean is to call findFile("vcardFile") on the multipart request to get the file that is being uploaded and take action on that.

A sample project (for netbeans) can be found here: http://sites.google.com/site/ironicprogrammer/home/jsf-file-upload-example

Netbeans tutorial "Generating a JavaServer Faces 2.0 CRUD Application from a Database"

2010-01-10T18:45:00.000+01:00

I went through this tutorial today. Nice example, but there was something I did not like. As a geek I sit on a sunday and try out a tutorial on my home computer. Lazy as I am I do not want to start my MySql instance, since adding more running components tend to slow down my almost four year old Toshiba laptop.

Since I was going to run Glassfish v3 anyway I was thinking a better idea is to have the database in the form of JavaDB, so I tweaked the script to fit into Derby instead, not much tweaking to do actually and doing it made me happy to not be forced to start MySql (My mysql is by the way a bit old and installed the same time as I installed InstantRails long time ago, and probably needs to be updated). The insert script in the tutorial work more or less as it is in JavaDB just remove the 3 SET-clauses in the top and the COMMIT in the end and it should work.

The script can be found at my google-site http://sites.google.com/site/ironicprogrammer/home/jsf-2-0-tutorial-db-script

Quote to think about

2009-11-30T10:37:00.000+01:00

Started to read the book "Refactoring. Improving the Design of Existing Code" by Martin Fowler the other day. Here is a quotation from that book that is what programming is about:

"Any fool can write code that a computer can understand. Good programmers write code that humans can understand"

So tru, so tru...

Simple Ajax sample

2009-10-19T15:56:00.002+02:00

Using Ajax in Grails is quite simple. Grails have the prototype library making it easy to use Ajax in Grails.

You can try it out by creating a Grails-project. Here is an example ant some explanation of what is nescessary to think of.

1. Create a grails project:

grails create-app ajaxsample

2. Create a controller:

grails create-controller ajax

3. Create 2 .gsp-files in the /views/ajax folder; index.gsp and _itemList.gsp (_ means that it is a template)

4. Change the /views/layout/main.gsp by adding the following row

<g:javascript library="prototype" />

5. In the index.gsp in your ajax folder, put in this code:

<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
    <meta name="layout" content="main"/>
  </head>
  <body>
    <h1>Ajax samples</h1>

    <h2>Adding item to list:</h2>
    <i>This example takes a string in a textfield and add it to a list (saved in the session) generated with a template</i>
    <g:form name="listForm">
    Item : <g:textField name="anItem" value=""/>
    <g:submitToRemote action="addItem" update="listItems" value="Add"/><br/>
    </g:form>

    <div id="listItems">
    </div>
    
  </body>
</html>

6. In your _itemList.gsp put this code:

<table>
  <g:each in="${listitems}" var="item" status="i">
  <tr class="${i%2==0?"even":"odd"}">
    <td>${item}</td>
    <td><g:remoteLink action="deleteItem" params="[index:i]" update="listItems">Remove</g:remoteLink></td>
  </tr>
  </g:each>
</table>

7. Now the only thing we need is some code in the controller. Open the AjaxController and put this code:

class AjaxController {

    def index = { }

    def addItem = {
        if(session["listitems"] == null){
            // create new list if empty
            session["listitems"] = []
        }

        session["listitems"] << params["anItem"]

        // render partial with the updated list
        String content = g.render(template:'itemList',model:[listitems:session["listitems"]])
        render content
    }

    def deleteItem = {
        def idx = params["index"].toInteger()
        if(session["listitems"] != null){
            // remove item at index
            session["listitems"].remove(idx)
        }

        // render partial with the updated list
        String content = g.render(template:'itemList',model:[listitems:session["listitems"]])
        render content
    }
}

So what makes this work?

First of all, Grails supply the tags submitToRemote which is used for the "Add" button and remoteLink which is used for the "remove" link. These are used to use Ajax from grails and the only thing needed to make them work is to include the "prototype" javascript library, which we did in the layout to make it available on all pages. If you look on these tags you see the property action="" and update="listItems" this properties tells the framework what action to call and what to do with the result when the action is finished. Look at the index.gsp and you will see a <div> with the id="listItems", if you remember this is the same id that you saw as text in the update-property, this i the glue between the action pointed out and where to put the result. Pretty easy and straightforward, to make it short you need this

Add prototype JavaScript library to the layout
Add a DIV for the result
Add some remote call and point it to the action you want to happen and where to put the result

Note : Ajax is used for enhanced functionality and to not to have to reload pages. Therefor using templates with small fragments of code is a convenient way to reuse code-blocks and functionality. It is better to think a bit ahead of what is needed to make nice solutions and reuse of code.

Making SSL calls with Grails

2009-10-07T13:29:00.001+02:00

All of a sudden I realised that I have to call a service I have behind a SSL connection.

Using grails, this is easy to set up. Just create a couple of Groovy-classes and use it from the controller. It may be nescessary though to add the certificat of the SSL site to your JVM.

Adding the certificate

download the certificate you need, name it to something usefull, for example theSite.cer.
use keytool to add it. (needs JAVA_HOME/bin in the path)

keytool -import -keystore /pathToJava/jre/lib/security/cacerts -file theSite.cer  

Class to make the call
this is the class to actually do the work, calling the SSL url you want to reach.

import javax.net.ssl.SSLSocketFactory
import javax.net.ssl.SSLSocket
import java.security.*;

/**
 * @author theironicprogrammer@gmail.com
 */
class SSLConnector {
    String theurl
    String user
    String password
    public SSLConnector(url, user, pwd){
        this.theurl = url
        this.user = user
        this.password = pwd
    }

    public String responseString(){
        // Dynamic registration of JSSE provider
        java.security.Security.addProvider(new com.sun.net.ssl.internal.ssl.Provider());

        // Need to be set
     System.setProperty("java.protocol.handler.pkgs","com.sun.net.ssl.internal.www.protocol");

        // Install Authenticator
        Authenticator.setDefault (new SSLAuthenticator(this.user,this.password));


        URL url = new URL(this.theurl);
        BufferedReader br = new BufferedReader(new InputStreamReader(url.openStream()));

        String line;

        StringBuffer buff = new StringBuffer()
        while ((line = br.readLine()) != null){
            buff.append(line)
        }

        br.close();

        return buff.toString()
    }    

}

Class for password authentication
this class is needed to authenticate you at the remote site.

/**
 * @author theironicprogrammer@gmail.com
 */
class SSLAuthenticator extends Authenticator{
    String user
    String password
    public SSLAuthenticator(un, pw){
        this.user = un
        this.password = pw
    }

    protected PasswordAuthentication getPasswordAuthentication() {
        return new PasswordAuthentication(user, password.toCharArray());
    }
 
}

In the controller you can then use the class to get a responsestring from the remote server.

// suggest to put these in config.groovy to get them from there, but not nescessary
def url = "http://pathToSite/.../thePage?id=x" // or grailsApplication.config.sslConnection.url,
def username = "theUsername" // or grailsApplication.config.sslConnection.username,
def password = "secret" // or grailsApplication.config.sslConnection.pwd,

// Create the connector
def ssl = new SSLConnector(url, username, password)
def responsString = ssl.responseString()

// do something...

/Peter

Grails and EJB

2009-10-03T19:21:00.000+02:00

Read Jason Rudolphs article about grails and EJB a while ago. Grails + EJB Domain Models Step-by-Step. It is a nice article but not what I want.

However it seamed a little bit of inventing the wheel again to import the classes into Grails in the same package structure, and I also wanted some other things that EJB3 gave me.

I want the EJB level to handle application rules. In that way, I can reuse the EJBs from other places as well
I have information i two different schemas in Oracle and grails was not supporting more than one.
I like "hit and run" session beans to provide services. which means that I use interface in the sessionbeans to provide CRUD and listing methods.

So I started to elaborate with putting the EJB.jar file in Grails library folder to try it out. It worked out nice, the only thing I have to do is to put some lookup code and start using the session beans. I also get access to all model POJOs.

I like this approach because I can use grails rapid application development with the robustness of EJB3 without doing very much extra.

The downside is that as I am a little bit lazy and avoid writing too much code, is that I have to run it on the same glassfish and I have to copy the jar-file every time I change the interface. I also have to make a war of the grails application and run it on glassfish.

/Peter

Tweaking grails pdf plugin

2009-09-17T14:57:00.007+02:00

I had a little bit of problem when I wanted to use grails PDF plugin. (By the way Aaron Eischeid, thanks for providing this). However some things is not as wanted.

pdfLink: this is a nice feature for generating PDF by providing an url to the page to generate as PDF. But I do not want to provide parameters by adding them to the url in get-form.

pdfForm: Somewhat better, But this method gives some other problems. First of all, pictures is lost since the PDF only generates from string, also the layout is not automaticly included in the render call.

So I did the followin tweaks. Sorry Aron, but I had to hack into the PdfService.groovy and PdfController.groovy to get it work and also set up a template in a specific way.

This is what I did:
In PdfService.groovy I added a method to be able to generate a PDF with images for pdfForm, to do this the renderer need to know the baseUri to get find the image.

I added this code:

code

byte[] buildPdfDoc(content, baseUri) {
    DocumentBuilder builder = DocumentBuilderFactory.newInstance().newDocumentBuilder();
    Document doc = builder.parse(new StringBufferInputStream(content));

    ByteArrayOutputStream baos = new ByteArrayOutputStream();

    ITextRenderer renderer = new ITextRenderer();
    try {
      renderer.setDocument(doc, baseUri);
        renderer.layout();
        renderer.createPDF(baos);
        byte[] b = baos.toByteArray();
        return b
    }
    catch (Throwable e) {
        log.error e
    }
}

In the method pdfForm for the PdfController.groovy, I replaced code with call to the new method.

code

def baseUri = request.scheme + "://" + request.serverName + ":" + request.serverPort +
   grailsAttributes.getApplicationUri(request)
....
//b = pdfService.buildPdfFromString(content)
b = pdfService.buildPdfDoc(content, baseUri)

I then made a template with the following structure to also use the layout, without the applyTemplate, I only got the template code (also note that the template is looked for in views/pdf/):

code

<g:applyLayout name="main">
    <h1>title</h1>
    <b>something</b>
</g:applyLayout>

I have not done this with the options for controller and action, just for template, since the g.include seams to return empty content. But I will continue to explore.

! An update. I realized that the pdf-plugin was build for grails 1.1. maybe this was the reason that the g.render in the plugin gave med nothing as a result. I was actually using grails 1.0.4. Using 1.1.1 however makes the action and controller work better. No pictures thoug, but they show when I am running pdf-plugin as an application after downloading the source.

I will try to find the proper place to address these issues to improve the plugin.

/Peter

Creating jasper reports in Grails

2009-09-09T10:19:00.005+02:00

I like to work with Grails, so to continue making useful stuff i elaborated a little bit about how to make reports in Grails using Japser (which is a nice reporting tool)

My aim is to do reports in PDF and Excel. And since it don't think it is the f...ing business of the report file to know anything about the database I intend to use my own models to send into the report.

Create an application:
grails create-app bookshelf

Install the plugin:
grails install-plugin jasper
This usually takes a while since Plugin list cache is almost always out of date when installing plugins. So go take a coffe or something.

First of all I want a domain-class to hold books. So create that one and set up a few books in the BootStrap.groovy to have somthing to test with.
grails create-domain-class Book

Add a few properties to the book, for example:

class Book {
    String title
    String author
    String publisher
    String category
}

and add a few books in the BootStrap.groovy to have something

def init = { servletContext ->
         new Book(title:'agile modeling', author:'Scott W. Ambler', publisher:'Whiley Computer Publishing', category:'Modeling').save()
         new Book(title:'The J2EE Architect\'s Handbook', author:'Derek C. Ashmore', publisher:'DVT Press', category:'Architecture').save()
         new Book(title:'Getting Started with Grails', author:'Jason Rudolph', publisher:'InfoQ', category:'Programming').save()
     }

To create a report from Jasper, you need to create a .jrxml and then compile it to.jasper report file.

To create a report for my purpose, I used iReport. this is a tricky one because the jasper plugin which I got when installing is 0.9.7 and has a mistake in it that makes the reports fail. However to solve that look up the lib folder in the plugin-installment (using grails1.0.4 is in the project, but using 1.1.1 it ends up in your home folder) and remove the jasperreports-2.0.5.jar, otherwise you will end up with some NullPointerExceptions when tryring to run the report. Loocking there you see that the other jar file is jasperreports-3.1.2.jar, so the iReport version to download is 3.1.2.

I choosed the standalone version but there is a version that plugs into Netbeans.

Making the following report and compile it makes an .jasper file to use from grails.

Now things is starting to get a little bit interesting. We now need to make a report of this. I genereated all stuff for book by using the command:
grails generate-all Book
we can then use the generated views to present the report.

Add the following line in the view to get a PDF and a XLS report:

g:jasperReport action="createReport" controller="book" format="PDF" jasper="books" name="All Books"
g:jasperReport action="createReport" controller="book" format="XLS" jasper="books" name="All Books"

(of course they should be tagged, but this stupid blog editor does not want to show it if if make them as tags)

for example under pagination, I used an action in the controller and a little bit of of code, not much really, to to what is needed.

def createReport = {
        def books = Book.list()
        chain(controller:'jasper',action:'index',model:[data:books],params:params)
    }

That's it. Try it out yourself!

(For the lazy one, a zip of the project can be found here: http://sites.google.com/site/ironicprogrammer/home/grails-and-jasper-1 as an attachement)

GroovyWS or Java WS Client?

2009-09-04T15:43:00.000+02:00

We had a project this winter/spring which I designed to use Webservices on a web connected to EJBs, the clients we build in Grails for fast development.

Well we started using GroovyWS because that gave us the opportunity to get up and running pretty quick.

First we had some problems with the time it took to generate the stubs everytime we called a web service, however, this was becaus we stupidly put the creation of the client stubs in the actions so it was done every time. This was easy to sol by making a Groovy singleton class and let it create the service stubs once.

Eventually I decided to move over to making JAX-WS clients in Java using Netbeans. The reasons for this was mostly to try to get better performance and the possibility to put on SSL on the Web Service layer. From information I found and the experience during the first month of the project, I realised some shortcommings with GroovyWS mainly:

there is no support for https calls using GroovyWS
for some reason I had to wrap lists into a class. GroovyWS only regarded the list as one object and took the first one and skipped the rest and making workarounds in the WS-layer to fix this is something I don't want
restarting Glassfish sometimes made problems with the grailsapps and made them crash, this I think is because some of the services tokk some time to start and when GroovyWS tried to make stubs they were not there and after that the wars refused to work.

However making a WSClient in Java from Netbeans is an easy task, even if Java 1.6_07 or later is required for JAX-WS it is not such a big deal to make an jar-file and put it into the grailsapps lib folder, after that it is easy to call the java-classes from the Groovy code.

Grails i18n

2009-09-04T14:40:00.001+02:00

Got a little bit pissed of regarding language changing on Grails.

Of the information I can find about it it should be fairly straightforward. Just providing the parameter "lang=xx". In my experience this works sometimes, but sometimes something magical happen running on Glassfish for a while. The switcing of language stops working. Don't know if this is a known problem.

Anyway, for now I made a workaround using the following code:

def beforeInterceptor = [action:this.&checkLang, except:['']]

    def checkLang() {
        if(session["lang"] == null){
            session["lang"] = "sv"
        }
        if(params.lang){
            session["lang"] = params.lang
        }
       if(params.lang && params.lang=="en"){
            def defaultLocale = new Locale("en","US")
            java.util.Locale.setDefault(defaultLocale)
       }         return
    }

Which works for now. Should think there is a better solution. Feel free to inform me if you know.