Everything I See

Removing and Subsetting Repositories in git versus svn

2011-03-08T13:29:00.000-08:00

If ever you decide to take a bit repository apart and make smaller ones, do it in git. In subversion, you first make a dump of the whole thing, then you create a new repository and use svndumpfilter to filter out what you want to add and add it with svnadmin:

svnadmin dump svnroot/ > svn.dump
svnadmin create svnroot-fixed
cd svnroot-fixed
svndumpfilter include what-to-include < ../svn.dump | svnadmin load ./

This works alright, but it's pretty slow and for any kind of complex subsetting/removal, you're going to have to do multiple passes. In the end, since I was transitioning things to git, I gave up after a little while.

Git, although it doesn't make the process entirely trivial, is a bit better. The workflow is about the same, but it's faster and more flexible. You clone the directory and then use 'git filter-branch' to do what you want to do. For example to remove a directory (or specific files):

git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch $files" HEAD

Note that this won't yet delete the files from disk, since git keeps around a cache of the files. To totally delete them forever, David Underhill made a nice little script.

To make a copy of a repository only containing a directory:

git clone $PARENT $NEW_REPO
cd $NEW_REPO
git filter-branch --subdirectory-filter $CHILD -- --all

And in case it wasn't obvious from the variable names, I made a quick script to promote all the subdirectories of a repository to their own repository:

#!/bin/bash
WRKDIR=`pwd`
PARENT=$1
echo $PARENT

cd $PARENT
CHILDREN=`git ls-tree --name-only HEAD`

for CHILD in $CHILDREN; do
cd $WRKDIR
NEW_REPO="$PARENT-$CHILD"
git clone $PARENT $NEW_REPO
cd $NEW_REPO
git filter-branch --subdirectory-filter $CHILD -- --all
done

That's all for now.

Gotcha git

2011-03-08T13:17:00.001-08:00

So hopefully mentioning them here will help me remember not to do a couple of things.

First DO NOT, EVER, push to a non-bare repository, unless you will immediately 'git reset HEAD'. When you push to a non-bare repository, git for some reason inserts the pushed files "before" the existing working directory. This means that current working directory, without your pushed changes, is seen as having changes after your commit. So if you commit the working directory you pushed to, you effectively revert everything you just pushed!!!

Second, be very, very careful with submodules. Every commit to a submodule has to be committed both the the submodule AND to the parent module! If you to do both, everyone dies. Well, maybe not, but you all other clones of the parent directory will be broken until you go pack and do both commits.

Even worse are the terrible terrible consequences of 'git add submodule/' with the trailing slash included. Rather than add the changes in the submodule to the list of changes to commit, this will remove the submodule from the repository and re-add all of the individual files contained in the submodule. I'm not quite sure why this was determined to be the desired behavior, but whatever.

Anyhow, I thought I'd also throw up a couple of handy references for generic git stuff.

Managing user files with git

2011-03-08T12:19:00.001-08:00

So now that I'm moving some of my projects over to git, I decided I should better automate syncing of user files across machines. Specifically, I want (some of) my $HOME/.* files and (some of) my $HOME/bin/* files version controlled and sym-linked to where they normally are.

So I made a repository with a conf directory and bin directory and symlinked the files to their respective $HOME and $HOME/bin locations. But then I wondered if I could do something a bit better. Specifically, I was thinking about what would happen when I added a new file to the repository. Inevitably, things would be fine, but on at least one computer, I'd forget to symlink the new files and I'd be confused for a bit.

In order to prevent what is for all practical purposes, a non-problem, I spent mucking around with git-hooks and git's config options, I came up with a solution. Beginning with a simpler solution that merely asks which files to replace and symlink when its run, I came up with a more elegant (stupid) solution:

#! /bin/bash

# Move all existing files for which a versioned file exists to a local branch
# and symlink to the versioned location. But make sure to stash the current
# branch, commit the local branch after you're done and then move back to the
# original branch and pop the stash back to where it was
#
function move_to_branch () {
VERSIONED_DIR=$1
TARGET_DIR=$2
cd $VERSIONED_DIR
FILES=`git ls-tree --name-only HEAD`
for FILE in $FILES; do
# if isn't already a symlink that points to the the versioned copy
 if [ ! -h $TARGET_DIR/$FILE ] ||
   [ ! `readlink -f $VERSIONED_DIR/$FILE` ==
     `readlink -f $TARGET_DIR/$FILE` ]; then
#if it exists, remove the versioned copy in the local branch and replace with local file
 if [ -e $TARGET_DIR/$FILE ]; then
 rm -rf $VERSIONED_DIR/$FILE
 mv $TARGET_DIR/$FILE $VERSIONED_DIR/$FILE
 fi
 ln --force --symbolic $VERSIONED_DIR/$FILE $TARGET_DIR/$FILE
 fi
done
}


cd `dirname $0`/../
#echo $0
ROOTDIR=`pwd`

if [ -z `git branch | grep local` ]; then git branch local; fi

CURRENT_BRANCH=`$ROOTDIR/scripts/get_current_branch.sh`
git stash
git checkout local


move_to_branch $ROOTDIR/conf $HOME
if [ ! -e "$HOME/bin" ]; then mkdir $HOME/bin; fi
move_to_branch $ROOTDIR/bin $HOME/bin

git commit -a -m "Backing up local copy of config files"
git checkout $CURRENT_BRANCH
git stash pop

This script goes through the file that are version controlled in the conf or bin directories, and for each one, replaces the current file with a symlink to the version-controlled file. However, it also places a copy of the original file (if there was one) in a branch called local. Now of course, to avoid horribleness, it also records the current branch before stashing it, creates the local branch if it doesn't exist, and the returns to the original branch and pops the stash.

Why? It's a perfectly reasonable question, but what this allows you to do, is have no user interaction, which allows you to call it from post-merge:

$ cat scripts/post-action
#!/bin/sh
./scripts/link_files_branch.sh

Now, every time I pull in a new version, this script will automatically symlink any new files and commit a copy of the original (non-versioned) file in a local branch. This way, I can keep an up-to-date version of everything in master and any local changes in local. "You fool!" you might say, wondering what will happen when I eventually push/pull from the local branch. Well, yeah, that'll be annoying, but I can minimize that by setting default branches to push/pull to/from:

$ cat scripts/setup_repository.sh
#!/bin/bash

cd `dirname $0`

git config remote.origin.pull master
git config remote.origin.push master

#echo $0
SCRIPT_DIR=`pwd`
ROOT_DIR=$SCRIPT_DIR/../
cp $SCRIPT_DIR/post-action $ROOT_DIR.git/hooks/post-merge

Well, now that I've got that 5-minute possible future confusion averted I can go back to work.

Playing with Git

2011-03-08T11:26:00.000-08:00

So I finally got fed up with subversion for managing my personal research projects and documents, etc. and have been playing around with git. So far, I really enjoy it for a few reasons:

I commit more
I branch more
I divide projects logically

Committing changes really is easier, so you do it more. With every commit pushing changes out to a remote repository, and possibly forcing you to update first, there's just enough overhead that I ended up not doing it as much as I should. With git, everything's local, so it's faster and easier. That said, it means that I'm really only committing for myself. In order to commit changes to other people or bring in changes from other people I have to either pull or push (to a bare repository!). However, for files that are mainly personal, I tend to work for a longer period on a single machine and then occasionally shifting to a different machine either to run large jobs or work not at my desk. So for me, a major point of (much) version control is just versioning stuff for personal consumption, not for coordinating with others.

Branching/Merging is easier, so you do it more. In git, there's no real distinction between pulling from a remote repository (e.g. for collaboration) and pulling in from another branch.

Using examples from the excellent gittutorial, pulling from a remote repository, ala svn update, is:

git pull /home/bob/myrepo

Or for the cautious who want to put the update from Bob's master into a new bob-incoming branch:

git fetch /home/bob/myrepo master:bob-incoming

In contrast, merging the experimental branch into the current one:

git pull . experimental

This means that branching and merging uses utilities and skills that you already need to know if you're using git at all! Since you're already familiar with the tools, branching and merging in the traditional sense become much less intimidating.

My repositories now make sense. Because I'm lazy, I'll tend to set things up in the way that seems easiest, not in the way that makes sense. Since svn makes it easy to checkout just part of a repository, I had basically everything I've done in the past 5 years in one repository. This meant coursework, papers (that I wrote), papers (that I read), code, scripts, etc. all went to one svn repository. This was not a good decision, and was ultimately part of why I left subversion. I had munged up something in part of the repository and didn't really know how to fixing other than exporting everything to a new repository.

In contrast, git does not allow you to checkout part of a repository, so you are required to split it up into the subcomponents that make sense. But if that was the end of the story, I'd be complaining about git instead of lauding it because it'd be a pain to have to manually checkout each repository for large projects. However, as pointed out in the git community book, submodules make this unnecessary.

For some modules, a, b, c, d that you have:

$ mkdir ~/git
$ cd ~/git
$ for i in a b c d
do
  mkdir $i
  cd $i
  git init
  echo "module $i" > $i.txt
  git add $i.txt
  git commit -m "Initial commit, submodule $i"
  cd ..
done

You just add them as submodules to the logical container:

$ mkdir super
$ cd super
$ git init
$ for i in a b c d
do
  git submodule add ~/git/$i $i
done

Unfortunately, cloning, updating and committing of submodules gets a bit more complicated. Rather than duplicate information that's better described (and maintained) elsewhere, I'll just say to look at the git community book.

Anyway, I'm pretty pleased with git so far, at least for personal stuff.

Automated Generation of rJava Interfaces with Annotations

2011-03-06T22:39:00.001-08:00

So I've recently decided to create an R interface to one of my Java projects to streamline analysis and ultimately, to make it more usable for other people. As pretty much the only option for running Java code in R, the rJava package makes things pretty simple. You just initialize the jvm with your required jar files and then use .jnew and .jcall to construct objects and run methods. I found the rJava vignettes quite helpful and the package documentation has more detail when you need it.

However, the API for my Java code is in a bit of flux right now and for at least the near future., so I didn't want to put in too much effort that would be lost as soon as the Java code changes. But I also wanted to start to get a sense as to how things would work going between R and Java because that could influence some of my design decisions (e.g. type inference, casting, etc.). In particular, I've been having a hard time deciding how much I should use generics. They're handy for the "type-checking" but a pain in the butt in terms of verbosity.

Anyway, rJava seemed nice and straightforward, but I know my java code would change, so I got a crazy idea in my head. What if I could use Java's user-defined annotations to markup the constructors and methods I wanted exposed in R and then use the apt annotation processing tool to convert those annotations into R code? Well, it turns out, you can, and it only distracted me from my research for about an afternoon.

The actual definition of your annotation is quite simple:

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Retention(RetentionPolicy.SOURCE)
@Target( { ElementType.METHOD, ElementType.CONSTRUCTOR })
public @interface AnnotateR {

}

And using your new annotation is just as easy:

public class HelloJavaWorld {
String message = "Hello Java World!";

public HelloJavaWorld() {

}

/**
 * Create a simple hello world message from Java using rJava
 *
 * @param message
 *            the message to be sent
 */
@AnnotateR
public HelloJavaWorld(String message) {
    if (message != null) {
        this.message = message;
    }

}

@AnnotateR
public String sayHello() {
    String result = new String("Hello Java World!");
    return result;
}

@AnnotateR
public void sayVoid() {}

@AnnotateR
public int[] sayArray() {
    return new int[] { 1, 2, 3 };
}
}

Now in order to do anything useful with those nice new annotations, you have to define a Factory class:

public class AnnotateRJavaFactory implements AnnotationProcessorFactory {
 // Process only the R annotations
 private static final Collection<String> supportedAnnotations = unmodifiableCollection(Arrays
         .asList("net.sourceforge.jannotater.AnnotateR"));

 // No supported options
 private static final Collection<String> supportedOptions = emptySet();

 public Collection<String> supportedAnnotationTypes() {
     return supportedAnnotations;
 }

 public Collection<String> supportedOptions() {
     return supportedOptions;
 }

 public AnnotationProcessor getProcessorFor(Set<AnnotationTypeDeclaration> atds, AnnotationProcessorEnvironment env) {
     return new AnnotateRProcessor(atds, env);
 }
}

This basically just generates objects of the class that does the work:


public class AnnotateRProcessor extends SimpleDeclarationVisitor implements AnnotationProcessor {
  private final AnnotationProcessorEnvironment mEnv;
  private final Set<AnnotationTypeDeclaration> mAnnotationDeclarations;
      .......
      .....

  AnnotateRProcessor(Set<AnnotationTypeDeclaration> annotationDeclarations, AnnotationProcessorEnvironment env) {
      this.mAnnotationDeclarations = annotationDeclarations;
      this.mEnv = env;

  }

  public void process() {
      for (AnnotationTypeDeclaration annotationType : this.mAnnotationDeclarations) {
          for (Declaration typeDecl : mEnv.getDeclarationsAnnotatedWith(annotationType)) {
              typeDecl.accept(getDeclarationScanner(this, NO_OP));
          }
      }
  }

  @Override
  public void visitMethodDeclaration(MethodDeclaration d) {
         //Process a method
      }
}

Then it's just a matter of running it all (using the -factory option forces apt to use your AnnotationProcessor):

apt -cp ./jannotater.jar:$JAVA_HOME/lib/tools.jar -factory net.sourceforge.jannotater.AnnotateRJavaFactory -nocompile -s ${packagename}_tmp $srcfiles

In my case, it also took some reading up on the structure of R packages and then I decided I might as well insert the javadoc comments into the R interface and use inlinedocs to generate the R documentation that are used to generate the help and manuals.

Anyway, I got it all working and threw it up onto sourceforge. For not I just focused on the basic functionality of exposing constructors and methods, but it wouldn't be hard to extend it to expose attributes, too. Also worth mentioning is that I'm doing absolutely nothing to prevent name collisions, so watch out for those. Despite the limitations, though, this makes it absolutely easy-peasy to expose Java functionality to R. Hopefully this is also of value to people who are not me.

Towards a Wrapping JComboBox

2011-03-04T14:06:00.000-08:00

Okay, so I'm going to go ahead and publicly document a terrible decision I made a while ago. I decided that in order to make the user interface I wanted, I needed a WrappingComboBox. I wanted a JComboBox-like interface that would show a collection of options of widely varying lengths but that would not need to expand to occupy the space of the largest item but that could takeup multiple lines of text. I basically wanted this:

I wanted to choose from a set of options, but not require the entire selected option be visible when selected.

This seemingly simple task led me on an increasingly frustrating and desparate struggle to create something acceptable, ultimately including digging into the OpenJDK source code to see what's going on. The end product does what I want, mostly, but it also prompted the following comment on the Java forum here:

"I've seen guys abuse Swing Components before, making them do things they shouldn't do... but I think you got em all beat!"

Okay, so here it is:


import java.awt.Component;
import java.awt.Container;
import java.awt.Dimension;
import java.awt.Insets;
import java.awt.LayoutManager;
import java.awt.Rectangle;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import java.awt.event.ComponentEvent;
import java.awt.event.ComponentListener;
import java.awt.event.InputMethodEvent;
import java.awt.event.InputMethodListener;
import java.util.Vector;

import javax.swing.BoxLayout;
import javax.swing.ComboBoxEditor;
import javax.swing.JComboBox;
import javax.swing.JComponent;
import javax.swing.JFrame;
import javax.swing.JLabel;
import javax.swing.JList;
import javax.swing.JScrollPane;
import javax.swing.JTextArea;
import javax.swing.JViewport;
import javax.swing.ListCellRenderer;
import javax.swing.ScrollPaneConstants;
import javax.swing.SwingUtilities;
import javax.swing.event.EventListenerList;
import javax.swing.plaf.basic.BasicComboBoxUI;
import javax.swing.plaf.basic.BasicComboPopup;
import javax.swing.plaf.basic.ComboPopup;
import javax.swing.text.View;

/**
* @author jfolson
*
*/
public class WrappingComboBox extends JComboBox {

   WrappingComboBox() {
       super();
       this.setUI(new WrappingComboBoxUI());
   }

   WrappingComboBox(Vector<?> items) {
       super(items);
       this.setUI(new WrappingComboBoxUI());
   }

   public static class WrappingComboBoxUI extends BasicComboBoxUI {
       WrappingHelper wrappingHelper;

       public WrappingComboBoxUI() {
           super();

       }

       public WrappingHelper getWrappingHelper() {
           if (wrappingHelper == null)
               wrappingHelper = new WrappingHelper();
           return wrappingHelper;
       }

       /* Since my Editor, Renderer and Popup are not UIResources, I have to overload these methods
        * or else installUI would call BasicComboBoxUI.createXXX anyway and replace mine
        * */
       @Override
       protected ComboBoxEditor createEditor() {
           return new WrappingComboBoxEditor();
       }

       /* See note for createEditor()
        * */
       @Override
       protected ListCellRenderer createRenderer() {
           return new WrappingList.WrappingListCellRenderer();
       }

       /* See note for createEditor()
        * */
       @Override
       protected ComboPopup createPopup() {
           return new BasicComboPopup(comboBox) {
               @Override
               protected JList createList() {
                   // use the WrappingList for the popup to make it update its
                   // sizes
                   return new WrappingList(comboBox.getModel());
                   // BasicComboPopup overrides default, apparently fixes some
                   // bug but I can't use it because BasicGraphicsUtils
                   // functions are private
                   /*return new JList(comboBox.getModel()) {
                       @Override
                       public void processMouseEvent(MouseEvent e) {
                           if (BasicGraphicsUtils.isMenuShortcutKeyDown(e)) {
                               // Fix for 4234053. Filter out the Control Key
                               // from the list.
                               // ie., don't allow CTRL key deselection.
                               Toolkit toolkit = Toolkit.getDefaultToolkit();
                               e = new MouseEvent((Component) e.getSource(), e
                                       .getID(), e.getWhen(), e.getModifiers()
                                       ^ toolkit.getMenuShortcutKeyMask(), e
                                       .getX(), e.getY(), e.getXOnScreen(), e
                                       .getYOnScreen(), e.getClickCount(), e
                                       .isPopupTrigger(), MouseEvent.NOBUTTON);
                           }
                           super.processMouseEvent(e);
                       }
                   };*/
               }
           };
       }

       /*Have to overrige getMinimumSize, rectangleForCurrentValue and layoutContainer in the Helper class
        * to make a smaller (reasonable) sized popup button, otherwise, for multi-line combobox you get
        *  ridiculously large buttons.
        * */
       @Override
       public Dimension getMinimumSize(JComponent c) {
           if (!isMinimumSizeDirty) {
               return new Dimension(cachedMinimumSize);
           }
           Dimension size = getDisplaySize();
           Insets insets = getInsets();
           // calculate the width and height of the button
           int buttonHeight = size.height;
           if (buttonHeight > arrowButton.getPreferredSize().width)
               buttonHeight = arrowButton.getPreferredSize().width;
           int buttonWidth = arrowButton.getPreferredSize().width;
           // adjust the size based on the button width
           size.height += insets.top + insets.bottom;
           size.width += insets.left + insets.right + buttonWidth;

           cachedMinimumSize.setSize(size.width, size.height);
           isMinimumSizeDirty = false;

           return new Dimension(size);
       }

       /**
        * Returns the area that is reserved for drawing the currently selected
        * item.
        */
       @Override
       protected Rectangle rectangleForCurrentValue() {
           int width = comboBox.getWidth();
           int height = comboBox.getHeight();
           Insets insets = getInsets();
           int buttonSize = height - (insets.top + insets.bottom);
           if (arrowButton != null) {
               buttonSize = arrowButton.getWidth();
           }
           if (true) {// BasicGraphicsUtils.isLeftToRight(comboBox)) { // this
               // method is not visible
               return new Rectangle(insets.left, insets.top, width
                       - (insets.left + insets.right + buttonSize), height
                       - (insets.top + insets.bottom));
           } else { // if I could tell, put the box on the other side for right
               // to left checkboxes
               return new Rectangle(insets.left + buttonSize, insets.top,
                       width - (insets.left + insets.right + buttonSize),
                       height - (insets.top + insets.bottom));
           }
       }

       @Override
       protected LayoutManager createLayoutManager() {
           return getWrappingHelper();
       }

       private class WrappingHelper implements LayoutManager {

           //
           // LayoutManager
           //
           /* Need to override layoutContainer to put in a smaller popup button
            *     */
           public void addLayoutComponent(String name, Component comp) {} // TODO

           public void removeLayoutComponent(Component comp) {} // TODO

           public Dimension preferredLayoutSize(Container parent) {
               return parent.getPreferredSize();
           }

           public Dimension minimumLayoutSize(Container parent) {
               return parent.getMinimumSize();
           }

           public void layoutContainer(Container parent) {
               JComboBox cb = (JComboBox) parent;
               int width = cb.getWidth();
               int height = cb.getHeight();

               Insets insets = getInsets();
               int buttonHeight = height - (insets.top + insets.bottom);

               int buttonWidth = buttonHeight;
               if (arrowButton != null) {
                   if (buttonHeight > arrowButton.getPreferredSize().width)
                       buttonHeight = arrowButton.getPreferredSize().width;
                   Insets arrowInsets = arrowButton.getInsets();
                   buttonWidth = arrowButton.getPreferredSize().width
                           + arrowInsets.left + arrowInsets.right;
               }
               Rectangle cvb;

               if (arrowButton != null) {
                   if (true) {// BasicGraphicsUtils.isLeftToRight(cb)) { //
                       // this method is not visible
                       arrowButton.setBounds(width
                               - (insets.right + buttonWidth), insets.top,
                               buttonWidth, buttonHeight);
                   } else { // if I could tell, put the box on the other side
                       // for right to left checkboxes
                       arrowButton.setBounds(insets.left, insets.top,
                               buttonWidth, buttonHeight);
                   }
               }
               if (editor != null) {
                   cvb = rectangleForCurrentValue();
                   editor.setBounds(cvb);
               }
           }
       }

   }

   public static class WrappingComboBoxEditor extends JScrollPane implements
           ComboBoxEditor, ComponentListener {
       EventListenerList listenerList = new EventListenerList();
       LineAwareTextArea text;

       public WrappingComboBoxEditor() {
           super();
           this.text = new LineAwareTextArea();
           this.setViewportView(text);

           setHorizontalScrollBarPolicy(ScrollPaneConstants.HORIZONTAL_SCROLLBAR_NEVER);
           setPreferredSize(new Dimension(100, 30));

           this.setItem("");
           text.setLineWrap(true);
           text.setWrapStyleWord(false);
           text.setEditable(true);

           text.addInputMethodListener(new InputMethodListener() {
               @Override
               public void caretPositionChanged(InputMethodEvent arg0) {}

               @Override
               public void inputMethodTextChanged(InputMethodEvent arg0) {
                   Object[] listeners = WrappingComboBoxEditor.this.listenerList
                           .getListenerList();
                   // Process the listeners last to first, notifying
                   // those that are interested in this event
                   ActionEvent actionEvent = null;
                   for (int i = listeners.length - 2; i >= 0; i -= 2) {
                       if (listeners[i] == ActionListener.class) {
                           if (actionEvent == null)
                               actionEvent = new ActionEvent(this,
                                       ActionEvent.ACTION_PERFORMED, null);
                           ((ActionListener) listeners[i + 1])
                                   .actionPerformed(actionEvent);
                       }
                   }

               }

           });
           text.addComponentListener(this);
       }

       @Override
       public void addActionListener(ActionListener l) {
           listenerList.add(ActionListener.class, l);
       }

       @Override
       public Component getEditorComponent() {
           return this;
       }

       @Override
       public Object getItem() {
           return text.getText();
       }

       @Override
       public void removeActionListener(ActionListener l) {
           listenerList.remove(ActionListener.class, l);
       }

       @Override
       public void setItem(Object anObject) {
           if (anObject == null)
               text.setText("");
           else
               text.setText(String.valueOf(anObject));
       }

       @Override
       public void selectAll() {
           text.selectAll();
       }

       public void componentHidden(ComponentEvent arg0) {}// TODO

       public void componentMoved(ComponentEvent arg0) {}// TODO

       @Override
       public void componentResized(ComponentEvent arg0) {
           JViewport view = this.getViewport();
           Dimension viewDim = view.getExtentSize();
           Insets insets = view.getInsets();
           Dimension wantDim = text.getPreferredScrollableViewportSize();
           // update preferred size of combobox
           if (true) {// wantDim.height > viewDim.height) { // there may be
               // times where
               // you don't want it to get smaller, would need to think some
               // more about this
               viewDim.height = wantDim.height + insets.top + insets.bottom;
               view.setPreferredSize(viewDim);
               view.revalidate();
               ((JComponent) this.getParent()).revalidate();
               SwingUtilities.windowForComponent(this).pack();
           }
       }

       @Override
       public void componentShown(ComponentEvent arg0) {}

   }

   /**
    * Only really need to expose rowHeight but might as well determine the
    * correct size it wants its viewport
    *
    * @author jfolson
    *
    */
   public static class LineAwareTextArea extends JTextArea {
       private int maxVisibleRows = 3; // arbitrary default value, should
                                       // probably be passed in instead

       public LineAwareTextArea() {
           super();
       }

       public LineAwareTextArea(String text) {
           this();
           setText(text);
       }

       @Override
       public Dimension getPreferredScrollableViewportSize() {
           Dimension d = this.getPreferredSize();
           float hspan = getWidth();
           View view = (getUI()).getRootView(this);
           view.setSize(hspan, Float.MAX_VALUE);
           // float vspan = view.getPreferredSpan(View.Y_AXIS);

           Insets insets = getInsets();
           // get the desired number of rows
           int rows = (d.height / this.getRowHeight());
           if (rows > maxVisibleRows)
               rows = maxVisibleRows;

           d.height = rows * getRowHeight() + insets.top + insets.bottom;
           /*System.out.println("prefer view height: " + d.height + " (" + rows
                   + " line(s))");*/
           return d;
       }

       public void setMaxVisibleRows(int rows) {
           this.maxVisibleRows = rows;
       }

       public int getMaxVisibleRows() {
           return this.maxVisibleRows;
       }
   }
}


import java.awt.Component;
import java.awt.Dimension;
import java.awt.Insets;

import javax.swing.JFrame;
import javax.swing.JList;
import javax.swing.JScrollPane;
import javax.swing.JTextPane;
import javax.swing.ListCellRenderer;
import javax.swing.ListModel;
import javax.swing.border.EmptyBorder;
import javax.swing.plaf.basic.BasicListUI;
import javax.swing.text.View;

/**
 * List that wraps its text! Whenever the list's layout (e.g. size) changes the
 * list's UI is instructed to update its layout too.
 */
public class WrappingList extends JList {

    public WrappingList() {
        setUI(new WrappingListUI());
        setCellRenderer(new WrappingListCellRenderer());
    }

    public WrappingList(ListModel model) {
        super(model);
        setUI(new WrappingListUI());
        setCellRenderer(new WrappingListCellRenderer());
    }

    @Override
    public void doLayout() {
        ((WrappingListUI) getUI()).updateLayoutState();
        super.doLayout();
    }

    @Override
    public boolean getScrollableTracksViewportWidth() {
        return true;
    }

    /**
     * ListUI implementation that exposes the method for updating its layout
     */
    private static class WrappingListUI extends BasicListUI {

        @Override
        public void updateLayoutState() {
            super.updateLayoutState();
        }

    }

    /**
     * List cell renderer that uses the list's width to alter its preferred size
     * TODO - override bound properties in the same way as
     * DefaultListCellRenderer
     */
    public static class WrappingListCellRenderer extends JTextPane implements
            ListCellRenderer {

        public WrappingListCellRenderer() {
            setBorder(new EmptyBorder(1, 1, 1, 1));
        }

        public Component getListCellRendererComponent(JList list, Object value,
                int index, boolean selected, boolean hasFocus) {

            setBackground(selected ? list.getSelectionBackground() : list
                    .getBackground());
            setForeground(selected ? list.getSelectionForeground() : list
                    .getForeground());
            // TODO - border, font etc.

            setText(String.valueOf(value));

            float hspan = list.getWidth();
            View view = (getUI()).getRootView(this);
            view.setSize(hspan, Float.MAX_VALUE);
            float vspan = view.getPreferredSpan(View.Y_AXIS);

            Insets insets = getInsets();
            setPreferredSize(new Dimension(insets.left + insets.right
                    + (int) hspan, insets.top + insets.bottom + (int) vspan));

            return this;
        }
    }

    public static void main(String[] args) {
        new Test();
    }

    public static class Test extends JFrame {

        public Test() {
            setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
            init();
            pack();
            show();
        }

        private void init() {
            JList list = new WrappingList();
            list
                    .setListData(new Object[] {
                            "Item One",
                            "Item Two content content content content Item Two content content content content Item Two content content content content Item Two content content content content" });

            setContentPane(new JScrollPane(list,
                    JScrollPane.VERTICAL_SCROLLBAR_AS_NEEDED,
                    JScrollPane.HORIZONTAL_SCROLLBAR_NEVER));
        }

    }
}

And a main to run:


public class ComboTest extends JFrame {

       public ComboTest() {
           setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
           init();
           pack();
           show();
       }

       private void init() {

           ScrollablePanel mainPanel = new ScrollablePanel();
           mainPanel.setScrollableWidth(ScrollableSizeHint.FIT);
           mainPanel.setLayout(new BoxLayout(mainPanel, BoxLayout.Y_AXIS));
           ScrollablePanel fieldPanel = new ScrollablePanel();
           fieldPanel.setLayout(new BoxLayout(fieldPanel, BoxLayout.X_AXIS));
           fieldPanel.setScrollableWidth(ScrollableSizeHint.FIT);

           Vector<String> items = new Vector<String>();
           items.add("Item One");

           JComboBox list = new WrappingComboBox(items);
           list.setEditable(true);

           JLabel fieldLabel = new JLabel("Label: ");
           fieldLabel.setAlignmentY(TOP_ALIGNMENT);
           list.setAlignmentY(TOP_ALIGNMENT);
           fieldPanel.add(fieldLabel);
           fieldPanel.add(list);
           mainPanel.add(fieldPanel);

           fieldPanel = new ScrollablePanel();
           fieldPanel.setLayout(new BoxLayout(fieldPanel, BoxLayout.X_AXIS));
           fieldPanel.setScrollableWidth(ScrollableSizeHint.FIT);

           items
                   .add("Item Two content content content content Item Two content content content content Item Two content content content content Item Two content content content content");

           list = new WrappingComboBox(items);
           list.setEditable(true);

           fieldLabel = new JLabel("Label: ");
           fieldLabel.setAlignmentY(TOP_ALIGNMENT);
           list.setAlignmentY(TOP_ALIGNMENT);
           fieldPanel.add(fieldLabel);
           fieldPanel.add(list);
           mainPanel.add(fieldPanel);
           JScrollPane mainScrolls = new JScrollPane(mainPanel,
                   JScrollPane.VERTICAL_SCROLLBAR_AS_NEEDED,
                   JScrollPane.HORIZONTAL_SCROLLBAR_NEVER);
           mainScrolls.setPreferredSize(new Dimension(200, 200));
           setContentPane(mainScrolls);
       }
   }

   public static void main(String[] args) {
       new ComboTest();
   }
}

So now it doesn't look nearly so bad, but Swing does a lot of delegating to helper classes, and I had to track down those helper classes, none of which make much of an appearance in the API or javadoc and make sure that my versions were being used. In particular, I had to override createEditor(), createRenderer() and createPopup() or else the super classes would just replace my classes with their own.

There was also a bit of math in determining sizes that I had to consult the OpenJDK to determine the right things to query. I changed it up a bit in order to reduce the size of the popup button for taller WrappingComboBoxes.

The ScrollablePanel class referenced comes from here but I don't think it's actually necessary for simple use cases but it's definitely worth knowing about. Feel free to ask questions. Hopefully this is helpful to at least one person somewhere sometime.

Going Parallel in R

2009-10-26T15:36:00.000-07:00

I recently had the need to get some parallelism going in R. I was doing some largeish-scale monte carlo and markov chain monte carlo simulations of latent space network models for NIPS. Anyway, my needs were quite simple, basically, I just need lots of repetitions and I wanted to easily spread the repetitions across a few processors/cores, all on a single machine.

I found a number of options and started out using the multicore package. This worked wonderfully and easily when I tested it out on my desktop and ported my code over to use it. Of course, I neglected to notice the not-so-small print on the manual:

SystemRequirements: POSIX-compliant OS (essentially anything but Windows)

Now, I had never really encountered system-specific packages for R before, so this kind of caught me by surprise when I scp'ed things over to our big ole' windows machine to run and found that I couldn't install multicore. Well, eventually wised up and replaced mutlicore with snowfall when on a windows system. Beginning to think I should have just used foreach, but that's another story.

Editing Mendeley Citation keys

2009-10-26T09:52:00.000-07:00

Mendeley is a nifty new tool for organizing and sharing academic references. I've been test driving it a bit, mainly using it to import files that I then transfer over to JabRef. This was working out alright, since Mendeley allows you to export to bibtex, which is the backend for JabRef and what I ultimately end up using to write my papers. I've been looking for a way to streamline this process, since I like Mendeley's interface a lot better and it has better search features, e.g. full-text search of all attached pdfs.

Unfortunately, Mendeley doesn't use bibtex for its backend, but it does use SQLite and will even let you export your database as a SQLite zip file. I can't seem to figure out where Mendeley stores my user files so far, so exporting seems the way to go.

Using the SQLite Database Browser, I managed to load up my Mendeley database. Things seem pretty obvious, with most of the things you'd want to futz with in the "Documents" table. For now I just removed all the citationKey values and generated my own the way that I wanted them, but it's nice to know that it's not too miserable to get at the raw data from Mendeley.

Note: You can also restore an edited zip of sqlite data, but you have to have it all flat, exactly the way that Mendeley exported it.

Analysing Spatiall Embedded Networks

2009-03-24T13:02:00.000-07:00

I realized this would probably be a decent place to describe some of the work I've been doing and hopefully get some feedback. I've been working at the intersection of spatial information and network information, trying to develop ways of integrating the two types of information. This is not the "spatial networks" that are used to analyze space syntax and transportation networks. Instead, I'm interested in looking at arbitrary networks of things where some of the things are labeled with spatial location information.

So far I've analyzed drug seizure networks, shipping networks, organizational network of terrorist groups, epidemiological networks and simulated social networks. Most of this analysis has been to test new methodologies I've developed. There are two main techniques I've been working on, manipulating network scale and aggregation and visualizing spatial dependencies in network topology.

There is a fundamental discord between network and space. Space is continuous; relationships in networks are defined as between discrete entities. This means that some level of aggregation of space (implicit or explicit) us required in order to do a meaningful analysis of the network. Different levels of aggregation can lead do quite different networks. I've been working on methods of capturing the tradeoffs of aggregation versus precision.

Second, visualization of social networks has been important historically in providing the intuition for many commonly used network statistics. My hope is that the visualization of spatially embedded network data will be similarly useful. However, simple visualizations of spatially embedded networks quickly become noisy and difficult to interpret. For this reason, I'm developing techniques for visualizing higher-level network topological properties and their interaction with spatial location. I am also working to develop a statistical measure of the spatial dependencies in structural properties of a spatially embedded network.

All of these techniques and more have been implemented in the Geospatial Network Visualizer in the ORA dynamic network analysis tool.

Managing Bibtex and PDFs with JabRef!

2009-03-01T16:09:00.000-08:00

A few days ago, I decided it was time for my assorted bibtex files to grow up and get organized. I also decided that my assorted pdfs of papers should also get centralized and organized. I had the crazy idea, that there might be a tool that could do both of these tasks in an open, portable format that I could easily transition if I grew tired of the tool. Unlike past endeavors, in this task I succeeded beyond all expectation. The JabRef reference manager is an open source reference manager that uses the standard bibtex format as a backend. Unlike other tools I tried, it works, simple and pretty feature-complete, it provides a centralized location to store references, notes/review/summaries of said references, as well as links to the actual pdfs (when you have them). JabRef will automatically generate bibtex keys for you, if you ask it, and if you save your pdfs with the respective key in its filename, JabRef can automatically find the file and associate it with the intended reference! Plus, like many of the other tools, it can automatically sync up to many of the online databases and download documents if you have access to them.

It also has extremely flexible keyword/category-based organization of references. Your categories can inherit either supercategories, subcategories, or neither.

Cisco VPN client in Ubuntu 8.10

2009-03-01T15:58:00.000-08:00

I'm not sure what changed, but I can't seem to get the Cisco VPN client working in Ubuntu 8.10. In 8.04 I'd gotten things working by following instructions like these. Once again, the good folks at tuxx-home.at had a few solutions, but I couldn't get them to work. Either seg faults when it ran or compiler errors and it wouldn't. Anyway, I'd been leery, of trying vpnc since my .pcf files for the SCS at Carnegie Mellon didn't quite look like any of the examples. However, when I eventually tried this decoder with my encoded group password, everything worked out! Now I just use the NetworkManager applet to manage my vpn and it all (mostly) works.

FGLRX Sadness

2009-03-01T15:55:00.000-08:00

I was reminded once again of why other people give up on linux when I walked over to my new desktop only to find that the 3d graphics driver, fglrx, had decided not to work. I spent several hours over the next few days trying to track down the issue, uninstalling, reinstalling, reverting to older versions of the driver, xserver, the linux kernel, all to no avail. Eventually, I gave up and am now using the 2d open source drivers. The fglrx driver had even stopped working adequately for simple window management things.

HIDPoint Rocks

2009-01-29T19:01:00.001-08:00

HIDPoint makes a free driver and configuration program for a variety of mice and keyboards for linux. I just installed their drivers for the Logitech MX3200 wireless keyboard and the MX600 wireless mouse and it works fabulously. All the special buttons and fancy things work! I couldn't be happier!

Also

2009-01-28T10:53:00.000-08:00

I love my G1

Android HTML parsing

2009-01-28T10:42:00.000-08:00

A few weeks ago, I spent an extended period of time trying to find an html parser that would work reasonably well in the Android mobile operating system. Long story short: fail. This site provides a helpful list of open source html parsers for java, but alas each and every one of them was too slow to be usable in android. I wasn't even trying to parse a large page, only ~100K, but it was still taking ~ 1 minute to parse. In the end, I gave up. Instead of parsing the site and redisplaying some of the info, I just manually skipped down to the content part of the page and fed the content as a String to the built-in webkit browser. Fortunately, Android makes it extremely straightforward to embed the browser in an app.