We did that by using a normal table alter statement but then started seeing the following error when we tried to insert test data in that column which takes an SDO_GEOMETRY object:

org.hibernate.exception.JDBCConnectionException: could not execute native bulk manipulation query
       at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:99)
       at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
       at org.hibernate.engine.query.NativeSQLQueryPlan.performExecuteUpdate(NativeSQLQueryPlan.java:219)
       at org.hibernate.impl.SessionImpl.executeNativeUpdate(SessionImpl.java:1310)
       at org.hibernate.impl.SQLQueryImpl.executeUpdate(SQLQueryImpl.java:396)
       at $Proxy53.insertTariffZone(Unknown Source)    
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
       at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
       at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
       at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
       at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
       at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
       at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
       at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
       at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
       at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
       at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
       at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
       at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
       at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
       at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
       at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:71)
       at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:202)
       at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
       at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: java.sql.SQLRecoverableException: No more data to read from socket
       at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1157)
       at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
       at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
       at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
       at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
       at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1044)
       at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1329)
       at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584)
       at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3665)
       at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1352)
       at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105)
       at org.hibernate.engine.query.NativeSQLQueryPlan.performExecuteUpdate(NativeSQLQueryPlan.java:210)
       ... 39 more

We couldn’t see anything particularly wrong in what we’d done and none of the error messages we got were being particularly helpful.

Eventually we asked the DBA on our team to help out and he showed us how to look up the Oracle system logs which in our case were located at:

/u01/app/oracle/diag/rdbms/orcl/orcl/trace

We ran the query again and noticed new files were being written to that location, one of which had the following error message:

Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x40] [PC:0x2FDAE7D, mdidxid()+2563] [flags: 0x0, count: 1]
DDE: Problem Key 'ORA 7445 [mdidxid()+2563]' was flood controlled (0x2) (incident: 3851)
ORA-07445: exception encountered: core dump [mdidxid()+2563] [SIGSEGV] [ADDR:0x40] [PC:0x2FDAE7D] [Address not mapped to object] []
ORA-13203: failed to read USER_SDO_GEOM_METADATA view
ssexhd: crashing the process...
Shadow_Core_Dump = PARTIAL

We’d forgotten to rename the table in the USER_SDO_GEOM_METADATA view, exactly as the message says!

Running the following statement sorted us out:

UPDATE user_sdo_geom_metadata SET table_name = 'NEW_NAME' where table_name = 'OLD_NAME';

I was testing the BlockProximityFusion class which is used to merge together adjacent text blocks.

I started off calling that class like this:

import static java.util.Arrays.asList;
 
@Test
public void willCallBlockProximityFustion() throws Exception {    
    TextDocument document = new TextDocument(asList(contentBlock("some words"), contentBlock("followed by more words")));
    BlockProximityFusion.MAX_DISTANCE_1.process(document);
}
 
private TextBlock contentBlock(String words) {
    TextBlock textBlock = new TextBlock(words, new BitSet(), wordCount(words), 0, 0, 0, 0);
    textBlock.setIsContent(true);
    return textBlock;
}

Which blows up like this:

java.lang.UnsupportedOperationException
	at java.util.AbstractList.remove(AbstractList.java:144)
	at java.util.AbstractList$Itr.remove(AbstractList.java:360)
	at de.l3s.boilerpipe.filters.heuristics.BlockProximityFusion.process(BlockProximityFusion.java:115)
	at de.l3s.boilerpipe.filters.heuristics.BlockProximityFusionTest.willCallBlockProximityFustion(BlockProximityFusionTest.java:63)

The code around that area is trying to remove an element from an iterator…

113
114
115
116
117

                if (ok) {
                    prevBlock.mergeNext(block);
                    it.remove();
                    changes = true;
                } else {

…which was created from the list that we passed into the constructor of TextDocument:

98

        for (Iterator<TextBlock> it = textBlocks.listIterator(offset); it

The remove method is not implemented on the list created by ‘Arrays.asList’ which is weird since I thought it created an ArrayList which does implement remove!

I’ve now learnt that the ArrayList created by ‘Arrays.asList’ is actually a private inner class of Arrays and doesn’t implement the remove method!

Who knew…

Googling for this takes me to this page and I always think I’ll just checkout the SVN repository and hook that up but it doesn’t seem to be available.

$ wget -S http://java.net/projects/jdk-jrl-sources/
--2012-02-11 09:51:34--  http://java.net/projects/jdk-jrl-sources/
Resolving java.net (java.net)... 192.9.164.103
Connecting to java.net (java.net)|192.9.164.103|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 404 Not Found
  Date: Sat, 11 Feb 2012 09:51:34 GMT

The alternative is therefore to download the jar provided which we can do like this:

wget http://www.java.net/download/jdk6/6u23/promoted/b05/jdk-6u23-fcs-src-b05-jrl-12_nov_2010.jar

The next step is then to execute the jar which I somehow didn’t realise until I unpacked it and had a look at the README:

java -jar jdk-6u23-fcs-src-b05-jrl-12_nov_2010.jar

You get asked to choose a folder location for the sources and then the code is under ‘src/share/classes’. So for me I need to give IntelliJ a source path of :

/Users/mneedham/github/j2se/src/share/classes

You can browse the different versions of the source code by changing the version number at the end of URLs like this one. At the moment version 23 is the latest one available.

Towards the end I pointed out that we weren’t actually going to be using an incremental approach as we’d initially thought which was due to a couple of constraints that we have to work under.

Hardware provisioning

One of the main reasons that we favoured an incremental approach is that we’d be able to deploy to production early which would allow us to show a quicker return on investment.

Unfortunately we later on came to learn that it takes around 6-9 months to provision production hardware.

It therefore didn’t make a lot of sense to take an approach where we tried to integrate into the existing system since we wouldn’t be able to deploy that work.

We’re working under the assumption that in 6-9 months we’ll probably be able to rewrite the whole thing and can therefore avoid the need to write the code which would allow us to integrate into the existing version.

We couldn’t see any value in writing bridging code between the existing and new versions of the application if it never sees production – we’d put in all the effort for no reward.

Running two systems side by side

Even if we had been able to provision hardware in time to release incrementally we came to learn that the business would be uncomfortable with having two versions of the same application in production at the same time.

The application is used to do pricing and the worry was that the different versions might produce different results for the same inputs.

It’s arguably something we may have been able to overcome if we could prove that the new version worked exactly the same as the existing one by running both applications against a set of scenarios and checking that they returned the same results.

Theoretically I suppose the first problem could also be overcome but it’s a battle we’ve chosen to leave alone for the moment.

What I found interesting in the discussions about the way we should deliver our solution was that I’d worked under the assumption that an incremental approach was always a better approach but with these constraints it isn’t.

In order to do that we needed to look for what Michael Feathers refers to as a seam:

A seam is a place where you can alter behaviour in your program without editing in that place

On previous times when I’ve been thinking about seams it’s been at a code level inside a single application but this time there were more than one pieces interacting.

We knew that there was a web application where the user could request a quote which would be calculated offline and then an email sent to them when it was ready to view.

That led us to believe that there was probably some sort of queue being used to store the outstanding requests and there’d probably be some sort of application processing the requests.

As it turned out the design of the system actually looked like the diagram on the right with the database effectively as a queue.

We then needed to work out which tables we had to read from/write to so that we’d be able to just replace the ‘polling application’ and leave the ‘web application’ alone.

We were then able to come up with a design whereby we isolated any interaction with the database into a ‘bridging application’ which then farmed requests out to a new application which we could scale horizontally.

It could also take care of writing the quotes back into the database so the existing application could read them back onto the screen.

Although we ended up not using this architecture for other reasons I think it’s a neat way of looking at systems to work out how we can change them with minimal impact.

The API of the various classes/methods is designed so it’s interoperable with Java code but in order to use functions like map/filter we need the collection to be a Scala one.

That’s achieved by importing ‘scala.collections.JavaConversions._’ which will apply an implicit conversion to convert the Java collection into a Scala one.

I needed to go back to the Java one again which can be achieved with the following code:

import scala.collection.JavaConversions._
 
val javaCollection = seqAsJavaList(Seq("abc"))

I also used that function in the StopWords.scala object in Goose.

There are a load of other functions available in JavaConversions as well for going to a Dictionary, Map, Set and so on.

-bash-3.2$ dbstart
 
ORACLE_HOME_LISTNER is not SET, unable to auto-start Oracle Net Listener
Usage: /u01/app/oracle/product/11.2.0/dbhome_1/bin/dbstart ORACLE_HOME
Processing Database instance "orcl": log file /u01/app/oracle/product/11.2.0/dbhome_1/startup.log

Ignoring the usage message we thought that setting the environment variable was what we needed to do, but…

-bash-3.2$ export ORACLE_HOME_LISTNER=$ORACLE_HOME
-bash-3.2$ dbstart
ORACLE_HOME_LISTNER is not SET, unable to auto-start Oracle Net Listener
Usage: /u01/app/oracle/product/11.2.0/dbhome_1/bin/dbstart ORACLE_HOME
Processing Database instance "orcl": log file /u01/app/oracle/product/11.2.0/dbhome_1/startup.log

We ended up looking at the source of dbstart to see what was going on:

# First argument is used to bring up Oracle Net Listener
ORACLE_HOME_LISTNER=$1
if [ ! $ORACLE_HOME_LISTNER ] ; then
  echo "ORACLE_HOME_LISTNER is not SET, unable to auto-start Oracle Net Listener"
  echo "Usage: $0 ORACLE_HOME"

The usage message does explain that you’re supposed to call it like this:

-bash-3.2$ dbstart $ORACLE_HOME

But it still seems a bit weird/misleading to me that you’d override the value of a global variable from inside a script which doesn’t suggest that it’s going to do that!

Such is life in Oracle land..

The aim is to get your scripts to the point where the outcome is reasonably deterministic so that we can have confidence they’re going to work the next we run them.

We noticed two ways in which we haven’t quite achieved determinism yet:

Accidental Dependencies

The first few times that we ran the scripts on top of a vanilla image we were doing it on a virtual machine which had VMware tools installed on it.

We’d forgotten that VMware tools had been installed on those VMs and ran into a problem with Oracle dependencies not being satisfied when we ran puppet on some machines which had CentOS installed directly (i.e. not on a virtual machine).

Those dependencies had been satisfied by our VMware tools installation on the VMs so we didn’t realise that we hadn’t explicitly stated those dependencies, something which we have done now.

External Dependencies

We couldn’t find the Firefox version that we wanted install on the default yum repositories so we created a puppet task which linked to a Firefox RPM on an external server and then installed it.

It worked originally but at some stage over the last couple of weeks the URI was changed as a minor version had been upgraded, breaking our script.

We also came across another way that external dependencies can fail today – if a corporate proxy blocks access to the URL!

We’re trying to get to the stage where we’re only relying on artifacts either coming from a yum repository or an internal repository where we can store any libraries which aren’t available through yum.

Don’t assume determinism

While trying to solve these dependency problems in our puppet scripts I made the mistake of assuming that if the script runs through once and works that it’s always going to be that way in the future.

Since we had achieved that previously in my mind it was impossible for it to fail in future which stopped me from properly investigating why it had stopped working.

I originally started using this technique while I was doing the programming assignments for ml-class because I wanted to know how much time I was spending on it each week and make sure I didn’t run down rabbit holes too often.

One interesting observation that I noticed from keeping the data of these pomodoros was that while during the early programming assignments it would take me 7 or 8 pomodoros to finish, by the end it was down to around 4.

I think this was due to the difficulty of the assignments decreasing as time went on, I didn’t improve that dramatically!

As I mentioned a few weeks ago I’ve also been using pomodoros in combination with a yak stack to make sure I don’t go off track and it’s been interesting applying the technique while trying to solve a problem I’m having with using the Jersey client on Android.

It’s such a fiddly problem and splitting my time into 25 minute slots has forced me to create a plan for what I’m going to try and do in that pomodoro, whether it be ruling out an approach or trying to understand the underlying code that isn’t working.

I haven’t been successful in solving my problem but I’m pretty sure that I’ve spent much less time trying to solve it than I would have otherwise. I can certainly imagine spending hours aimlessly trying things that have no chance of working.

One thing I’ve been experimenting with is reducing the length of the pomodoro to 15 minutes when I know there’s something specific that I want to investigate and I’m fairly sure it won’t take a full length pomodoro.

Previously I would end up just killing time for 10 minutes or just resetting the pomodoro because I didn’t have anything else to do.

I generally enjoy coding much more by applying this time constraint and I think the reason for that is explained by The Progress Principle, which I’m currently reading:

If people are in an excellent mood at the end of the day, it’s a good bet that they have made some progress in their work. If they are in a terrible mood, it’s a good bet that they have had a setback.

To a great extent, inner work life rises and falls with progress and setbacks in the work. This is the progress principle

Using a pomodoro seems to reduce the amount of time that is spent dealing with setbacks and it creates frequent opportunities to discard an approach you’re taking if it’s clear that it’s not going anywhere.

A disadvantage that I’ve sometimes felt when working on the Jersey/Android problem is that I really don’t want to spend 25 minutes working on it because I’ve been getting absolutely nowhere with it for about 6/7 pomodoros now.

I’d rather delude myself that I’m going to magically fix it just by fiddling around with the code for an indeterminate period of time!

In a way constraining coding in this way does take some of the fun out of it as well because it’s now more structured and you tend to have fun when you’re just randomly doing stuff and lose track of time.

On the other hand I probably end up doing a lot more of the stuff I want to do when I constrain it in this way!

Decisions, decisions…

Unfortunately Oracle Linux builds don’t come with any yum repos configured so when you run the following command…

ls -alh /etc/yum.repos.d/

…you don’t see anything

We eventually realised that there are a list of public yum repositories on the Oracle website, of which we needed to download the definition for Oracle Linux 5 like so:

cd /etc/yum.repos.d
wget http://public-yum.oracle.com/public-yum-el5.repo

We then need to edit that file to enable the appropriate repository. In this case we want to enable ol5_u7_base:

[ol5_u7_base]
name=Oracle Linux $releasever - U7 - $basearch - base
baseurl=http://public-yum.oracle.com/repo/OracleLinux/OL5/7/base/$basearch/
gpgkey=http://public-yum.oracle.com/RPM-GPG-KEY-oracle-el5
gpgcheck=1
enabled=1

I made the mistake of enabling ol5_u5_base which led to us getting some really weird problems whereby yum got confused as to which version of libselinux we had installed and was therefore unable to install libselinux-ruby as its dependencies weren’t being properly satisfied.

Calling ‘yum list installed’ suggested that we had libselinux 1.33.4.5-7 installed but if we ran ‘yum install libselinux’ then it suggested we already had 1.33.4.5-5 installed. Very confusing!

After trying to uninstall and downgrade libselinux and pretty much destroying the installation in the process, another colleague spotted my mistake.

We also found that we had to add the epel repo which gave us access to some other packages that we needed:

rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm

After all that was done we were able to run the command to install puppet:

yum install puppet

That installs puppet 2.6.12 as that’s the latest version in that repo. The latest stable version is 2.7.9 but I think we’ll need to hook up a puppet specific repo to get that working.