Check out my new post on comparing build management in Eclipse C and Java development tools, here:

Comparing Build Management in Eclipse CDT and JDT

A workflow management tool is only as intelligent as the scripts it is being asked to execute. Build management tools that function like a job scheduler simply takes a command an executes it on a particular machine. If that command is a make, maven or ant script, it is going to send that script to be executed on a particular machine. That is NOT Distributing the build. It is distributing the build script. Build scripts are rarely written to support distributed build processing, so your workflow solution cannot magically turn that script into an intelligent process that can be distributed. In fact most scripts have no understanding of dependencies, so there is no way for them to know what order to build the application, much less distribute those calls out to external machines.

OpenMake Meister does understand dependencies and does have the intelligence to know the order in which to build artifacts. In fact, out of the box, it does this for your builds, on a single machine using multi-core processors calling the compilers in multi-threaded mode. So if you are looking for speed and need to distribute builds across multiple machines, look no farther than Meister. This is what Build Automation is.

Selvi’s Blog

Adam Gabor’s Blog

Sean Blanton’s Blog

The content up to this point will appear in both places for convenience. Please update your RSS feeds if you are using them. See you there!

Sean

I’ve set up a JBoss build in Meister involving a web service and consumer. This was originally developed in MyEclipse, but the Meister Eclipse plug-in allows the native Eclipse build to be externalized (a better approach than Maven). So, I am just dealing with that, which simulates a development continuous integration or QA build.

I have a workflow that has the two main Meister build tasks, “Generate Build Control File”, and “Build Targets”. The first gathers local and server metadata consolidating that into something like a makefile or Ant script depending on your perspective. The second is similar to running make or Ant.

This is a simple workflow to illustrate our messaging technique. Normally, we’d use an email notification task, but I’ve replaced that with an activity called “Tweet”. This calls a simple Perl script, called omtweet.pl that uses Net::Twitter. Here is why we use Perl a lot - the code is simple to update your Twitter status:

1
2
3
4
5
6
7

use Net::Twitter;
 
my $twit = Net::Twitter->new( {username => "myuser", password => "mypass"; });  
my $result = $twit->update( {status => "My current Status"});

I created a slightly more elaborate script that takes arguments and passes in a “Build Successful” message. If Java is your thing, look at Davanum Srinivas’ post and for .NET, there is Twitteroo.

Next, I run the build and we see the steps complete in real time on the workflow monitor.

The key is the last step, of course:

And, the proof is in the pudding:

I used the $(JOB_NUMBER) macro in Meister to get an incremental build number passed to omtweet.pl. Of course, following @builds and choosing device updates, I got a text message on my cell phone. I tried to get a picture of my cell phone with the message, but I haven’t worked out how to do that yet and it turned out terrible.

Well, this is a simple example, with a lot of possibilities. Meister wraps all build output in HTML and pushes to a web server. I’d like to include a shortened URL for that in the update. Currently we don’t have anything outside of a firewall, maybe I’ll try Yammer for that.

If you have any other suggestions, let me know. @seanblanton

I have run maven a couple of times but don’t use it. (I had to build some open source jars). And today I decided to clean up my harddrive on my Mac using OmniDiskSweeper. Low and behold maven created a hidden directory in my home directory called .m2. In that was 320MB of jar files and classes. Why does maven need to hide that? Shouldn’t it be nice and put it in a location that I know about so I can manage it. Instead of hiding junk on me.

The idea comes from Lean Manufacturing principles developed by Toyota applied to software development. An “andon” paper lamp was a way to let other workers know there was a problem on the assembly line. The idea for builds is that while a build is running, an amber light flashes, letting other developers know not to run a build, a green light signifies the last build was completed successfully, and a red light means the build is broken and needs to be fixed.

While I’m sure it is an entertaining exercise to set this up, some simple software automation could achieve the same results, and it has obvious faults, like what if I am working from home?

Email is the de facto standard for build notification. There is a hardcoded mailing list somewhere and you get the email, even if you don’t want it, and you have to go and check your mail to receive it.

Twitter, on the other hand, puts the user in control of the notifications in true Web 2.0 fashion. Tweets are also ideal for receiving notifications via cell phone, where email is generally not. Consider the following cases:

• I can follow the build when I want, and unfollow when I want. No going to an admin to remove me from the list.
• I can turn on device updates so I get an SMS message about the result (maybe an important release is coming up), or I can turn off device updates, say, when I’m on vacation.
• I can retweet (forward) the build notification.
• You can put links in the message. Meister uploads all build logs in HTML format to a web server and we would include the link to that.

About a year ago, I used Meister with a post-build activity to Tweet the build result after a build. The activity was a simple Perl script using Net::Twitter. Let’s just say this experiment sort of fell on deaf ears.

Now I’m ready to roll with this again and you can follow my progress. I had created a Twitter account, @builds, for this purpose. In a day or two, you’ll see some tweets from that guy from one of my builds.

I’m interested in what other people have to say. Meister already uses Eclipse RCP as it’s front end, but most other build tools are still command-line and in the 1970’s. Let’s bring build management tools into the 21st century with Web 2.0 features.

Ask for a build and get a build. Ask for continuous integration and test automation and coalescence and get it with Meister. Product owners should not be losing cycles to debugging builds, tweaking parameters to get different types of builds or in resolving confusion about what did or did not happen in a build or continuous integration workflow.

The client-server architecture and highly reusable build and workflow metadata extend the ScrumBut elimination to all developer machines and continuous integration build servers, creating a highly consistent build. Team members working on these different machines know exactly what to expect.

Since they know what to expect from the build, they can focus on resolving actual integration issues and testing rather than trying to make the build work in a consistent manner.

I posted a comment about the reluctance of script writers to give up their scripting on their blog here.

As customers go to machines with more and more cores, fewer machines are needed in the application lifecycle infrastructure, particularly for builds and code retrievals - the most resource intensive functions. This is helping to simplify the infrastructure, reduce maintenance and administration and drive down costs.

Several of our customers are running around 5000 builds and non-build workflows per month on two machines. The primary reason for two machines, in fact, is for disaster recovery, and the goal is to run both machines at less than half capacity so that in the event that one machine (or datacenter) fails, all the current capacity can be run as a contingency on the one machine that’s left.

Thread control is very simple with Meister and Mojo. Both use the omsubmit dependency manager program to handle this. Meister’s om program translates build events into workflow steps using omsubmit. The OMSUBMIT_MAX_USER_PROC value sets the maximum allowed number of threads.

You might think that if you are running dual, quad-core build machines that you should set the max threads at 8. However, Meister posts build operations to one thread and the associated logging operation to another. Compile operations notoriously use a lot of memory and CPU resources, but the logging operation posts to a server and waits for the operation to complete. There is really no disadvantage to setting the max threads higher than 16 in this case, so go ahead and do it.

As a non-build workflow example, I worked on JBoss deployments to 48 Linux machines. The workflow was parallelized into 48 activities each of which deployed to a single machine in parallel. The deployment activity was largely a remote execute operation that extracted archives on the remote machine. The extraction took about a second for a medium sized application. Again, this is a waiting situation where machine resources are essentially idle while the thread is in use, so use more threads. The machine was a dual, dual-core build machine and we set OMSUBMIT_MAX_USER_PROC to 50.

Watching the workflow monitor as the deployment ran, we could see roughly half of the machines light up (meaning actively running) at any one time and the entire deployment process synchronized all 48 machines in a little over two seconds.

So, don’t simply match your machine’s CPU threading capabilities - overclock! Aim high for max threads and try to determine where your performance is optimized. I’d love to provide you with some metrics as a function of thread count, but usually once something is working it’s on to the next project. I barely have enough time to blog!

Let me explain, Fred. The simple approach most of us take (and have to take when getting things started) has developers commit, commit, commit, and when it is time to deploy, check out the code, do a build, and then deploy the application. There is room here for both problems and optimizations. Doing a full check out of the code tree is more costly in terms of time than checking out only what has changed. Updating the code tree with a single commit is less costly than updating with a large number of commits.

You may be limited by the technology in-hand and how much you’ve invested in learning the technology and possibly customizing it. For example, if your file control tool can only do a full check out of a source tree, or that’s the only command you had time to implement in order to meet the deadline, or you don’t trust your tool to do incremental updates, then you are basically running the longest builds possible.

On the other hand, if you could update the code tree every time a developer does a commit with only the changed files, then you are ready to execute a build at any moment. This requires some deft manipulation of your file control tool, and that’s why you don’t see it more often.

You might think “continuous integration” will take care of this. Developer commits, update checked out, build is run. However, you may end up with a build, test execution and deployment that takes longer than the typical time between developer commits. You still have to do incremental updates and it only solves the problem in cases with very low developer activity.

I’d like to point out one tool that does an excellent job of post-commit code checkout, CA Software Change Manager for Distributed. CA SCM (for short) is the tool, formerly known as Harvest, from the company formerly known as Computer Associates. CA SCM is a highly scalable (1000’s of developers) file control tool with a great lifecycle process model. We at OpenMake Software still have our very first customer still using OpenMake/Meister with CA Harvest/SCM after 11 years. While we have a reseller arrangement with CA, our partnership with CA in services has extended to 14 years.

About 10 years ago, OpenMake Software developed an integration with the then, Platinum Technologies’ Harvest product, modeled after the now dead Computer Associates product, Endevor Workstation, that had an excellent post-action code tree update. (Endevor for z/OS, a.k.a CA SCM for Z/OS is still very popular and has a similar functionality called ‘output libraries’ - following all this?) Our integration had the horrific name, ‘Har-refresh’.

As product partners, we finally transitioned Har-refresh from an external add-on to CA who have turned it into a core functionality of the product, called Hrefresh (a better name.) Rather than simply a post-commit check out, HRefresh updates the code tree after any action that updates a dynamic code view. This includes, renames, deletions, commits and code promotions and demotions. We like this because CA SCM does all the work and we cherry-pick sets of up-to-date code trees to build up an application source code stack for a build. We align Meister dependency directories with HRefresh-managed file system directories for a tight SCM (software configuration management) build.

This mechanism distributes the resource load for checking out code to times when builds are not required. It’s true that often times people want to build as soon as their code is checked in (or promoted), but on average it is a very big net win reducing build times.

This is just an example of the type of sophistication that is out there to prevent pre-build code check outs and save time on your builds.

We’ve seen a trend among our customers to simplify their build management infrastructure by going to fewer build machines with more CPU cores. Builds in particular use relatively more CPU resources than other resources as code is interpreted and compiled in memory and then finally written to disk. By reducing the total number of machines, rack space, procurement, administration and other IT overhead costs are reduced at great cost savings per machine eliminated.

Recently, I was at one of the big chip makers where they used dual quad-core CPU Linux machines for their development and builds. They had two machines and were able to control access to allow separate areas for development, testing and release builds in keeping with best practices. Having all the horsepower of 8 CPU cores on a single machine kept them from needing more machines.

Another customer does 6000 builds per month with Meister on just two build machines.

IBM, when selling BuildForge, likes to talk about big build server farms, because their tool does remote execution on multiple machines, as does Meister. However, BuildForge does not do builds at all. It can remotely execute your existing build scripts, but there is little real value add to that. BuildForge is also famously expensive. What happens over the next few years to the high investment in multi-machine remote execution software as the number of machines declines, perhaps dramatically?

A similar argument can be for Electric Cloud’s Electric Accelerator product. It’s possible in some cases, for C/C++ builds to gain an edge by pushing a compile operation to another machine, and then bringing it back. You would only do this to gain access to additional CPU resources. In the past, you might have 8 build machines that Electric Accelerator would farm operations out to. Now, you can pull all those operations into a single machine and there is no need for that functionality. Also, you are stuck with converting your GNU makefiles into other GNU makefiles.

Meister is optimized for multi-core CPU build machines and offers multi-threaded capability to both build events and non-build workflow events. You know where your build is and there are fewer dependencies on network resources. Both BuildForge and Electric Accelerator add additional overhead to build administration to coordinate across multiple machines - a dying practice, that no organization wants to invest in. Meister is the best bet for a future with fewer build machines with more horsepower.

• It’s the Ruby on Rails default DB
• Adobe Air
• Mozilla Firefox
• Mac OS X

It’s such a popular tool, that its creator, Dr. D. Richard Hipp, won a 2005 Open Source Award from Google and O’Reilly. About a year later, Dr. Hipp gave a Google Tech Talk introducing SQLite to Google. In that talk, he remarked on a number of features and benefits of SQLite:

• SQLite is designed to be embedded, it can be less than 250 KB in size.
• Some uncommon SQLite uses:
  • stand-in for client-server DBMS during testing/debugging.
  • Local database caching.
  • Implementing complex data structures.
  • Sorting large amounts of data.
  • Configuration files.
  • IPC via database.
  • Application file formats.
• SQLite is very convenient to use as a tool to teach basics of SQL, as it just works.

Of the uncommon uses for SQLite, I can speak to at least two of them, because that’s how we use it internally. Originally, I was looking for a good way to persist tree data to disk, since I was having timing issues with reading and writing linked-list data to disk. I literally googled “persist binary trees to disk” and one of the hits said “use SQLite”. Steve took this idea further and used the database as a way to do interprocess communication between the main omsubmit daemon and the child processes that run each activity. This allowed us to move away from shared memory and message queues that seemed to be a bit flaky.

As for the second point, SQLite is a good tool to teach SQL. But more importantly, it gives a really good insight into how to use a database. With the big “enterprise” databases, you read and write and go on your way. With SQLite embedded in your program, you have to actually think! For example, if you do multithreaded design, you can’t have two threads updating the database or you’ll get a deadlock. Furthermore, if you do a big SELECT with a bunch of processing for each row on a worker thread, you’ll get writer starvation on the main thread. We had to come up to speed pretty quickly in terms of good design to make sure we didn’t introduce new timing issues using SQLite, since our goal was to reduce timing issues.

We’ve also used SQLite in our om build engine executable to do more efficient wildcard matching and we’re seeing really good speed improvements there. That improvement will be available in our upcoming release.

Here are some tools to investigate SQLite:

• Perl DBI module. Great for prototyping code and databases, especially from the debugger.
• GUIs
  • SQLite Browser
  • SQLite Spy
  • SQLite Manager, which is a Firefox Add-on!

Hope this helps you get into using SQLite.

- Jim

However, even if you have the correct versions of Perl and Java installed, you might run into Perl or Java validation errors when you install Meister. The most common cause of these errors is conflicts with Oracle products. Oracle ships with its own version of Perl, which has been optimized to work with the Oracle database. When Oracle is installed, it modifies two environment variables: the PATH variable, and the PERL5LIB variable.

When Meister performs its Perl validation routines during installation, it recognizes the Perl version that Oracle has placed in the PATH. This is where the first conflict arises. Meister validates the Perl installation by running a `perl -v` command, and checking against the output of that command. Oracle’s version of Perl returns an unexpected version number string, and does not pass validation (this is by design, since Oracle’s version of Perl will not work with Meister’s Perl libraries.)

In order to pass Perl validation, then, you will need to modify the PATH variable so that the Oracle’s “bin” directory is no longer in the PATH.

Once you have cleared out references to Oracle Perl from your PATH, you will still run into one more error: Meister installation will hang if the PERL5LIB variable is definied. You will need to remove the PERL5LIB variable when you install Meister.

Openmake Support’s recommendation is to initialize a Meister install from a shell prompt. From inside the shell, modify the PATH variable to remove Oracle’s bin directory (where their perl.exe is installed), and unset the PERL5LIB. Once you have modified the environment, you can run the install of Meister. After Meister installed, you will be able to start it from your normal environment (including Oracle’s environment settings.)

This causes a number of problems, as other Visual Studio projects also use the “unmodeled” project type. For example, database projects return the same project Kind, and I’m not interested in them at all. I needed a way to differentiate between sub-project types where the overall project type was “unmodeled”.

A quick and dirty way to do this would be to look at the

project.FileName

property and see what extension it has. A “.dbp” extension would be a database project and a “.wdproj” extension would be a web deployment project. There are two things wrong with this approach. It’s frail — what if Microsoft updates database projects to be .dbproj files using MSBuild (they may have already done this in VS 2008)? More importantly, both database projects and web deployment projects do not implement the FileName property (it throws a NotImplemented exception).

The correct way to do this is to find out the project’s subtype GUID. Looking into the XML of the .wdproj file, and looking at the registry under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\8.0\Projects, I know the Project Type GUIDS are:

• Database : {4F174C21-8C12-11D0-8340-0000F80270F8}
• WebDeploy : {2CFEAB61-6A3B-4EB8-B523-560B4BEEF521}

You can also see this in a solution file. Each project is listed in this format:

Project("<Project Type GUID>") = "<Project Name>", "<Location>", "<Individual Project GUID>"

So, I needed a way for the project object inside of Visual Studio to return its project type GUID, so I could determine what type of project I’m dealing with.

At this point, I was at the limit of the Extensibility API. A quick Googling revealed that I would have to use the VSPackage interfaces, the underlying API of Visual Studio. Originally, I had thought that you could not do this without downloading super-secret APIs from the Visual Studio Integration Partner (VSIP) program, but according to Craig Skibo, the SDK is now free. You just have to access the Visual Studio SDK.

The MZ-Tools HOW-TO series has a nice article on how to access the project GUID for a project throught the IVsHeirarchy and IVsAggregatableProject interfaces. The two problems are that it relies on using the project.UniqueName property, which throws a NotImplemented exception in my case; and that some projects won’t implement the IVsAggregatableProject interface. I added a quick work-around for it, which is not perfect, but seems to do the trick. (Sorry for the weird formatting).

private static String getProjectTypeGuids( EnvDTE.Project proj )
{ String sProjectTypeGuids = ""; String name; Object objService; EnvDTE.Solution sln = proj.DTE.Solution; Microsoft.VisualStudio.Shell.Interop.IVsSolution objIVsSolution; Microsoft.VisualStudio.Shell.Interop.IVsHierarchy objIVsHierarchy = null; Microsoft.VisualStudio.Shell.Interop.IVsAggregatableProject objIVsAggregratableProject = null; int iResult; objService = _getService(proj.DTE, typeof(Microsoft.VisualStudio.Shell.Interop.IVsSolution)); objIVsSolution = (Microsoft.VisualStudio.Shell.Interop.IVsSolution) objService; try { name = proj.UniqueName; iResult = objIVsSolution.GetProjectOfUniqueName(name, out objIVsHierarchy); if (iResult == 0) { objIVsAggregratableProject = objIVsHierarchy as Microsoft.VisualStudio.Shell.Interop.IVsAggregatableProject; iResult = objIVsAggregratableProject.GetAggregateProjectTypeGuids(out sProjectTypeGuids); } } catch { // Since UniqueName is not implemented, try looping over the // project files in the solution to see if we get a match on the // project.Name. It's not ideal, as the project.Name might not // be part of the project Filename. // call once to get size uint nProjects; iResult = objIVsSolution.GetProjectFilesInSolution( (uint) Microsoft.VisualStudio.Shell.Interop.__VSGETPROJFILESFLAGS.GPFF_SKIPUNLOADEDPROJECTS, (uint) 0, null, out nProjects); // Call again once we know the size string[] projectFiles = new string[nProjects]; iResult = objIVsSolution.GetProjectFilesInSolution( (uint) Microsoft.VisualStudio.Shell.Interop.__VSGETPROJFILESFLAGS.GPFF_SKIPUNLOADEDPROJECTS, nProjects, projectFiles, out nProjects); name = proj.Name; foreach ( string projectFile in projectFiles) { if ( ! String.IsNullOrEmpty(projectFile) && projectFile.Contains( proj.Name) ) { name = projectFile.Remove( 0, sln.FullName.LastIndexOf("\")+1); break; } } iResult = objIVsSolution.GetProjectOfUniqueName(name, out objIVsHierarchy); if (iResult == 0) { Microsoft.VisualStudio.Shell.Interop.IVsProject objIVsProject = objIVsHierarchy as Microsoft.VisualStudio.Shell.Interop.IVsProject; if (objIVsProject != null) { // cast the project to get its Moniker Name string projFile; iResult = objIVsProject.GetMkDocument(Microsoft.VisualStudio.VSConstants.VSITEMID_ROOT, out projFile); if (iResult == 0) { uint dwReserved = 0; Guid pGuidProjectType; iResult = objIVsSolution.GetProjectTypeGuid(dwReserved, projFile, out pGuidProjectType); if (iResult == 0) { sProjectTypeGuids = pGuidProjectType.ToString().ToUpperInvariant(); } } } } } return sProjectTypeGuids;
} private static Object _getService( Object serviceProvider, System.Type type)
{ return _getService(serviceProvider, type.GUID);
}
private static Object _getService(Object serviceProvider, System.Guid guid)
{ Object objService = null; Microsoft.VisualStudio.OLE.Interop.IServiceProvider objIServiceProvider; IntPtr objIntPtr; int hr; Guid objSIDGuid; Guid objIIDGuid; objSIDGuid = objIIDGuid = guid; objIServiceProvider = (Microsoft.VisualStudio.OLE.Interop.IServiceProvider) serviceProvider; hr = objIServiceProvider.QueryService(ref objSIDGuid, ref objIIDGuid, out objIntPtr); if (hr != 0) { System.Runtime.InteropServices.Marshal.ThrowExceptionForHR(hr); } else if (!objIntPtr.Equals(IntPtr.Zero)) { objService = System.Runtime.InteropServices.Marshal.GetObjectForIUnknown(objIntPtr); System.Runtime.InteropServices.Marshal.Release(objIntPtr); } return objService;
}

So far, so good. I can now identify and differentiate database projects from web deployment projects based on GUIDs. The next step is to figure out how to instantiate the sub APIs for the web deployment projects so I can get at the real data I need.

• Due to a bad VPC image, the Windows 2003 R2 Server license on the VPC expired early. We had to burn one of our own W2k3 Licenses just to keep the VPC running. Since I understand as much Microsoft licensing as much as the next guy (i.e. not at all), I wanted to upgrade on our own licensed W2K3 machine.
• I wanted to see how hard it was to upgrade.

Well, it turns out, it’s really hard. Even without updating SQL Server or Sharepoint, and keeping the application tier and the data tier on the same machine, and upgrading Workgroup to Workgroup, it still didn’t work.

First, I had to uninstall all of these products. That’s a long list. The last install was the actual server install. And of course, it failed. The failure seems to be related to the user account doing the uninstall, but after three hours, I couldn’t get it to work, so I now have a junked install. I can’t uninstall the old version, and cannot install the new version without uninstalling the old one.

So, I’m now pulling down the new Virtual PC image for TFS RTM that runs until Dec 31, 2008 (the ever-helpful Martin Woodward has a list of all the URLs if you use a download manager). We can evaluate the RTM there.

- Jim

Show Annotated

I don’t know how long this has been in the Eclipse Editor, but turning on the Editor Presentation action set and then selecting the Show Annotated is really cool. You can highlight a function or method and then only work on that section of code in the editor.

AnyEdit

The AnyEdit plugin adds some really useful features to the Editor window for coding, like

• Convert tabs to spaces and back,
• Convert CamelCase to underscores,
• Convert to and from HTML,
• etc, etc

It’s already part of most of the EasyEclipse installs, but I just didn’t know about it. Here it is in action

Compare With Outline

I never understood why the Eclipse Compare view always created the “structured compare” panel in the top of the compare editor. It seemed to just waste a lot of real estate, especially since there was the Outline view just sitting there waiting to be utilized. I even thought about opening a bug on it … Well, don’t I feel dumb; it’s a preference

Hope these help.

– Jim

For unit testing, we’ve added two simple post build steps to our CLC builds: C Unit tests and .NET Unit tests.

For .NET, we use the extremely easy NUnit framework. This framework is really trivial to set up and use, and we can easily use it inside the Visual Studio IDE (before checkins) and during the build process. This post, towards the end (point #12), shows how to set up an “External Tool” in Visual Studio to run NUnit tests. You can also go with TestDriven.NET, (I think it’s free for personal use and $95 for a professional copy) which will integrate and consolidate NUnit, MSTest, NCover, NCoverExplorer and all the great unit-testing frameworks out there. We could also run MSTest, the Microsoft testing frameworks, but that’s not easily integrated into Visual Studio 2005 Professional. Since we’re just ramping up, and getting the unit tests up and running is my first goal, NCover will come later, as we push our unit testing coverage up to a respectable number.

For our C and C++ development, picking a unit testing framework was harder, as there is no clear leader in the space. After some Googling, I found this blog post by Noel Llopis from a few years back that compared the leaders in the space. One of his highest recommendations was for CxxTest, an incredibly simple unit test framework that only uses C++ header files. All the tests are written as header files, and are extremely easy to set up. Trust me, if I can do it, anyone can. It requires Perl or Python to auto-generate the .cpp code from your test headers, but since Openmake requires Perl, not much of a problem. It was so easy that I added a CxxTest rule to our standard MSVC Console Build Service so that everything is built and run in the same workflow.

– Jim

My XP partition isn’t going to be a full step-up. I am mostly going to run virtual machine images for our demos, and maybe do a bit of coding on the side. So here’s a list of what I put on the machine, and how I did it.

The most important thing I wanted to do was to set it up in a safe manner. As Jeff Atwood points out, the single biggest issue with Windows security is running as an Adminstrator. On my desktop, I run as an Admin, and I’m too far gone in terms of switching back to a regular user. But on the laptop, I wanted to see if I could run only as a regular user.

First I created two users, “jim” and “root”. “Root” is an admin account, and “jim” is my day to day account. I needed the admin account so that I could install the .NET 2.0 framework before installing the next piece.

The most critical piece was installing Sudowin, a sudo clone for Windows. This allows you to run certain programs (or all programs, if configured that way) as an Administrator, without being an admin. In the Sudowin About page, there is a long discussion why this is a different, and better, solution to running with elevated privileges than the native “Run as ..” or Vista UACs. This install allowed me to install the remainder of my software while still running in my local account. It also has a command-line “sudo” tool.

The remaining software that I installed was

• Microsoft Virtual PC and VMWare VM Player — for our demos,
• A text editor (I’m switching between Textpad and Activestate’s Komodo Edit) — for simple text editing,
• 7-Zip — free zip/unzip,
• Perl — needed for day-to-day scripting,
• CutePDF — for printing to PDFs,
• Microsoft Visual Studio 2005 and 2008, and NUnit, FxCop, and Reflector — for when I want to actually write code,
• Firefox,
• Microsoft PowerToys for XP– to tweak the UI the way I like it,
• Sharp Keys — to tweak the keyboard the way I like it,
• Sysinternals Process Explorer — to see what’s running,
• Winpatrol — to keep tabs on anything that might get surreptitiously installed or run,
• PuTTY — to SSH into the VM Ware Linux images.

So far, so good. I’ve been really impressed with my laptop as a Windows box, and more so with running the VPC and VM images for our demos. See you at VS Live!

– Jim

I moved to a Macbook Pro as my laptop about six months ago, and I really enjoy it. I also like the fact that I’m working in multiple environments. Mac OSX on the laptop, Windows XP on the desktop, and Linux on our servers.

But for our Microsoft demos, I have to run Windows on the laptop. Fortunately, some have said a MacbookPro is some of the best hardware to run Windows! My plan was to use Boot Camp to create a separate Windows partition (because I will want to run our demos at native speed), and then use VMWare Fusion to run off that partition if I want to run OSX and Windows simultaneously.

Preparing for the partition was difficult. Because I have been using my laptop for six months, I already have lots of kruft on the disk. The folklore is that Mac drives don’t get fragmented, but this is just not true. Boot Camp could not create the partition, and on occasion would go into kernel panic mode and bomb out. This thread discusses some ideas why I couldn’t partition. Based on their suggestions, I downloaded the demo version of iDefrag to have a peek at my HD. Files were scattered over the drive, and I needed a good 1/3 of the disk free at the end. I tried lots of manual moving of files, deleting unnecessary files, and running Disk Utility and fsck -f from single user mode. Quite a pain. After a few hours late Tuesday night, I decided my time was worth $35 and purchased the full iDefrag. Good thing I did — it ran for 26 hours compacting my disk! But it all worked out, and Boot Camp worked like a charm after that.

I elected to go with a 32Gb NTFS partition. NTFS is a bit quicker and more robust than FAT32, and the downside of not being able to write to the NTFS partition from the Mac didn’t seem like a big deal, because I have external drives that I use for big files.

Funny to be back to the dual-boot days of when I ran Windows 2000 and Red Hat on my desktop. But the VMWare Fusion technology is great — it’s the best of both worlds. In my next post I’ll detail how I set up the Windows machine, as I wanted to try to keep it clean.

– Jim

I ran across a newer compiled Java language, JavaFX. Check out:

Developing Your First Compiled JavaFX Script Program
— The next JavaFX Puzzler will be posted at 18:00 GMT (1:00PM EST) on Wednesday, November 28, 2007. It will be a compiled JavaFX Script puzzler, so you’ll need to build the JavaFX Script Compiler in order to participate in the Puzzler. To help prepare you for the Puzzler, I’d like to you to develop a very basic ‘Hello World’ style program.

This is interesting since they decided to write their own command line compiler instead of just sticking with Ant Tasks. Looks just like another pre-processor for Java code.

We’re obviously not the only ones interested in a thriving Team Foundation cross-platform community. Microsoft wants Team Foundation Server to be an SCM tool for all shops, not just .NET shops. Teamprise got the ball rolling by creating a cross-platform client for Team Foundation, so that one can to check-ins and check-outs from TFS (kinda important). They are also getting interested on the build front, too. Martin Woodard posted an extension for running Ant Builds from TFS. This is a important first step — it allows you to reuse existing scripts from within TFS on a Windows machine. We go further; we support builds on multiple OSs and all kinds of legacy builds and other compilers as well as Java, as we should given that it’s our primary focus, whereas SCM is Teamprise’s primary focus.

The next release of Teamprise (3.0, now in preview), allows the Teamprise clients to access the Team Build management functionality of TFS, what was previously only available in Visual Studio Team Suite. I’m looking forward to accessing this functionality to see how we can plug into it and leverage our plugins to automate adding the OpenMake integration to Team Build.

– Jim

Our hard work is starting to get noticed. Brian Harry blogged about our “X-plat builds”. Feels good to get noticed at that level.

- Jim

Still don’t understand how the map files that relgen uses is related to CVS. I was looking at the CVS tags and I can see the ones for major release out there, ie R_3_3_1_1 and ones based on dates. But the map files are referring to other version tags such as the date tags. I looks like its a pick and hope type of layout for the map file when choosing versions of dependent source to pull from CVS.

Relgen has a buildAll.xml file that I am going to try to run. Any takers on wether its going to work?

I started trying to build Eclipse today. I knew it would be a big project. As I was suspecting, finding where to start is hard. I know I need to check out the source but how. Looks like you need to use .map files that then reference the appropriate CVS tag and repositories.

I am trying to do this from the command line so I can automate the process. The goal I am trying to get to is a step by step process of building Eclipse not a bunch of pasted together processes from each project. Looks like there is no one way of pulling all the pieces together.

Stay tuned to see if I actuallt get the source checked out.

1. Running the debugger: we compile the Windows versions of our apps using the native Microsoft compilers, which Eclipse CDT debugging doesn’t yet support. So to run the debugger, I switch into Visual Studio. This is the one place where I wish the Eclipse debugger keyboard shortcuts matched Visual Studio’s (yes, I know I can remap them, just haven’t done it yet!).
2. C# development. In the last year or so, we’ve been working hard on our Microsoft integrations, through the MSBuild API on the client side, and a Visual Studio Addin. Here it would be crazy to attempt to use Eclipse.

However, I still need to check into CVS, which is trivial to do from Eclipse, but not natively supported in Visual Studio. I was back to the bad old days of switching out of Visual Studio to Windows Explorer and firing up TortoiseCVS or WinCVS … until I found TamTam CVS.

TamTam CVS is a great little ($10) add-on to Visual Studio that provides, as Dave says:

… a Microsoft Source Code Control (MSSCCI) compliant implementation that seamlessly integrates Concurrent Versions System NT (CVSNT) client with Microsoft Visual Studio / Visual Studio.NET 2002/2003/2005, MatLab, FoxPro, 3D Studio Max and other MSSCCI compliant applications.

I’m also using it in Visual Studio 2008.

TamTam “just worked” when I installed it, giving me 95% of the functionality I wanted right away. I felt there was one feature missing (the ability to store “template” commit comments, which we use to link commits back to the issues that we are fixing), and I asked Dave if that feature were possible. He added it in under a week! (See version 1.2.1). Thanks Dave!

For $10, less than you might spend on lunch, you’re avoid tons of seemingly minor annoyances that really add up during the day. And the support was excellent. I haven’t used TamTam SVN, but I bet it’s of similar quality.

– Jim

It appears that the libapr is using the WSARecv, WSASend etc functions instead of the recv and send functions. Also, it setup the connection a little differently than I was. But it was very easy to use. I need to test on Unix but I don’t suspect too many problems.

Check it out at: http://apr.apache.org/

Here’s the step to handle reading chunked data.

1. Read the http headers from the socket looking for the Transfer-Encoding: chunked header
2. After the headers, there will be a new line chr(13) and then a line with only a single number on it. This number is the length of the next chunk of data in hex (use strtol()) to change the hex to decimal)
3. Read from the socket the number of bytes in step #2
4. Next is a possible new line chr(13) or not (depending upon the server) and then the hex number on the next line.
5. You then repeat step #3 until the hex number is 0

Make sure you test against several different servers/clients since the format (extra newlines or not) are not consistent between the servers.

Also, look under the covers to see whats happening. For example, the WebSphere deploy script is really a shell/batch script that wrappers a call to a Java class. Inside the script you can get a better idea of what it expects as inputs and outputs.

The compilers will sometimes use a response file in which all of the inputs and parameters are put into and then the response file is passed to the compiler. Compilers that use response files are great since they avoid the problem of limited command line character limits.

I asked the customer to reinstall, and once again, everything would be working fine… until the machine was restarted. This went on for a few days, until we figured out that the Mojo Uninstaller that was deleting the files. However, it was the Mojo Uninstaller from a previous install that was deleting the files. It was deleting files from beyond the grave! (or the Recycle Bin)

The Mojo Uninstaller is based on a third-party software that stores a list of files that need to be deleted in an obscure location on the file system. Normally, this list is cleared once all files are deleted. However, in the customer’s case, not all files were deleted: the Knowledge Base server was still running during the uninstall, so any .jar files used by the KB Server could not be deleted. Therefore, the “files to delete” list still had these jar files listed. Once these files were freed up (i.e. on restart of the machine) the uninstaller jumped in and deleted the files. Since this list of files is stored in an obscure, relatively inaccessible place, we could not tell the uninstaller to quit uninstalling these files.

So, DO NOT use uninstall.exe when uninstalling Mojo on Windows. Unless of course you like headaches.

“So, Mr. Support Guy, how do I uninstall Mojo?” you ask?

Glad you asked. It’s a simple process:

1. Stop the Knowledge Base server - this will free up the KBServer jar files for uninstalling
2. delete all the files from the directory where you installed Mojo.
3. Delete the omenvironment.properties file.

If you have already uninstalled Mojo using uninstall.exe, and files are mysteriously disappearing from your Mojo install, then we recommend you do the following:

1. Uninstall mojo using the above process.
2. Reinstall Mojo in a different directory, so the uninstaller cannot find the .jar files it wants to delete.

To simplifiy operation in Mojo and Meister, we have introduced the omenvironment.properties file to help manage your Mojo/Meister environment. This file contains OpenMake specific environment configuration and is automatically set up for you upon install of an OpenMake Software product. For Windows users, this file is found in two locations: the “openmake” directory under your user’s “Application Data” folder (C:\Documents and Settings\USERNAME\Application Data\openmake) and the client\bin directory. For those of you on UNIX or GNU/Linux, the file is found under the “.openmake” directory under your user’s home directory, as well as the client/bin directory. The purpose of having multiple files is so that different users could have different environment settings, by using the file in their user home directories, or the environment can be managed globally with the file in the client\bin directory. If the file exists in the user’s home directory, then that file will take precedence. Otherwise, the file from client\bin will be used.

The main function of this file is to insure that when the Mojo or Meister interfaces are used to kick off a Workflow, these core variables required by Mojo/Meister are guaranteed to be part of the environment. Also, by setting OPENMAKE_SERVER, it insures that you can connect to the KB server when you start the Mojo/Meister workbench.

There is another place where the omenvironment.properties file comes into play and sometimes causes problems: upgrading or reinstalling Mojo or Meister. I will take on that topic in my next post.

Having worked in the field for close to 15 years, I feel I have earned the right to comment on what these terms mean:

Build Management - The ability to manage the creation of binaries including the enforcement of source file locations, third party library usage and compile flags. It also must include the ability to track all artifacts used to create the binary, not just show what was checked into a SCM tool. 

Build Automation - The ability to run automatic builds, without the need to baby sit ad hoc scripts. Build Automation is not the process of scheduling scripts or executing scripts from some sort of workflow tool. This is just “script scheduling” - not build automation. There is nothing automatic about writing build scripts, or babysitting build scripts.

Dependency Scanning - This is the process of watching the compiler and scanning source code for nested references to source code and libraries. It is not running a “check-out” report against a SCM tool to determine what was checked out to the local build directory prior to a build, referred to as a Bill of Material (BOM) report. A BOM report cannot tell you what it does not know, and many artifacts there are used to create your binary come from places other than the local build directory. Dependency Analysis is just that, a thorough review of what occurred in the build, even if an SCM tool is not used. So if your build relies on an SCM tool for “dependency scanning”, it is not doing dependency analysis.

This list can go on for some length, but I will stop here. The point is that the Build Management market can be as confusing as builds themselves. This is proof that we, as an IT community, have not spent enough time discussing, improving and standardizing this critical component of application development.

Well guess what - developers don’t really work that way anymore, but they continue to rely on this waterfall approach to builds. What developers do is work within IDEs where their builds are nicely managed for them. They then try to setup a continuous integration build, or nightly build process, and find themselves falling back into the waterfall process model and calling on a build expert to create an Ant, Maven or Make script. If you are trying to improve your process, automate the creation of your build script, in the same way that you use IDEs to perform automatic code creation. You cannot do agile development without streamlining the build step. Developers should be able to build inside or outside the IDE without the overhead of a manual scripted process designed for a waterfall approach to application development. I’m not saying scripting is bad, just that done ad hoc, it is highly redundant and forces agile development into a waterfall phase of build management. Just some thoughts to chew on.

All license requests now need the following information to generate a proper license.kb:

- hostid of the KB Server (see installation guide for how to obtain this for the different OS’s. On windows it’s vol c: and most unix flavors, a derivative of the uname command)

- host names of any Remote Build Servers that are licensed, if any.

- number of named licensed users.

- contact information (name, e-mail and phone number is sufficient)

- whether or not you have a permanent license or are just requesting a demo version.

For more information about licensing, you may also always contact our Openmake sales team.

So to get your Windows exes signed you need a certificate from Thawte, Verisign or a like. We have a certificate that we use for signing our jar files that run via a Java Applet. (Only way to make a Java Applet talk to multiple machines and the file system).

The certificate is stored in the key3.db and cert7.db Netscape data files. The trick was to get the certificate from there into something Microsoft would recognize, since you need to use the Microsoft signing tools instead of the Netscape ones to sign a Windows exe.

Here are the steps I did to get me over to a Microsoft certificate store

1. Download the Netscape tools from:
http://www.mozilla.org/projects/security/pki/nss/tools/index.html

An awesome set of tools and documentation for Cryto stuff!!!

2. Since my cert was stuck in a cert7.db instead of a cert8.db I had to go back and get the version of the tools to handle that.
I used the nss-3.6.1.zip and nspr-4.2.2.zip pre-built packages for Windows.

3. I used pk12util to pull the certificate out of the cert7.db

It prompts you for the password to the DB and then another password for the .p12 file

4. At this point I had the certificate in the PKCS12 format in the mycert.p12. Next was to add it to the Microsoft certificate store.

Just run certmgr.exe and a GUI will walk through getting the certificate imported. I just took the defaults. Used my password that
I saved the .p12 file with and Marked the private key as exportable. I left enable strong private key encrytion off. Took the default
of Automatically picking the store. And your done.

5. Now that its in the Microsoft store you can sign your exes. You need to use the signcode.exe or signtool.exe (depending on the version of the Microsoft SDK) to do the signing.

Note: Use a timestamp server /t option

6. To check that it was signed correctly, run signtool.exe verify or chktrust.exe

Thats it!!

It appears that the first fgets() would get all of data except the EOF. The next time around fgets() should return NULL since EOF would be read. But what happens is that the program uses 99% CPU and gets stuck there. I didn’t go into the debugger to see exactly what was going on but the fix was to allocate the buffer bigger than the file size. And use that larger size on the fgets().

Fixed Code

The Virtual Build System that they setup was to have many VMWare images running and have each image run a subset of the build. I theory it looks like it would work well. Each VMWare image dedicated to running a slice of the whole build, so the compiles should run faster. But this is not that case, VMWare actually is getting in the way. No, matter how well VMWare works there are 2 things getting in the way. First, the I/O (network and disk). Secondly, the number of machine instructions executed is high. When the compiler runs, VMWare is running the compiler instructions and passing the instructions on to the Real OS. For VMWare to do this the processor must execute the instructions for VMWare itself as well as the compiler instructions. Therefor, no matter what you do the build is going to be slower.

The Virtualization does buy you a level of abstraction to move a build from one machine to another. But this begs the question, why is the build machine dependent? Shouldn’t a build be able to be recreated on any machine at any time? Openmake was designed with that requirement in place. To repeat a build on another machine you need a OS to run on. Does it need to be the same OS?, no. Running a build on Windows 2000 and Windows XP will produce the same resullts if the compiler version and source code are the same. The compiler can be checked in with the source and checked out with the source if you choose to do so to get a completely traceable build.

Most shops have a standard that they install for the compiler so checking it in and out is usually not need. I recommend just putting the CD on a shelf or off to Off-site storage for disaster recover and archival purposes.

I will get to the first bottle neck and solutions in a future post.

BTW, the Build Meister did confirm that Virtual Build System was slower in the testing that they have been doing. I just reconfirmed their findings.

OPENMAKE_SERVER will normally be set automatically whenever you install an OpenMake product. The form of the variable is as follows:

http://<KB SERVER HOSTNAME>:<KB SERVER PORT>/openmake

Where people often run into problems is the hostname part of the variable. The problem is there is sometimes difficulty in resolving the hostname to find the machine, in particular when your machine is part of a domain that uses a name server. The way to work around this is to have OPENMAKE_SERVER use the IP address of the machine.

In the past few weeks, I have encountered about three different support cases where this fixed whatever error was reported (and they were completely different errors that were reported). Since the errors that come from this problem are often misleading, the best policy is to set OPENMAKE_SERVER to use the IP address instead of the hostname whenever possible.

Overtime the underlying development languages and environments have moved towards the user and minimized the need to know the details of the underlying machine. Object oriented techniques and IDE’s are examples of this, and most developers today work in such a space.

This virtualization has not taken place in the build area and it remains a magic art at many unenlightened  companies, outside the bounds of control. Many project plans ignore or even forget the build, config and deploy steps in their program time an resource estimates, causing last minute panics and over-time at the end of many projects! But why fix it - we have all lived with this for years.

It matters today for three reasons:

1. New programming techniques such as Agile are increasing demand for non-redundant, repeatable, automatic builds.
2. Internal audit is oft uncovering the fact that build / release management is not repeatable, traceable nor is there separation of duties.
3. One key test of the ability of an organization to manage its home grown software is how quickly it can respond to production issues. These can often be more costly than missing a delivery commitment in the first place. Every configuration in production needs to directly point not only to the source code that produced it, or the environment in which it resides but also the libraries (application, opensource etc) that it utilized. Only the actual build step really knows what is in the binaries.

These three problems can be addressed by virtualizing scripts up to a template / build method that is abstracted to a degree that minimizes the environment and hardware and for that matter the compilers. By using build methods the management software (build management)  inherently achieves the benefits of build reuse, auditability and dependency discovery.

Some vendors claim that these benefits can be achieved with a process automation tool - most notably IBM / Build Forge and Electric Cloud. This is a step in the right direction but amounts to trying to herd cats, for if you do not virtualize the build scripts you are attempting to manage chaos with a process - it may look good for a while but will surely break.

1. Windows2000, Windows XP, or Windows 2003 server.

2. Active State distibution of Perl installed, with “bin” folder of the installation included in the PATH environment variable; 5.8.8 is the current preferred Perl version, though 5.6.1 is supported.

3. Any Java Runtime Environment version from 1.4.2_05 up to the latest 1.5.0 revision.

4. User must have privileges to edit the registry.

Once these conditions are met, the installation is merely a matter of unzipping the supplied zip file into its own directory and running mojo.exe.

Also, please note that these instructions are for new users who have not installed other OpenMake products. If you are an existing OpenMake Software customer and have problems with installing Mojo, please contact your OpenMake support representative.

Now with Live Writer front ended to WordPress I am set to go.

While most people liked the new features of omsubmit, we still found that some customers wanted things to behave like they did in 6.3, before omsumbit came to be. One particular reason people asked for this is that when running builds through omsubmit, the logging is written to a local log file instead of appearing directly at the console.

Anyway, if you want to bypass omsubmit, all that is needed is to pass the -l flag to bldmake or om. This will cause the build to effectively run as it did in 6.3. Note that this means that non-dependent build steps will not execute in parallel and that the build activities will not be logged to the Build Manager.

As of a few weeks ago, Openmake version 6.3 can be used to build Visual Studio 2005 applications, though the full feature set of the 6.41.1 Add-in is not available. With the latest update, 6.3 versions will be able to have Openmake automatically generate the Visual Studio 2005 Target Definition (.tgt) files and run VS2005 builds. The main deficiency however, is that Openmake 6.3 cannot run Openmake builds from inside the VS2005 IDE, which is something that can be done with the 6.41.1 Add-in. For more information on running VS2005 builds with Openamke 6.3 or 6.41.1, please contact an Openmake support representative

From time to time, I will be involved in a support case that will stump me for hours, days, or even weeks. These are cases in which absolutely nothing seems to work. The most basic step of the most basic build will crash and burn, leaving me with no place to even begin looking for clues as to what went wrong.

When nothing seems to work, my first instinct is to look for the most complicated explanation - the issue must lie deep in the guts of Openmake, the dark hinterlands of Openmake code that I haven’t touched in my two years as support person. In reality, the most catastrophic problems often have the most basic explanations.

One of the basic elements of an Openmake installation is the Perl libraries in the command line client. These libraries power nearly every aspect of an build, from dependency analysis to logging. Openmake’s Perl libraries are automatically installed during Command Line Client installation. However, if Perl itself isn’t installed, these libraries are useless, and a build will fall apart at the seams… literally nothing will work.

Now, one of the first things I check for when nothing seems to work (instead of trying to pick apart the innards of Openmake), is a working installation of Perl.

Due to changes in how Internet Explorer handles JavaScript, the Openmake Web Client will not load in Internet Explorer 7 without a small fix.

In order to allow the Web Client to load in IE7, you need to comment out a line of code in the index.html.template file in your <KNOWLEDGE BASE SERVER>/tomcat/webapps/openmake.ear/openmake.war directory.

Open the index.html.template file in a text editor, and do a search for the following line: if (_ie == true)

Two lines beneath this conditional (between curly brackets) is a line that starts with: document.writeln(’…

Comment out this line by typing two forward slashes (”//”). Restart your Knowledge Base server, and you will be ready to use Openmake with Internet Explorer 7.