Check out the full results on technistas.com.

- Matt Laudato

Director, Customer Solutions

matt.laudato@openmakesoftware.com

http://technistas.com/2010/03/22/the-build-engineers-desktop/

Happy Building!

- Matt 

We announced today that we have become a member of the CollabNet technology partner program adding OpenMake Meister and Mojo to the collabXchange, an online integration marketplace.  CollabNet TeamForge with Subversion and OpenMake Meister are ideal solutions for improving distributed software development practices, particularly around agile methods from check-in through build and release.  

Meister extends the centralization features of the CollabNet framework into the critical step of compiling and linking the Subversion-managed source code.  Meister allows the CollabNet framework to orchestrate the complexities of the software build, improving software quality and minimizing costly bottlenecks due to broken or ‘bad’ builds. Together, Meister and CollabNet offer a complete solution to enterprise ALM challenges.

“This partnership furthers our commitment to delivering industry-leading solutions to our customers.  OpenMake Software was added to our list of technology partners for two reasons.  First, their Meister technology improves and accelerates the Continuous Integration process for Agile developers” states Jim Ensell, VP Business Development, CollabNet. “In addition, it delivers intelligent, enterprise level audit controls around the build and release process.”

“OpenMake Meister integrates with Subversion to deliver accelerated builds for both .Net and Java improving continuous integration by eliminating build bottlenecks.  It also exposes all Subversion managed source used during the build and release process providing a complete enterprise level audit trail” explains Tracy Ragan, COO and Co-founder, OpenMake Software.   OpenMake Software also offers a free Continuous Integration Server, called Mojo that integrates with the CollabNet framework.

CollabNet provides the industry’s most open and extensible platform for distributed ALM, enabling CollabNet Technology Partners to rapidly and easily create integrated products for distribution on collabXchange, expanding their potential customer bases to include the millions of software developers using the CollabNet platform today.

Yes, you can do this for Java, .Net or any native language such as C/C++.  What does this buy you. First, in most cases your CI compile and link step will execute in less than a few minutes.  Secondly, you are given a platform for doing pre-flight builds where any developer can re-compile the entire application, locally, without wasting an hour (or a few hours).

If you want your CI process to run really fast and accurately, make your builds “smart” so you don’t have to run “clean all” every time you execute a pre-flight or CI build.  Hey it is 2010 already. Start using your computer to orchestrate your compile and link process and stop writing static scripts that lack the ability to support your development process without your constant attention.  This is real build automation.

For more information on how Meister’s builds are “smart” see “Smart Builds for Fast Continuous Integration” found at http://www.openmakesoftware.com/build-management-white-papers/

“Build Avoidance” should be a primary function of your Build and Release Management solution.  It is not “new” by any means.  The biggest challenge with “Build Avoidance” is to support it for all languages from Unix C to .Net and Java. For example,  the “gnumake” based scripting languages only supports C/C++ languages. Ant/Maven scripting supports “pseudo incremental builds” which really is the opposite of what you would think.  Maven’s “pseudo incremental builds” does a “clean all” on the Projects that have changed, not on just the source code that has changed.  Doing a “clean all” is not avoiding a build it is making sure that objects will be rebuilt by deleting them.

Meister has supported Build Avoidance for all languages for over 15 years.  Only OpenMake Meister can actually make the claim for C/C++, .Net, Java and yes even COBOL.

So the moral of the story is, if your build management tool is not supporting Build Avoidance, or some how touts that this is a new concept, you should look for a better Build and Release Management solution.

The demo can be viewed from the OpenMake site here. If you’re interested in a writeup of the technical details, you can also visit my blog Technistas, where I talk about this system in a recent blog post.

 Happy building!

 - Matt

http://technistas.com/2009/12/09/practical-salesforce-programming/

Happy deploying!

- Matt

Check out my new post on comparing build management in Eclipse C and Java development tools, here:

Agile: A Set of Common Sense Optimizations

Check out my new post on comparing build management in Eclipse C and Java development tools, here:

Comparing Build Management in Eclipse CDT and JDT

A workflow management tool is only as intelligent as the scripts it is being asked to execute.  Build management tools that function like a job scheduler simply takes a command an executes it on a particular machine.  If that command is a make, maven or ant script, it is going to send that script to be executed on a particular machine.  That is NOT Distributing the build. It is distributing the build script.  Build scripts are rarely written to support distributed build processing, so your workflow solution cannot magically turn that script into an intelligent process that can be distributed.  In fact most scripts have no understanding of dependencies, so there is no way for them to know what order to build the application, much less distribute those calls out to external machines.

OpenMake Meister does understand dependencies and does have the intelligence to know the order in which to build artifacts.  In fact, out of the box, it does this for your builds, on a single machine using multi-core processors calling the compilers in multi-threaded mode.  So if you are looking for speed and need to distribute builds across multiple machines, look no farther than Meister.  This is what Build Automation is.

Selvi’s Blog

Adam Gabor’s Blog

Sean Blanton’s Blog

The content up to this point will appear in both places for convenience. Please update your RSS feeds if you are using them. See you there!

Sean

I’ve set up a JBoss build in Meister involving a web service and consumer. This was originally developed in MyEclipse, but the Meister Eclipse plug-in allows the native Eclipse build to be externalized (a better approach than Maven). So, I am just dealing with that, which simulates a development continuous integration or QA build.

I have a workflow that has the two main Meister build tasks, “Generate Build Control File”, and “Build Targets”. The first gathers local and server metadata consolidating that into something like a makefile or Ant script depending on your perspective. The second is similar to running make or Ant.

This is a simple workflow to illustrate our messaging technique. Normally, we’d use an email notification task, but I’ve replaced that with an activity called “Tweet”. This calls a simple Perl script, called omtweet.pl that uses Net::Twitter. Here is why we use Perl a lot - the code is simple to update your Twitter status:

1
2
3
4
5
6
7

use Net::Twitter;
 
my $twit = Net::Twitter->new(
  {username => "myuser", password => "mypass"; }); 
 
my $result = $twit->update(
  {status => "My current Status"});

I created a slightly more elaborate script that takes arguments and passes in a “Build Successful” message. If Java is your thing, look at Davanum Srinivas’ post and for .NET, there is Twitteroo.

Next, I run the build and we see the steps complete in real time on the workflow monitor.

The key is the last step, of course:

And, the proof is in the pudding:

I used the $(JOB_NUMBER) macro in Meister to get an incremental build number passed to omtweet.pl. Of course, following @builds and choosing device updates, I got a text message on my cell phone. I tried to get a picture of my cell phone with the message, but I haven’t worked out how to do that yet and it turned out terrible.

Well, this is a simple example, with a lot of possibilities. Meister wraps all build output in HTML and pushes to a web server. I’d like to include a shortened URL for that in the update. Currently we don’t have anything outside of a firewall, maybe I’ll try Yammer for that.

If you have any other suggestions, let me know. @seanblanton

I have run maven a couple of times but don’t use it.  (I had to build some open source jars).  And today I decided to clean up my harddrive on my Mac using OmniDiskSweeper.  Low and behold maven created a hidden directory in my home directory called .m2.  In that was 320MB of jar files and classes.  Why does maven need to hide that?  Shouldn’t it be nice and put it in a location that I know about so I can manage it. Instead of hiding junk on me.

The idea comes from Lean Manufacturing principles developed by Toyota applied to software development. An “andon” paper lamp was a way to let other workers know there was a problem on the assembly line. The idea for builds is that while a build is running, an amber light flashes, letting other developers know not to run a build, a green light signifies the last build was completed successfully, and a red light means the build is broken and needs to be fixed.

While I’m sure it is an entertaining exercise to set this up, some simple software automation could achieve the same results, and it has obvious faults, like what if I am working from home?

Email is the de facto standard for build notification. There is a hardcoded mailing list somewhere and you get the email, even if you don’t want it, and you have to go and check your mail to receive it.

Twitter, on the other hand, puts the user in control of the notifications in true Web 2.0 fashion. Tweets are also ideal for receiving notifications via cell phone, where email is generally not. Consider the following cases:

• I can follow the build when I want, and unfollow when I want. No going to an admin to remove me from the list.
• I can turn on device updates so I get an SMS message about the result (maybe an important release is coming up), or I can turn off device updates, say, when I’m on vacation.
• I can retweet (forward) the build notification.
• You can put links in the message. Meister uploads all build logs in HTML format to a web server and we would include the link to that.

About a year ago, I used Meister with a post-build activity to Tweet the build result after a build. The activity was a simple Perl script using Net::Twitter. Let’s just say this experiment sort of fell on deaf ears.

Now I’m ready to roll with this again and you can follow my progress. I had created a Twitter account, @builds, for this purpose. In a day or two, you’ll see some tweets from that guy from one of my builds.

I’m interested in what other people have to say. Meister already uses Eclipse RCP as it’s front end, but most other build tools are still command-line and in the 1970’s. Let’s bring build management tools into the 21st century with Web 2.0 features.

Ask for a build and get a build. Ask for continuous integration and test automation and coalescence and get it with Meister. Product owners should not be losing cycles to debugging builds, tweaking parameters to get different types of builds or in resolving confusion about what did or did not happen in a build or continuous integration workflow.

The client-server architecture and highly reusable build and workflow metadata extend the ScrumBut elimination to all developer machines and continuous integration build servers, creating a highly consistent build. Team members working on these different machines know exactly what to expect.

Since they know what to expect from the build, they can focus on resolving actual integration issues and testing rather than trying to make the build work in a consistent manner.

I posted a comment about the reluctance of script writers to give up their scripting on their blog here.

As customers go to machines with more and more cores, fewer machines are needed in the application lifecycle infrastructure, particularly for builds and code retrievals - the most resource intensive functions. This is helping to simplify the infrastructure, reduce maintenance and administration and drive down costs.

Several of our customers are running around 5000 builds and non-build workflows per month on two machines. The primary reason for two machines, in fact, is for disaster recovery, and the goal is to run both machines at less than half capacity so that in the event that one machine (or datacenter) fails, all the current capacity can be run as a contingency on the one machine that’s left.

Thread control is very simple with Meister and Mojo. Both use the omsubmit dependency manager program to handle this. Meister’s om program translates build events into workflow steps using omsubmit. The OMSUBMIT_MAX_USER_PROC value sets the maximum allowed number of threads.

You might think that if you are running dual, quad-core build machines that you should set the max threads at 8. However, Meister posts build operations to one thread and the associated logging operation to another. Compile operations notoriously use a lot of memory and CPU resources, but the logging operation posts to a server and waits for the operation to complete. There is really no disadvantage to setting the max threads higher than 16 in this case, so go ahead and do it.

As a non-build workflow example, I worked on JBoss deployments to 48 Linux machines. The workflow was parallelized into 48 activities each of which deployed to a single machine in parallel. The deployment activity was largely a remote execute operation that extracted archives on the remote machine. The extraction took about a second for a medium sized application. Again, this is a waiting situation where machine resources are essentially idle while the thread is in use, so use more threads. The machine was a dual, dual-core build machine and we set OMSUBMIT_MAX_USER_PROC to 50.

Watching the workflow monitor as the deployment ran, we could see roughly half of the machines light up (meaning actively running) at any one time and the entire deployment process synchronized all 48 machines in a little over two seconds.

So, don’t simply match your machine’s CPU threading capabilities - overclock! Aim high for max threads and try to determine where your performance is optimized. I’d love to provide you with some metrics as a function of thread count, but usually once something is working it’s on to the next project. I barely have enough time to blog!

Let me explain, Fred. The simple approach most of us take (and have to take when getting things started) has developers commit, commit, commit, and when it is time to deploy, check out the code, do a build, and then deploy the application. There is room here for both problems and optimizations. Doing a full check out of the code tree is more costly in terms of time than checking out only what has changed. Updating the code tree with a single commit is less costly than updating with a large number of commits.

You may be limited by the technology in-hand and how much you’ve invested in learning the technology and possibly customizing it. For example, if your file control tool can only do a full check out of a source tree, or that’s the only command you had time to implement in order to meet the deadline, or you don’t trust your tool to do incremental updates, then you are basically running the longest builds possible.

On the other hand, if you could update the code tree every time a developer does a commit with only the changed files, then you are ready to execute a build at any moment. This requires some deft manipulation of your file control tool, and that’s why you don’t see it more often.

You might think “continuous integration” will take care of this. Developer commits, update checked out, build is run. However, you may end up with a build, test execution and deployment that takes longer than the typical time between developer commits. You still have to do incremental updates and it only solves the problem in cases with very low developer activity.

I’d like to point out one tool that does an excellent job of post-commit code checkout, CA Software Change Manager for Distributed. CA SCM (for short) is the tool, formerly known as Harvest, from the company formerly known as Computer Associates. CA SCM is a highly scalable (1000’s of developers) file control tool with a great lifecycle process model. We at OpenMake Software still have our very first customer still using OpenMake/Meister with CA Harvest/SCM after 11 years. While we have a reseller arrangement with CA, our partnership with CA in services has extended to 14 years.

About 10 years ago, OpenMake Software developed an integration with the then, Platinum Technologies’ Harvest product, modeled after the now dead Computer Associates product, Endevor Workstation, that had an excellent post-action code tree update. (Endevor for z/OS, a.k.a CA SCM for Z/OS is still very popular and has a similar functionality called ‘output libraries’ - following all this?) Our integration had the horrific name, ‘Har-refresh’.

As product partners, we finally transitioned Har-refresh from an external add-on to CA who have turned it into a core functionality of the product, called Hrefresh (a better name.) Rather than simply a post-commit check out, HRefresh updates the code tree after any action that updates a dynamic code view. This includes, renames, deletions, commits and code promotions and demotions. We like this because CA SCM does all the work and we cherry-pick sets of up-to-date code trees to build up an application source code stack for a build. We align Meister dependency directories with HRefresh-managed file system directories for a tight SCM (software configuration management) build.

This mechanism distributes the resource load for checking out code to times when builds are not required. It’s true that often times people want to build as soon as their code is checked in (or promoted), but on average it is a very big net win reducing build times.

This is just an example of the type of sophistication that is out there to prevent pre-build code check outs and save time on your builds.

However, even if you have the correct versions of Perl and Java installed, you might run into Perl or Java validation errors when you install Meister. The most common cause of these errors is conflicts with Oracle products. Oracle ships with its own version of Perl, which has been optimized to work with the Oracle database. When Oracle is installed, it modifies two environment variables: the PATH variable, and the PERL5LIB variable.

When Meister performs its Perl validation routines during installation, it recognizes the Perl version that Oracle has placed in the PATH. This is where the first conflict arises. Meister validates the Perl installation by running a `perl -v` command, and checking against the output of that command. Oracle’s version of Perl returns an unexpected version number string, and does not pass validation (this is by design, since Oracle’s version of Perl will not work with Meister’s Perl libraries.)

In order to pass Perl validation, then, you will need to modify the PATH variable so that the Oracle’s “bin” directory is no longer in the PATH.

Once you have cleared out references to Oracle Perl from your PATH, you will still run into one more error: Meister installation will hang if the PERL5LIB variable is definied. You will need to remove the PERL5LIB variable when you install Meister.

Openmake Support’s recommendation is to initialize a Meister install from a shell prompt. From inside the shell, modify the PATH variable to remove Oracle’s bin directory (where their perl.exe is installed), and unset the PERL5LIB. Once you have modified the environment, you can run the install of Meister. After Meister installed, you will be able to start it from your normal environment (including Oracle’s environment settings.)

I ran across a newer compiled Java language, JavaFX. Check out:

Developing Your First Compiled JavaFX Script Program
— The next JavaFX Puzzler will be posted at 18:00 GMT (1:00PM EST) on Wednesday, November 28, 2007. It will be a compiled JavaFX Script puzzler, so you’ll need to build the JavaFX Script Compiler in order to participate in the Puzzler. To help prepare you for the Puzzler, I’d like to you to develop a very basic ‘Hello World’ style program.

This is interesting since they decided to write their own command line compiler instead of just sticking with Ant Tasks. Looks just like another pre-processor for Java code.

Still don’t understand how the map files that relgen uses is related to CVS. I was looking at the CVS tags and I can see the ones for major release out there, ie R_3_3_1_1 and ones based on dates. But the map files are referring to other version tags such as the date tags. I looks like its a pick and hope type of layout for the map file when choosing versions of dependent source to pull from CVS.

Relgen has a buildAll.xml file that I am going to try to run. Any takers on wether its going to work?

I started trying to build Eclipse today. I knew it would be a big project. As I was suspecting, finding where to start is hard. I know I need to check out the source but how. Looks like you need to use .map files that then reference the appropriate CVS tag and repositories.

I am trying to do this from the command line so I can automate the process. The goal I am trying to get to is a step by step process of building Eclipse not a bunch of pasted together processes from each project. Looks like there is no one way of pulling all the pieces together.

Stay tuned to see if I actuallt get the source checked out.

It appears that the libapr is using the WSARecv, WSASend etc functions instead of the recv and send functions. Also, it setup the connection a little differently than I was. But it was very easy to use. I need to test on Unix but I don’t suspect too many problems.

Check it out at: http://apr.apache.org/

Here’s the step to handle reading chunked data.

1. Read the http headers from the socket looking for the Transfer-Encoding: chunked header
2. After the headers, there will be a new line chr(13) and then a line with only a single number on it. This number is the length of the next chunk of data in hex (use strtol()) to change the hex to decimal)
3. Read from the socket the number of bytes in step #2
4. Next is a possible new line chr(13) or not (depending upon the server) and then the hex number on the next line.
5. You then repeat step #3 until the hex number is 0

Make sure you test against several different servers/clients since the format (extra newlines or not) are not consistent between the servers.

Also, look under the covers to see whats happening. For example, the WebSphere deploy script is really a shell/batch script that wrappers a call to a Java class. Inside the script you can get a better idea of what it expects as inputs and outputs.

The compilers will sometimes use a response file in which all of the inputs and parameters are put into and then the response file is passed to the compiler. Compilers that use response files are great since they avoid the problem of limited command line character limits.

I asked the customer to reinstall, and once again, everything would be working fine… until the machine was restarted. This went on for a few days, until we figured out that the Mojo Uninstaller that was deleting the files. However, it was the Mojo Uninstaller from a previous install that was deleting the files. It was deleting files from beyond the grave! (or the Recycle Bin)

The Mojo Uninstaller is based on a third-party software that stores a list of files that need to be deleted in an obscure location on the file system. Normally, this list is cleared once all files are deleted. However, in the customer’s case, not all files were deleted: the Knowledge Base server was still running during the uninstall, so any .jar files used by the KB Server could not be deleted. Therefore, the “files to delete” list still had these jar files listed. Once these files were freed up (i.e. on restart of the machine) the uninstaller jumped in and deleted the files. Since this list of files is stored in an obscure, relatively inaccessible place, we could not tell the uninstaller to quit uninstalling these files.

So, DO NOT use uninstall.exe when uninstalling Mojo on Windows. Unless of course you like headaches.

“So, Mr. Support Guy, how do I uninstall Mojo?” you ask?

Glad you asked. It’s a simple process:

1. Stop the Knowledge Base server - this will free up the KBServer jar files for uninstalling
2. delete all the files from the directory where you installed Mojo.
3. Delete the omenvironment.properties file.

If you have already uninstalled Mojo using uninstall.exe, and files are mysteriously disappearing from your Mojo install, then we recommend you do the following:

1. Uninstall mojo using the above process.
2. Reinstall Mojo in a different directory, so the uninstaller cannot find the .jar files it wants to delete.

To simplifiy operation in Mojo and Meister, we have introduced the omenvironment.properties file to help manage your Mojo/Meister environment.  This file contains OpenMake specific environment configuration and is automatically set up for you upon install of an OpenMake Software product.  For Windows users, this file is found in two locations: the “openmake” directory under your user’s “Application Data” folder (C:\Documents and Settings\USERNAME\Application Data\openmake) and the client\bin directory.  For those of you on UNIX or GNU/Linux, the file is found under the “.openmake” directory under your user’s home directory, as well as the client/bin directory.  The purpose of having multiple files is so that different users could have different environment settings, by using the file in their user home directories, or the environment can be managed globally with the file in the client\bin directory.  If the file exists in the user’s home directory, then that file will take precedence.  Otherwise, the file from client\bin will be used.

The main function of this file is to insure that when the Mojo or Meister interfaces are used to kick off a Workflow, these core variables required by Mojo/Meister are guaranteed to be part of the environment.  Also, by setting OPENMAKE_SERVER, it insures that you can connect to the KB server when you start the Mojo/Meister workbench.

There is another place where the omenvironment.properties file comes into play and sometimes causes problems: upgrading or reinstalling Mojo or Meister.  I will take on that topic in my next post.

Having worked in the field for close to 15 years, I feel I have earned the right to comment on what these terms mean:

Build Management - The ability to manage the creation of binaries including the enforcement of source file locations, third party library usage and compile flags.  It also must include the ability to track all artifacts used to create the binary, not just show what was checked into a SCM tool. 

Build Automation - The ability to run automatic builds, without the need to baby sit ad hoc scripts.  Build Automation is not the process of scheduling scripts or executing scripts from some sort of workflow tool.  This is just “script scheduling”  - not build automation.  There is nothing automatic about writing build scripts, or babysitting build scripts.

Dependency Scanning - This is the process of watching the compiler and scanning source code for nested references to source code and libraries.  It is not running a “check-out” report against a SCM tool to determine what was checked out to the local build directory prior to a build, referred to as a Bill of Material (BOM) report.  A BOM report cannot tell you what it does not know, and many artifacts there are used to create your binary come from places other than the local build directory.  Dependency Analysis is just that, a thorough review of what occurred in the build, even if an SCM tool is not used.  So if your build relies on an SCM tool for “dependency scanning”, it is not doing dependency analysis.

This list can go on for some length, but I will stop here.  The point is that the Build Management market can be as confusing as  builds themselves.  This is proof that we,  as an IT community,  have not spent enough time discussing, improving and standardizing this critical component of application development.

Well guess what - developers don’t really work that way anymore, but they continue to rely on this waterfall approach to builds.  What developers do is work within IDEs where their builds are nicely managed for them.  They then try to setup a continuous integration build, or nightly build process, and find themselves falling back into the waterfall process model and calling on a build expert to create an Ant, Maven or Make script.  If you are trying to improve your process, automate the creation of your build script, in the same way that you use IDEs to perform automatic code creation.  You cannot do agile development without streamlining the build step.  Developers should be able to build inside or outside the IDE without the overhead of a manual scripted process designed for a waterfall approach to application development.  I’m not saying scripting is bad, just that done ad hoc, it is highly redundant and forces agile development into a waterfall phase of build management.  Just some thoughts to chew on.

All license requests now need the following information to generate a proper license.kb:

- hostid of the KB Server (see installation guide for how to obtain this for the different OS’s. On windows it’s vol c: and most unix flavors, a derivative of the uname command)

- host names of any Remote Build Servers that are licensed, if any.

- number of named licensed users.

- contact information (name, e-mail and phone number is sufficient)

- whether or not you have a permanent license or are just requesting a demo version.

For more information about licensing, you may also always contact our Openmake sales team.

So to get your Windows exes signed you need a certificate from Thawte, Verisign or a like. We have a certificate that we use for signing our jar files that run via a Java Applet. (Only way to make a Java Applet talk to multiple machines and the file system).

The certificate is stored in the key3.db and cert7.db Netscape data files. The trick was to get the certificate from there into something Microsoft would recognize, since you need to use the Microsoft signing tools instead of the Netscape ones to sign a Windows exe.

Here are the steps I did to get me over to a Microsoft certificate store

1. Download the Netscape tools from:
http://www.mozilla.org/projects/security/pki/nss/tools/index.html

An awesome set of tools and documentation for Cryto stuff!!!

2. Since my cert was stuck in a cert7.db instead of a cert8.db I had to go back and get the version of the tools to handle that.
I used the nss-3.6.1.zip and nspr-4.2.2.zip pre-built packages for Windows.

3. I used pk12util to pull the certificate out of the cert7.db

It prompts you for the password to the DB and then another password for the .p12 file

4. At this point I had the certificate in the PKCS12 format in the mycert.p12. Next was to add it to the Microsoft certificate store.

Just run certmgr.exe and a GUI will walk through getting the certificate imported. I just took the defaults. Used my password that
I saved the .p12 file with and Marked the private key as exportable. I left enable strong private key encrytion off. Took the default
of Automatically picking the store. And your done.

5. Now that its in the Microsoft store you can sign your exes. You need to use the signcode.exe or signtool.exe (depending on the version of the Microsoft SDK) to do the signing.

Note: Use a timestamp server /t option

6. To check that it was signed correctly, run signtool.exe verify or chktrust.exe

Thats it!!

It appears that the first fgets() would get all of data except the EOF. The next time around fgets() should return NULL since EOF would be read. But what happens is that the program uses 99% CPU and gets stuck there. I didn’t go into the debugger to see exactly what was going on but the fix was to allocate the buffer bigger than the file size. And use that larger size on the fgets().

Fixed Code

The Virtual Build System that they setup was to have many VMWare images running and have each image run a subset of the build. I theory it looks like it would work well. Each VMWare image dedicated to running a slice of the whole build, so the compiles should run faster. But this is not that case, VMWare actually is getting in the way. No, matter how well VMWare works there are 2 things getting in the way. First, the I/O (network and disk). Secondly, the number of machine instructions executed is high. When the compiler runs, VMWare is running the compiler instructions and passing the instructions on to the Real OS. For VMWare to do this the processor must execute the instructions for VMWare itself as well as the compiler instructions. Therefor, no matter what you do the build is going to be slower.

The Virtualization does buy you a level of abstraction to move a build from one machine to another. But this begs the question, why is the build machine dependent? Shouldn’t a build be able to be recreated on any machine at any time? Openmake was designed with that requirement in place. To repeat a build on another machine you need a OS to run on. Does it need to be the same OS?, no. Running a build on Windows 2000 and Windows XP will produce the same resullts if the compiler version and source code are the same. The compiler can be checked in with the source and checked out with the source if you choose to do so to get a completely traceable build.

Most shops have a standard that they install for the compiler so checking it in and out is usually not need. I recommend just putting the CD on a shelf or off to Off-site storage for disaster recover and archival purposes.

I will get to the first bottle neck and solutions in a future post.

BTW, the Build Meister did confirm that Virtual Build System was slower in the testing that they have been doing. I just reconfirmed their findings.

OPENMAKE_SERVER will normally be set automatically whenever you install an OpenMake product. The form of the variable is as follows:

http://<KB SERVER HOSTNAME>:<KB SERVER PORT>/openmake

Where people often run into problems is the hostname part of the variable. The problem is there is sometimes difficulty in resolving the hostname to find the machine, in particular when your machine is part of a domain that uses a name server. The way to work around this is to have OPENMAKE_SERVER use the IP address of the machine.

In the past few weeks, I have encountered about three different support cases where this fixed whatever error was reported (and they were completely different errors that were reported). Since the errors that come from this problem are often misleading, the best policy is to set OPENMAKE_SERVER to use the IP address instead of the hostname whenever possible.

Overtime the underlying development languages and environments have moved towards the user and minimized the need to know the details of the underlying machine. Object oriented techniques and IDE’s are examples of this, and most developers today work in such a space.

This virtualization has not taken place in the build area and it remains a magic art at many unenlightened  companies, outside the bounds of control. Many project plans ignore or even forget the build, config and deploy steps in their program time an resource estimates, causing last minute panics and over-time at the end of many projects! But why fix it - we have all lived with this for years.

It matters today for three reasons:

1. New programming techniques such as Agile are increasing demand for non-redundant, repeatable, automatic builds.
2. Internal audit is oft uncovering the fact that build / release management is not repeatable, traceable nor is there separation of duties.
3. One key test of the ability of an organization to manage its home grown software is how quickly it can respond to production issues. These can often be more costly than missing a delivery commitment in the first place. Every configuration in production needs to directly point not only to the source code that produced it, or the environment in which it resides but also the libraries (application, opensource etc) that it utilized. Only the actual build step really knows what is in the binaries.

These three problems can be addressed by virtualizing scripts up to a template / build method that is abstracted to a degree that minimizes the environment and hardware and for that matter the compilers. By using build methods the management software (build management)  inherently achieves the benefits of build reuse, auditability and dependency discovery.

Some vendors claim that these benefits can be achieved with a process automation tool - most notably IBM / Build Forge and Electric Cloud. This is a step in the right direction but amounts to trying to herd cats, for if you do not virtualize the build scripts you are attempting to manage chaos with a process - it may look good for a while but will surely break.

1. Windows2000, Windows XP, or Windows 2003 server.

2. Active State distibution of Perl installed, with “bin” folder of the installation included in the PATH environment variable; 5.8.8 is the current preferred Perl version, though 5.6.1 is supported.

3. Any Java Runtime Environment version from 1.4.2_05 up to the latest 1.5.0 revision.

4. User must have privileges to edit the registry.

Once these conditions are met, the installation is merely a matter of unzipping the supplied zip file into its own directory and running mojo.exe.

Also, please note that these instructions are for new users who have not installed other OpenMake products. If you are an existing OpenMake Software customer and have problems with installing Mojo, please contact your OpenMake support representative.

Now with Live Writer front ended to WordPress I am set to go.

While most people liked the new features of omsubmit, we still found that some customers wanted things to behave like they did in 6.3, before omsumbit came to be. One particular reason people asked for this is that when running builds through omsubmit, the logging is written to a local log file instead of appearing directly at the console.

Anyway, if you want to bypass omsubmit, all that is needed is to pass the -l flag to bldmake or om. This will cause the build to effectively run as it did in 6.3. Note that this means that non-dependent build steps will not execute in parallel and that the build activities will not be logged to the Build Manager.

As of a few weeks ago, Openmake version 6.3 can be used to build Visual Studio 2005 applications, though the full feature set of the 6.41.1 Add-in is not available. With the latest update, 6.3 versions will be able to have Openmake automatically generate the Visual Studio 2005 Target Definition (.tgt) files and run VS2005 builds. The main deficiency however, is that Openmake 6.3 cannot run Openmake builds from inside the VS2005 IDE, which is something that can be done with the 6.41.1 Add-in. For more information on running VS2005 builds with Openamke 6.3 or 6.41.1, please contact an Openmake support representative

From time to time, I will be involved in a support case that will stump me for hours, days, or even weeks. These are cases in which absolutely nothing seems to work. The most basic step of the most basic build will crash and burn, leaving me with no place to even begin looking for clues as to what went wrong.

When nothing seems to work, my first instinct is to look for the most complicated explanation - the issue must lie deep in the guts of Openmake, the dark hinterlands of Openmake code that I haven’t touched in my two years as support person. In reality, the most catastrophic problems often have the most basic explanations.

One of the basic elements of an Openmake installation is the Perl libraries in the command line client. These libraries power nearly every aspect of an build, from dependency analysis to logging. Openmake’s Perl libraries are automatically installed during Command Line Client installation. However, if Perl itself isn’t installed, these libraries are useless, and a build will fall apart at the seams… literally nothing will work.

Now, one of the first things I check for when nothing seems to work (instead of trying to pick apart the innards of Openmake), is a working installation of Perl.

Due to changes in how Internet Explorer handles JavaScript, the Openmake Web Client will not load in Internet Explorer 7 without a small fix.

In order to allow the Web Client to load in IE7, you need to comment out a line of code in the index.html.template file in your <KNOWLEDGE BASE SERVER>/tomcat/webapps/openmake.ear/openmake.war directory.

Open the index.html.template file in a text editor, and do a search for the following line: if (_ie == true)

Two lines beneath this conditional (between curly brackets) is a line that starts with: document.writeln(’…

Comment out this line by typing two forward slashes (”//”). Restart your Knowledge Base server, and you will be ready to use Openmake with Internet Explorer 7.