Nerds Central

All Hard Drives Go To Hell

2013-05-20T09:15:00.000Z

When I was a kid (13ish I guess) a magnetic tape failed and lost me a day's work - I smashed the tape in anger. Recently a magnetic hard drive did a similar thing to me so I shot it.

Hmm - I am thinking therapy. But in the meantime, here are the results:

On Youtube - you can see the whole thing.

Air Gun Pellet Ballistics From Just Audio

2013-05-18T22:56:00.001Z

HandN Copper Point (left) and Dynamic SN2 (right)

A Chronometer will tell you how fast a pellet is going very soon after it has left the muzzle of your air gun. That is very far from the whole story.

A pellet has to push through air. Air gun pellets at 10-20 or so grains do not have a lot of kinetic energy to help push the air aside. We might think that the main reason for the limited range of an air rifle is the velocity of the pellet and hence its time to target. The slower the pellet is, the longer it takes to get to the target and so the more it will drop between the muzzle and the target. But, this is not the only issue, the air resistance and the mass of the pellet both have an effect. Another aspect is the way the pellet flies: does it precess and what is its ballistic coefficient (which effect one another).

All that is a bit on the complex side - can we do some simple stuff?

This post is not really about going into super complex stuff. It is more about how we can investigate some properties of different pellets with nothing more than a gun, a target and a video recorder.

Here is a video showing what I am talking about:

To get the figures I used in this video I computed the final speed of the pellets from the estimated muzzle velocity and average velocity to target. One big guess was the rate of deceleration I took this to be a proportional to the square of the velocity. Another big guess was the muzzle velocity, which I worked out from the mass and the 11 ftlb rating of the gun. What I want to do next is repeat the idea using distances of 5,10,15,20,25 and 30 yards. This will let me use extrapolation to get the muzzle velocity and an accurate model of the rate at which the pellets slows down.

C++ Unit Test Framework Takes Shape

2013-05-14T12:33:00.002Z

Unit test frameworks can be a bit like hangers.
Massive and don't have very much inside.

My self injecting unit test approach for C++ is working out nicely.

I hate complex frameworks which seem to make life simple but actually just add layers of translation and hours of maintenance pain. So, for Task I want a very simple unit test framework. Having just spend 6 months working on legacy C++ with no unit tests (what are called unit tests are actually much higher level - working at the executable level), I want the confidence of each non trivial method having its own unit test.

I have already discussed the system here. Now I would like to show some real unit tests which are testing real functionality!

#import "../taskParser.h"
#import "taskAllTests.h"
namespace taskUnitTest
{
  class lexerStage1TestA : taskUnitTest
  {

  public:
    lexerStage1TestA()
    {
      addUnitTest(this);
    }

    virtual const taskUnitTestResult run()
    {
      task::task_parser parser;
      task::t_string toParse=task::make_atom(new std::string("a b  dog\r\nd\tfrog$cat 12"));
      task::t_vector(lexerFistPassToken) tokens = parser.parse(toParse);
      bool ok=tokens->size()==6;
      ok &= *((*tokens)[0]->getValue()) =="a";
      ok &= *((*tokens)[1]->getValue()) =="b";
      ok &= *((*tokens)[2]->getValue()) =="dog";
      ok &= *((*tokens)[3]->getValue()) =="d";
      ok &= *((*tokens)[4]->getValue()) =="frog$cat";
      ok &= *((*tokens)[5]->getValue()) =="12";
      if(!ok)
      {
        puts("FAIL:");
        for(size_t i=0;i<tokens->size();++i)
        {
          printf("\t%d '%s'\r\n",(int)i,(*((*tokens)[i]->getValue())).c_str());
        }
      }
      return taskUnitTestResult(toString(),ok,"Simple white space splitting and character agregation");
    }
    virtual const std::string toString()
    {
      return "Lexer stage 1 simple";
    }
  };

  class lexerStage1TestB : taskUnitTest
  {

  public:
    lexerStage1TestB()
    {
      addUnitTest(this);
    }

    virtual const taskUnitTestResult run()
    {
      task::task_parser parser;
      task::t_string toParse=task::make_atom(new std::string("a!b  dog\r\nd\tfrog&cat 1=2"));
      task::t_vector(lexerFistPassToken) tokens = parser.parse(toParse);
      bool ok=tokens->size()==5;
      ok &= *((*tokens)[0]->getValue()) =="a!b";
      ok &= *((*tokens)[1]->getValue()) =="dog";
      ok &= *((*tokens)[2]->getValue()) =="d";
      ok &= *((*tokens)[3]->getValue()) =="frog&cat";
      ok &= *((*tokens)[4]->getValue()) =="1=2";
      if(!ok)
      {
        puts("FAIL:");
        for(size_t i=0;i<tokens->size();++i)
        {
          printf("\t%d '%s'\r\n",(int)i,(*((*tokens)[i]->getValue())).c_str());
        }
      }
      return taskUnitTestResult(toString(),ok,"Simple white space splitting and character agregation for special tokens");
    }
    virtual const std::string toString()
    {
      return "Lexer stage 1 special";
    }
  };

  // Singletons
  static lexerStage1TestA testA;
  static lexerStage1TestB testB;

}

First thing I am expecting readers to point out is that I do not redirect all output to a logging facade around a login framework-framework management system. Yeh - you're right I don't and I'm not going to!

Having said that - I hope it is clear how easy writing a unit test is in this framework (ette):

Create a class which inherits taskUnitTest.
Implement run to perform the test.
A bit of output giving a clue as to why a test fails is never a bad idea.
Sticking the string FAIL: in there will not hurt as grep is a very handy tool.
Implement toString as this is part of the API.
Stick to using the Task memory managed objects t_string and t_vector help.
Implement the constructor which adds the test to the framework.
Create the singletons.

And remember:

SN2 (Tin) Pellets - Nerds Central Air Gun Video

2013-05-13T10:36:00.000Z

We had a whole bunch of fun shooting the Dynamic SN2 .22 tin pellet:

Rifles:
Gamo CFX Royal .22 underlever
Gamo Camo'Rocket .22 break barrel

Pellets:
Dynamic SN2 .22 :
http://www.guns.gb.com/contents/en-uk/d94_Prometheus_airgun_pellets.html#p653
HandA Copper Point .22:
http://www.amazon.co.uk/gp/product/B006SYPB8S
RWS Super Dome .22 - Also Mentioned (but not shot):
http://www.amazon.co.uk/gp/product/B007ECH9S8

Note that the SN2 pellets used here were washed in isopropyl alcohol, dried and the lubricated before shooting.

The music is Poison 1:
http://youtu.be/yJJ0xCklB7U

Be Safe!

C++ Example: Order Of Construction Of Global State

2013-05-06T06:56:00.001Z

Why an I coding when I could be chilling in
the sunshine?

In C++ we can mess with the global state to ensure some stuff is constructed before main is called - but we need to be very careful.

The C++ language and runtime make no guarantees about the order of construction of the global state. This is not a problem in C because no user code actually executes before main (the runtime does construct its self though). In C++ stuff can/does execute user code before main and this can be very useful indeed; however, that which runs before main has to be super careful about the state of construction of its environment.

I hit this issue just now and thought I should blog about it. I am working on a very simple unit test system for a C++ project. The idea is that unit tests self inject themselves into the unit test system by constructing singleton objects in the global state:

#import "../taskParser.h"
#import "taskAllTests.h"
namespace taskUnitTest
{
    class testTest : taskUnitTest
    {

    public:
        testTest()
        {
            addUnitTest(this);
        }

        const taskUnitTestResult run()
        {
            return taskUnitTestResult(toString(),true,"Dummy test which should pass!");
        }
        virtual const std::string toString()
        {
            return "Test test";
        }
    };
    static testTest test;
}

[Note to purists - these singletons are not enforced singletons - just singletons by convention].

In the above code the constructor of testTest instance test adds its self to the unit test system. This means the unit test system does not need a list of unit tests to run or some form of runtime test discovery. Unit tests ask to be tested rather than the test framework finding them and testing them. I really like this idea, but it is playing in that dangerous pre-main state.

The code below is the fixed version. The bug I hit was a very odd one indeed which I will discuss after this code.

#import "taskAllTests.h"
#import <iostream>
static std::vector<const taskUnitTest::taskUnitTest*>* tests = NULL;

int taskUnitTest::runUnitTests()
{
    int ret=0;
    cout << "About to run " << tests->size() << " unit tests." << endl;
    for(
            std::vector<const taskUnitTest::taskUnitTest*>::iterator it=tests->begin();
            it!=tests->end();
            ++it
    ){
        taskUnitTestResult result=(const_cast<taskUnitTest*>(*it))->run();
        cout << result.toString() << endl;
        if(!result.getSuccess())ret=1;
    }
    return ret;
}

void taskUnitTest::addUnitTest(const taskUnitTest* test)
{
    if(tests==NULL)
    {
        tests=new std::vector<const taskUnitTest::taskUnitTest*>();
    }
    tests->push_back(test);
}

So what happened?
The original had

static std::vector<const taskUnitTest::taskUnitTest*> tests ;

Rather than

static std::vector<const taskUnitTest::taskUnitTest*>* tests = NULL;

The code for constructing the vector was not required so the addUnitTest code was something like this:

void taskUnitTest::addUnitTest(const taskUnitTest* test)

{
    tests.push_back(test);
}

This looks much better doesn't it? However, when adding two unit tests, the vector reset its self between calls to addUnitTest. This meant that only one object was placed in the vector and only the last unit test to add its self was actually run!

The only explanation I could come up with was global state setup. Now, that is a bit of a worry because it could just be luck that my new code works. There is nothing to guarantee that vector will be set up correctly in time for this code to run. Really, I should hand implement a vector with no global state in it rather than hope the stl will be OK. But for now - on gcc - this seems to work fine.

If I have missed something obvious - don't be shy - let me know and call me a plonker!

And here is the output of the unit test code running two tests:

About to run 2 unit tests.
PASS: Lexer stage 1 (not yet implemented)
PASS: Test test (Dummy test which should pass!)

Eclipse - sometimes I wish it did actually look like this.

What is the third best selling laptop in the UK?

2013-05-05T08:56:00.000Z

The ARM Chromebook hits the number 3 spot.

The ARM powered Chromebook!

Yeh, whilst the word (me included) wring our hands over the success of iPhones and the slow death march of Windows, the humble Chromebook is going from strength to strength. Is it going to replace all other PCs? I doubt it, but it might become a very big player in the laptop world eventually; if this is not in the corporate definitely in the home computing world.

No matter how hard I try, I am not managing to believe the Chromebook will be a world dominant technology. However, I also see it as being very successful. You don't need to take over the world to be a success; maybe we in the IT world have become addicted to hyperbole?

I am writing this post on a Chromebook (and old Samsung). This morning I had a Mac Book Pro Retina 15inch and this machine next to my bed. I picked up the Chrombook and used it. "Why?" you may well ask. Here are a few reasons:

1) I will not grieve for a decade if I spill coffee on it.

2) The battery lasts for ever.

3) It is light weight.

4) It does exactly what I want.

5) It boots even faster than the Mac.

6) It is secure and I can leave it hanging around - the Mac has my accounts on it for goodness sake!

So - the Chromebook is, for me, the perfect second machine. My family share it. My daughter loves it because it just works and my younger son uses it when his big machine is on the fritz.

Chromebooks are a really good idea - but I don't think they will take over the world any time soon. I don't think that is the world we are moving into. We are moving into a multi-machine multi-purpose world of smart phones, tablets and Chromebooks.

As a final thought - in our house the Chromebook gets massively more use that the Sony tablet...

The Only Thing Which Saved Windows 7 Was Windows XP 64 - Now the Game Is Up

2013-05-04T18:07:00.000Z

From http://www.netmarketshare.com/
Market share from browser use April 2013

Seriously, Windows is dead-man-walking. Yes, it will hang around for decades, but the platform has well and truly hit the bumpers.

Let us look at some cold hard facts about capitalist consumerism, the thing which provides the wealth which Microsoft runs on.

1) Software has a near zero cost of manufacture. This makes all software a commodity.

2) We pay for software to amortize the cost of development and maybe to cover the cost of support.

3) We do not get support from Microsoft in the OS cost (usually) and so we are just paying for the cost of development.

4) If new development adds nothing we need - then there is no point in paying for it.

Thus, the only thing which drives Windows development is the creation of new stuff which we need. Just as I pointed out in my discussion on the slowing PC market - that is not happening any more. Actually, it has not happened since Windows XP other than one important thing.

If Windows XP 64 had been any good, Windows 7 would have been unnecessary.

There were a bunch of nice updates and kernel changes which went into 7 which made it worth while. If XP had a new schedular and better x64 support, then even Windows 7 would never have been required. Windows 8 brings nothing what so ever to the table. How could it? There is nothing to bring; Windows 7 does everything anyone wants to do with modern PC hardware. Touch is all very well - but people are not that bothered about it on a PC and Windows for tablet is POINTLESS - the market is dominated by much more mature players and Windows brings nothing unique - just Tiles-R-Us colours and a the comfort that it will crash and ask you to reboot for updates on a regular basis - just the way everyone loves on the PC.

Linux has the same problem, developers strive to provide "better" GUIs when actually the new ones are not as good as KDE3 this is why xfce is becoming ever more popular;
why fix that which is not broken?

This is scary stuff. A powerhouse driving the software industry forward has started to fail. Windows replacement does not just drive MS profits, it drives the whole industry forward. Companies spend hundreds (maybe thousands) of millions of dollars, pounds or what ever on porting their software to the new version of Windows. Windows 8 is a PITA for this, there a lot of breaking changes which are subtle but make the cost of porting to it high. So, maybe people will not bother? Maybe the Windows 8 experience for more specialist software will be a sparse experience for some time as many high value software systems remain unsupported, or poorly supported on Windows 8.

Bottom line - there is zero doubt that the corporate world is not moving over to Windows 8 any time soon. Maybe it never will?

The graph above is from Net Applications. For the sake of fairness below is the table of desktop OS market share figures from Wikipedia for the same period:

Source	Date	Microsoft Windows					Apple		Linux kernel based		Other^[a]
Source	Date	8	7	Vista	XP	Other versions	OS X	iOS	GNU/Linux	Android	Other^[a]
Net Market Share^[14]^[15]^[16]	Apr-13	3.29%	38.50%	4.09%	32.98%	0.15%	6.04%	8.21%	1.04%	3.62%	2.08%
StatCounter Global Stats^[17]	Mar-13	3.90%	52.61%	6.13%	23.38%	0.25%	7.29%	4.03%	1.00%	1.12%	0.34%
W3Counter^[18]	Apr-13	4.17%	43.72%	4.55%	21.28%	0.34%	8.21%	9.28%	2.28%	4.48%	1.07%
Wikimedia^[19]	Mar-13	2.6%	34.12%	4.06%	14.31%	0.56%	6.71%	25.19%	1.46%	6.19%	0.60%

As we can see, Windows 8 has not taking off. It is a flop. However, I am suggesting that it might not be Windows 8 that is the problem. The problem for Microsoft is that the market simply does not need, want or care about a new OS.

Love Of A Dog - Or Maybe C++?

2013-05-01T05:49:00.002Z

My son and Hector having a cuddle - what could
be better!

Maybe there is one thing greater than the love of a dog - my love for clean C++.

Nope - dogs win - but C++ can be such a beautiful language. It gets dirty really quickly, but if you have the time to work with it and you know what you want - it is a thing of beauty.

This evening I got chance to do a bit more work on the parser for Task - my super simple C++ based interpretor which uses nothing but closures to form the language. I am going for a hand written parser for speed and clarity because the whole idea is that the language is so simple a grammer based parser would be massive overkill.

So here is my evening's work. It is in header form for now though I expect I'll refactor out some of the guts to definitions at some later date. A brief description of some of the key points follows the code:

/*
 * task-parser.h
 *
 *  Created on: Apr 8, 2013
 *      Author: alexanderturner
 */

#ifndef TASK_PARSER_H_
#define TASK_PARSER_H_
#include "task-memory.h"
namespace task
{
    class token
    {
        const size_t line;
        const size_t character;
        const t_string file;
        const t_string value;
    public:
        token(size_t lineIn,size_t characterIn,const t_string fileIn,const t_string valueIn) :
        line(lineIn),
        character(characterIn),
        file(fileIn),
        value(valueIn){};

        size_t getLine()          {return line;      }
        size_t getCharacter()     {return character; }
        const t_string getFile()  {return file;      }
        const t_string getValue() {return value;     }
        string toString()
        {
            //TODO not exception safe
            char* number=new char[32];
            string ret="File: '";
            ret+=*file;
            ret+="' Line: ";
            sprintf(number,"%zd",line);
            ret+=number;
            ret+=" Column: ";
            sprintf(number,"%zd",character);
            ret+=number;
            ret+=" Value: '";
            ret+=*value;
            ret+="'";
            delete number;
            return ret;
        }
    };

    class task_parser
    {
        const bool isWhiteSpace(const char what)
        {
            switch(what)
            {
            case '\t':
            case ' ':
            case '\r':
            case '\n':
                return true;
            default:
                return false;
            }
        }

        const bool isNewLine(const char what)
        {
            switch(what)
            {
            case '\n':
                return true;
            default:
                return false;
            }
        }

    public:
        t_vector(token) parse(t_string src)
        {
            return parse(src,make_atom(new string("unknown")));
        }

        t_vector(token) parse(t_string src,t_string file)
        {
            t_vector(token) result=make_vector<token>();
            size_t line=0;
            size_t character=0;
            size_t index=0;
            // Remove all leading white space
            for(;index<src->size();++index)
            {
                const char c=(*src)[index];
                if(isWhiteSpace(c))
                {
                    if(isNewLine(c))
                    {
                        ++line;
                        character=0;
                    }else{
                        ++character;
                    }
                }else{
                    break;
                }
            }
            bool wasWhiteSpace=false;
            t_string thisToken=make_atom(new string(""));
            for(;index<src->size();++index)
            {
                const char c=(*src)[index];
                if(isWhiteSpace(c))
                {
                    if(!wasWhiteSpace)
                    {
                        result->push_back(make_atom(new token(line,character,file,thisToken)));
                        thisToken=make_atom(new string(""));
                    }
                    if(isNewLine(c))
                    {
                        ++line;
                        character=0;
                    }else{
                        ++character;
                    }
                    wasWhiteSpace=true;
                }else{
                    wasWhiteSpace=false;
                    (*thisToken)+=c;
                    ++character;
                    wasWhiteSpace=false;
                }
            }
            return result;
        }
};

}
#endif /* TASK_PARSER_H_ */

...

There are now two defines in the task_memory header:

#define t_string         task_atom<string>::type
#define t_vector(type__) task::task_vector<type__>::type

Here are some thoughts on the design:

Lots of const.
Use classes with accessors not structs with direct field access.
I choose what is white space or not rather than using a standard function.
Open, clear form using next line brace layout.
Heavy use of shared_ptr which is largely hidden using templates, helper functions and a couple of #defines.
I don't use vector iterators that much because index access of vectors is very fast and looks cleaner without all that type flab.
size_t - no int, int32_t and all that dangerous stuff - if it is a size - use size_t!
Not given enough thought to exception safety yet - should at the least be using auto_ptr not raw pointers were new and delete where shared_ptr is not being used.
Build functionality up in stages.

On that last point, let me explain further. The parser here only breaks the source into high level tokens - it is actually a lexer. However, it is not even the full lexer in that the tokens it produces will be lexically analysed further and made into more differentiated smaller tokens before there is a full semantic analysis.

The coding approach:

Lots of small methods.
Each method does one simple thing which is easy to test.
Unit tests at the method level.
Build functionality up in iterations.

This approach makes managing multiple projects at once much easier and builds testability into the code base. In my case, I have come across this more in agile Java development, but I am seeing huge benefits for it in agile C++ development as well. The guts of a large method which does many different things are the natural hiding place of a nasty bug!

Next Step:
Go write some unit tests. This is an exemplar project so I intend to do it right!

Easy Encryption: Passing Sensitive Files Via Dropbox Or Google Drive

2013-04-29T12:57:00.002Z

Hmm - nope, cannot think of a good excuse for this dogy picture
but they are cute!

Google Drive and its competitors are super good for sharing files, but security is an issue. Free, easy crypto to the rescue.

Cryptography has legal issues - I am not going into those - you must make sure you do not break any laws.

On Linux and OSX I recommend using the command line tool openssl. After I have discussed this, I will describe the brilliant 7-Zip tool which is ideal for Windows.

To Package:
tar -cvzf myFile.tgz myDirectory

To Unpackage
tar -xvzf myFile.tgz

To Encrypt:
openssl aes-128-cbc -e -pass pass:S0mEPA55W0rd -in myFile.xyz -out myFile.xyz-enc

To Decrypt:
openssl aes-128-cbc -d -pass pass:S0mEPA55W0rd -in myFile.xyz-enc -out myFile.xyz

The chances are that you knew about making an archive with tar, but I threw it in anyway! However, a few minutes looking at the openssl command line might help.

The first argument is the command openssl is going to perform. In my example I am telling it to do some encryption or decryption using the aes cryptography algorithm with a key size of 128 bits using cypher-block-chaining. This means that we are using a standard and well respected form of encryption with a reasonable width key and we are linking one encrypted block to the next so that parallel attack is inhibited.

From then on it is pretty obvious what the arguments do. Using -in and -out as different files is wise. If you miss the pass:xxxxx command after the -pass argument, openssl asks to have the password typed-in which is more secure because you do not end up with the password in the command history buffer.

Windows And 7Zip:

In Windows Explorer, right click and select 7-Zip>Add to archive.

If you are not sure if the other user(s) has 7-Zip you can create a self extracting archive.
Here I am encrypting at 256 bits AES - just 'cause I can.

A note on key length. 128 is near impossible to crack for any normal organisation. If you expect GCHQ or the NSA to not crack it - you are in over your head. No level of encryption is any good unless the whole system - end to end - is solid. If your system is solid enough to justify 256 bit AES then you already know everything here!

Sharing between *nix and Windows:

Not sure! Any good suggestions? I have used command line 7-Zip on Linux before but that was a very long time ago.

Ultra Fast Locking - Rebirth of the spin lock

2013-04-27T19:37:00.001Z

Little, cute and agile - a balanced approach to locking.
Copyright Dr Alexander J Turner all rights reserved.

Locks (monitors or what ever) are becoming an ever more important aspect of software engineering. Yet, we tend to know so little about them.

With the invention of critical sections and/or the futex, a lot of programmers have stopped considering locks as a key part of development. This is a shame because futex approaches only offer a benefit for uncontended locking. Uncontended locking being, well, no locking at all. In other words, a futex is a good solution to needing a lock but not actually using it most of the time.

Let use thing about an example of what I am getting at.

1) Get a lock.

2) Read from a map.

3) Release the lock.

Consider that two threads try to perform step 1 at once. Thread A gets the lock and the only effect of this is an atomic value set. Thread B reads the lock value and so we get a cache flush if A and B are on different cores. So far nothing too expensive has happened. The problem now is 'what should thread B do?' The futex/critical-section answer is go to sleep. This means relinquishing its quanta to the kernel which is a context switch at the very least.

A context switch on modern operating systems is generally considered to take thousands of cycles. There is no way reading from and std::map can take thousands of cycles unless the map is HUGE. If the map is HUGE then an unordered map should be used. Then that will not take thousands of cycles unless the keys are HUGE. Basically, a map that takes thousands of cycles to read is simply broken.

This means that the moment our lock if contended, our solution is wasting masses (and masses) of clock cycles context switching.

But it is worse!

Remember that Thread B lost its quantum. Sometimes we don't mind. If our application is not time sensitive this does not matter. However, what if it is real time or time sensitive? An example of a highly time sensitive program is a profiler. If the profiler code looses its quantum early then the resulting timing numbers for the profiled code will be useless. In the real world, loosing quanta causes a lot of trouble and should be avoided. A thread should go to sleep when its quanta run out, not but deliberately relinquishing them. Early quantum relinquishing also puts strain on the kernel schedular which is an other scaling issue with multi-core machines.

Lock Free And Light Locking:

Lock free algorithms can really help. By lock free, I mean that the algorithm does not use any locks which can loose a quantum. Building a lock free map is quite straight forward as long as it never needs to be re-balanced. The solution is, on mutation, to simply create a new map which shares all the same nodes other than those from the root to the point of mutation. Then, the root access to the map can be updated atomically so that all readers only see the original or the new map and never a mixture of the two. This means that the only point of contention is the atomic update of the root node.

Is this really lock free? Yes it is in terms of the algorithm never resulting in a context switch or loss of a the current schedular quantum. However, in another sense it is not lock free. Let us consider that two threads are mutating the map at the same time. In this scenario we will consider three threads: A, B and C where A is reading only but B and C are mutating the map.

Broken:

A is reading the map. A gets the root node and follows it. No problem.
Whilst A is reading the map B mutates the map. No problem because B is creating a new root node and not interfering with A in any way.
At the same time C is mutating the map. C also creates a new root node which is a different root node to B's new node. C is unaware of B.
We can now forget A.
B atomically changes the root node pointer to its new root node.
C atomically changes the root node pointer to its new root node - and in so doing erases B's changes.

Fixed:

A is reading the map. A gets the root node and follows it. No problem.
Whilst A is reading the map B mutates the map. No problem because B is creating a new root node and not interfering with A in any way.
At the same time C is mutating the map. C also creates a new root node which is a different root node to B's new node. Again, C is unaware of B at this stage.
We can now forget A.
B uses compare and swap to change the root node pointer to its new root node. B compares the root node pointer with its value when B started creating a new root node and as they match its new pointer is swapped in (see compare and swap - cas).
C uses compare and swap to change the root node but the cas operation fails because B changed the root node pointer.
C starts again.
C keeps doing this till it manages to get a matching cas.

This is actually a write spin lock:

C is spinning, recreating new root nodes and the chain of nodes to the mutated node until its cas operation is successful. Could we create a simple read/write spin lock to achieve the same thing?

Why are spin locks bad:

It is commonly thought that spin locks are bad. Once upon a time they were. They don't really make all that much sense on single core machines. On multi-core (or multi-socket) machines spin locks make a lot of sense for locking very short operations (like reading and writing from/to maps). Further, we can see that 'lock free' algorithms can easily start to behave like spin locks so why not give up the prejudice and just use a spin lock?

Read/Write locks:

A simple spin lock will strictly serialise access to a piece of code. Alternatively, a read/write lock has the following behaviour.

If unlocked, a reader or a writer can acquire the lock.
If read locked, another reader can share the lock but a writer is locked out.
If write locked, everything else is locked out.

This is perfect for a map because we do not stop multiple threads accessing the map for read. I am not suggesting it is a better solution than a well crafted lock free algorithm, but is sure is a good quick and dirty alternative.

Note: because we are using cas, a well constructed caching architecture will not require a cache flush as multiple readers acquire simultaneous locks.

How:
We have a memory location which can take 3 values - 0,1 and 2.

0 means unlocked
1 means read locked
2 means write locked.

Lock semantics:

To get a read lock, cas is performed with the set value of 1 and a compare value of 0. If the return value is 1 or 0 then the lock is acquired, otherwise try again. In other words, if it is read locked, do nothing; if it is unlocked, read lock it; if it is write locked, spin.
To get a write lock, cas is performed with a compare value of 0. If 0 is returned then the lock is returned, otherwise try again. In other words, if it is locked at all, spin.
To release either lock, set to 0.

Problems:
This is only a reasonable approach if the time spinning is very short. This is a technique for protecting short operations. It also makes no attempt at all to be fair. Some form of queueing system might help with this - but that is beyond the scope of this post.

STOP THE JAVA SECURITY NONSENSE!

2013-04-20T09:38:00.001Z

If you want to look silly - stick to pulling faces
and don't talk nonsense about Java.

What a load of tosh. It is major scary to find out how miss-understood software is even by self proclaimed professionals.

Some terminology:
Java, is a programming language which is compiled to byte code.
Byte code, a low level programming language used by the JVM.
JVM the software which runs programs which are compiled to or written in byte code.
Browser a piece of software for accessing stuff of the web using standard protocols (HTTP and HTML).
HTTP a web protocol for transmitting data.
HTML a web protocol for defining how to display stuff in browsers.
Browser plugin, a piece of software which offers extended features to a browser beyond what can be done by HTML.
HTML5 a very powerful, recent and still evolving version of HTML.

You cannot install or uninstall Java from a browser

Sadly JVM plugins are called Java plugins. But they are not. A browser plugin which is often referred to as 'Java' is actually the JVM. Code written in any pure byte-code generating language (Scala for example), or just hand written in byte-code by a human, can run in a JVM plugin.

Nearly all uses for the JVM are stand alone - not in plugins

Interestingly, you do not need to install the JVM. It is just a program which can be downloaded and run (technically a shared library and launcher application). Its use in plugins is starting to be a legacy. HTML5 can replace a lot of what JVM plugins do. I can see a world where JVM browser plugins are a pure thing of the past. You do need to install a JVM plugin, but not stand alone JVM. Programs you have running might have the JVM as part of them and you would never know.

Real world example

I am planning a training course in Java/JVM performance analysis. As part of this I will be providing people with remote desktop access to a single machine. I could have chosen JVM as the way to deliver this to browsers but HTML5 proved to be a better choice!

Many very powerful programs run on the JVM. These tend to be high end systems. Stuff like the Eclipse developer environment, databases and cloud data mining systems. All these use the JVM and most of the software for them is written in Java. Next time you book an airline ticket or go to your bank's website you will almost certainly be interfacing with huge Java/JVM based systems. You will also probably be interfacing with COBOL/Mainframe based systems which are talking to the Java/JVM systems - but that is another story.

If the JVM plugin for browsers dies out, it will have almost no impact on the use of Java. Oracle are not interested in the JVM plugin to any great extent. All the money in Java/JVM is on high end stand alone software not trivial plugins.

According to the TIOBE programming language survey
April 2013, Java is the world's second most popular programming
languageand is in place to retake the top
spot any time now.

Java/JVM applications are no more or less secure than any other application

Once you download an application and run it you have given up any chance of security. You have to be sure the program is from a reputable supplier and will not do bad things. In Windows, most programs are written in C++. This language makes even less attempt to make security assertions than Java; neither make any serious attempt to maintain security when an application is running on your computer. If someone states Java is insecure, they are talking very dangerous nonsense because they are implying something else is secure.

You cannot, must not, ever, under any circumstances think that a downloaded application can be secure because of the technology it is written in. This thinking can only be applied to browser plugins and they are a thing of the past.

Flash is another plugin that you might well be using for watching Youtube videos. One day that will die away too. The 'Java' plugin stuff is really a debate about plugins in general, they are always going to represent a security hole.

Why JVM plugins are insecure

There is a specific problem with JVM plugins. The JVM offers too much power to the plugin and then tries to restrict the ability of software running in that plugin to use the power. This is in contrast to Flash which is just limited end-of-story. The flash model is a much better model for plugins.

What should you do?
Don't, get all silly and remove every trace of Java and the JVM from your computer. That will make no difference to anything other than stop some stuff working.
Do, feel free to disable or remove JVM plugins from your browsers.
Do, if you have to have a JVM plugin for a work email system (for example) or some other application like that, install two browsers (say Chrome and Firefox) and have Java in one and not in the other. Only go to safe sites (like your work site) with the browser which has the JVM in it.
Don't, listen to idiots who have no idea what they are talking about.

The Real Reason PC Sales Are Declining

2013-04-17T12:36:00.000Z

Shining some light on the declining PC market.
Copyright Dr Alexander J Turner all rights reserved.

Doh! It is so obvious yet so hard to see if you're stuck in last decade's mind set.

PCs traditionally sold due to:

1) People buying for the first time.

2) New hardware being the only way to make things go faster.

3) New features which significantly enhance the range of things a computer could do.

All these have hit the buffers:

1) Everyone has a PC who can afford one and/or has an interest in one.

2) Browsers are getting faster and faster due to software enhancements and processing mostly all being done in the cloud.

* A 3 year old netbook can do more now than when it was new.

* 90% of most user's experience is either their computer being a thin client to a cloud, an intra-net web-application or using MS Office to write a document which a turn-of-the-century PC could easily manage.

3) Touch screens are the big new feature but they do not add to what the computer can do. Look back a few years and the killer application for a PC was being about being able to play surround sound, or video at 1080p or be high resolution colour etc. There is nothing you can actually do with a touch screen that you cannot do with a non-touch PC - touch screens are user interface not functionality.

Conclusion:

The technology of a modern PC is good enough to service the limited physical realities of its users. A modern PC can render video at a higher resolution than we can see and sound at a higher quality than we can hear. We can type faster on a keyboard than a touch screen and touch-pads are so good we can navigate in any way we want with the machine. A good PC from 2 years ago has no significant disadvantage over a good PC now in the functionality it provides so why buy a replacement?

We drive cars with the same control layout they had 80 years ago; sooner or later a design is simply good enough. That happened a few years ago with the PC.

The Future:

There is no slowing of the progress of IT. We are at most only half way through the IT revolution. Its biggest effects are yet to come. Indeed, I predict that if humanity is going to survive the next 200 years in anything like a civilised way, it will be because of the effects of the IT revolution. However, the days of the ever growing PC market are well and truly over.

Making Breasts Look Bigger Using The GIMP

2013-04-16T10:29:00.002Z

Has this image been edited?

The GIMP is free and very powerful. Do not trust any image use see, even on flicker or facebook. Anyone can make themselves look bigger, better, thinner or whatever. In the 21st century, photographs always lie!

The images on this page are derived from this work which is provided under a creative commons license for commercial and derivative work with attribution. Thanks 'prayitno'.

Is the image top-right edited? When I asked this question to my 16 year old daughter, after some moments thought, she concluded that it was not. I then asked my 21 year old son; he said that it probably was because most images are, and I was asking the question (he is studying philosophy - can you tell?). However, he could not actually tell by looking at it that it was edited.

I have no doubt that a lot of people who are experienced with photo editing will see what I have done instantly - but that is not the point!

Below is a before and after comparison of the image:

Left before editing, right after enlarging the breasts and narrowing the waste.

Breasts here one second - gone
the next.

To really drive home the magnitude of the editing done - which two savvy young people did not spot - here is an animated gif:

In a moment I will give a detailed description of what I did to perform this edit. However, let me finish this section by making a few points:

I am a computer programmer not a professional image editor.
This edit took me 10 minutes one evening sat on a sofa.
The GIMP is free and on every major computer platform.
Even with my level of skill, had I taken more time, the effect would be even harder to spot.

Pick one side or the other to start with, it is easier to
get one right and make the other match than try to do
two at once.

Select Filters/Distorts/IWarp the default setting is
as shown above - this is not what you want.

Set to grow and make the radius bigger. The radius
will depend on your image size. But start with
something about 1/3 the radius of the area you
want to enlarge. Then start clicking to warp starting at where you
think the nipple should be and work out. This mimics the way
breasts actually grow and so will give you the best chance of
imitating nature.

The smaller radius is good but can start to distort the image
a bit too much, some work with a larger radius will help this.
Make sure not to over do clicking in any one place.

At this point a note on anatomy. In a bra, a breast is usually compressed and forced up and back. This makes larger breasts force the side of the bra out under the arm and in towards the cleavage. The tissue above the breast is also forced up. You need to be careful to replicate these effects to make the image look natural. The sides are especially important as humans are very sensitive to silhouette.

Now we need to make the other side match.

This is tricky but very important. You need to imagine the original
in 3D so that the offset angles match. In this case the subject is
facing very slightly to the viewer's right. This means that as the left
breast (on the right) is enlarged it will bulge out more to the viewer's
right than the left one did to the viewers left.

By enlarging the left breast a compression of the left arm can happen.
Some small radius grow distortion in the middle of the arm
helps compensate. If you are lucky, it might even look
like the breast is pushing against the arm making the image look
more real and the breast look firm.

After some tweaking, we now have a fairly balanced
pair of breasts, but the image looks wrong because we do not
expect to see such large rounded breasts without more waist.
This is not a comment on the human form but more the
expectation that modern media place upon us.

Here I have used a shrink with radius 55 and moved around a
bit shrinking the sides of the waste. A couple of clicks right on
the belly button pulls the abdomen in to match the narrower
waste. The hips then look wider and more rounded.

Never trust an image!

Neat Code == Bad Code

2013-04-12T11:06:00.000Z

Some people believe perfect code layout is utterly essential to good coding practice. I completely disagree.

Good code layout is a very good idea. Code should be easy to read and indentation should be used to clearly identify functional and/or semantic blocks. However, placing more emphasis on neatness than these guidelines suggest is like drinking 20 espressos - it makes matters worse not better.

Why this rave?

I saw the following piece of barf worthy code in the source for the JDK - yes really:

public final void writeInt(int v) throws IOException {

out.write((v >>> 24) & 0xFF);

out.write((v >>> 16) & 0xFF);

out.write((v >>> 8) & 0xFF);

out.write((v >>> 0) & 0xFF);

incCount(4);

}

1) Unsigned shift 24 bits to the right on a 32 bit word enforces the left 24 bits to be zero.
2) Unsigned shift 0 bits does nothing!

This code looks great - but it makes no sense at all. It might be the case that the compiler will have the smarts to optimise away this silliness but that still leaves the silliness there to confuse future programmers.

We all accidentally write not-so-good code; that is part of being human. My point here is that this code has been lovingly sculptured to be brain dead.

Never mistake neat code for good code!

PS - this is a terrible implementation anyhow - I have no idea how anyone could think this was a good approach! It should write to a 4 element byte array and then call write on that. I guess as it has been in the JDK since 1.0 when object allocation was _slow_ I can understand the logic, but for a modern system when object allocation is faster than method calling - this hurts to look at. Why has it not been updated?

Hmm - now I sit back and wait for someone to give me a kicking for these comments - such is life.

Secretes Of Profiling With DPJ - Introduction

2013-04-10T12:31:00.000Z

Want to find find out why and object has not been garbage collected -
DPJ will telll you! I love this view - it says so much
about how a program actual works.

An introduction to profiling with DPJ

Before You Start

[ note that if you click any image in this post a full size version should open in a new window ].

Profiling can be fun and it can be confusing. Simply turning on a profiler and expecting to have the inner secrets of your program laid naked before you is a nice dream; but it is only a dream.

Profilers answer questions. Before they can do that, we need to work out what questions to ask.

Some Example Questions:

Which method takes the most time?
Why does doing A take longer than doing B?
Which part of a transaction takes the most time?
Why does the application run out of memory on Friday evenings?
Which pieces of code does this test touch?
Is my application ever running through methodX?
Why is this object not being garbage collected?
Under which situations is methodY called?

All the above are examples of targeted questions a profiler can help answer.

How do I make my application go fasters and use less memory.

That is an un-targeted question which cannot be directly answered by a profiler. We need to break it down into sub-questions and then answer those.

Profiler Overhead, Instrumentation and What To Profile

Java is very, very fast indeed. Java code will typically run as fast or faster than the equivalent C. Java code is nothing like PHP, Python or Ruby, it is a ends up compiled to native and runs at native speed. Let us consider an example of some Java:

short i=0;
i &= b0 & 0xff;
i <<= 8;
i &= b1 & 0xff;
i <<= 8;

The above piece of bit twiddling will be compiled down (by the JVM Just In Time compiler) to a handful of machine instructions all of which, especially on a 64 bit architecture, are most likely to be register based.

No profiler can possibly find out which of those statements is taking the most time because measuring time will take longer than the statements take to run. It would be like trying to measure someone's heart rate with a calendar. In some ways we can see parallel to Heisenberg's uncertainty principle where there is a fundamental limit to what we can measure.

From a practical stand point, a profiler has to place extra code into the executing program to cause it to measure time. This is called instrumentation. The above example might, after transformation, but equivalent to something like this (though instrumentation is not done at the source code level, but a the intermediate representation and via hooks the JVM places into the running code):

measure_time();
short i=0;
measure_time();
i &= b0 & 0xff;
measure_time();
i <<= 8;
measure_time();
i &= b1 & 0xff;
measure_time();
i <<= 8;

measure_time();

By being smart about how we use the profiler, we can work around these fundamental timing limits and at the same time ensure we only get information we are interested in rather than being swamped by the 'noise' generated by doing too much profiling.

Method vs Source Based Profiling

DPJ can instrument and profile at the method level or the source level. Source level is effectively like placing instrumentation between every line of the source code. Method based profiling only measures at the resolution of methods.

Method based profiling is always the best place to start for Memory or Performance profiling. Once we have a good idea what is going on - then and only then step down to source based profiling.

Switch between method and source (line) level profiling (red and green respectively).

Method based profiling should not significantly slow down an application. This is especially true for performance profiling*. By not instrumenting 'trivial methods' we can be even more sure that the profiler is not slowing down code too much. What we need to avoid is the profiler instrumenting a method which is called very frequently and which does very little. If that happens, we can end up measuring the performance of the profiler not of the application!

Trivial method exclusion means that DPJ analyses the content of methods before instrumenting them. If a method is likely to be a target for in-lining by the JVM compiler, then DPJ will not instrument it. This works pretty well but sometimes we also need to use class exclusion (see below).

* Memory profiling demands more of the profiler because it has to keep track of object allocations and object collections. This also causes the profiled application to scale over multiple cores slightly less than an non-profiled application.

Class Inclusion And Exclusion

Profiling everything can be a huge waste of time. By default, DPJ does not instrument code from the JRE its self and it also excludes other common packages in frameworks like Spring and JEE. This means that we get to see our code and ignore the framework code - which is good. Turning off even more profiling can help us pick up the useful signals we are interested in from the noise surrounding them.

Choose what not to profile (red) or what to profile only (green)

If, for example, we are interested in the performance characteristics of a numeric analysis package, we should instrument only that package and ignore everything else.

If we are not interesting in a thing then we should not profile it! Exclude as much as possible to get just the information we are interested in.

DPJ can either be told to exclude stuff, which is great for trying to get a broad picture of what is going on. It can also only profile those things we explicitly tell it to via 'collect data only for'.

Combining Method/Source Techniques With Exclusions/Inclusion

To find a memory or performance issue a good approach is to do a general profile at the method level then a detailed source level profile of just a few classes which were picked up as of interest by the method profile. For example, in the images below, a method profile lead to seeing that getChannels was a huge consumer of memory and then ...

Method level profiling can show up were we have a method of interest.

Drilling down into the method call graph tells us where is was called from to
give context as to what it is up to.

... a source level profile found the exact line which was consuming so much memory:

OK, so we know that memory usage is somewhere in this method but is a a long method.
No worries, source level profiling can nail the exact line - over quater of a gig allocated
from a single statement - good catch!

Transaction Profiling

1: Turn entry point tracking on and off.
2: Learn how to set up entry points.
3: Put entry points in this box.

Many Java systems come up and run for considerable amounts of time and perform millions of individual tasks during their lifetime. We might be interested in only one of those tasks. For example, why is one page slow to be served when others are not? This can be achieved bu setting up entry point tracking as shown in the diagrams.

DPJ has a lot of built in help - do not be afraid to use it!

Round Up

Profiling is like any form of debugging; it requires good tools, a systematic approach and a bit of patience. Given DPJ, some of these tips and some experience, we can all get Java applications running faster and in less space. That means happier users and lower IT bills - I love win/win situations!

Task - An Ultra Simple C++ Implements Closure Based Language - Started

2013-04-05T10:59:00.000Z

The first page of code!

FOSPL becomes Task - the project has started.

I decided to implement FOSPL in C++ and rename it something more appropriate - Task. I have called it that because it focuses on tasks where each task is the invokation of a closure. The language design is base on FOSPL Chapter 2.

Implementation goals are:

Platform independent.
C++/TR1/Boost - not C++11 yet.
Everything needs to be clean and simple.
Parallel performance is key, serial performance not so.
Lead platform Eclipse/Mac OSX 10.8 xcode gcc4.2

JFDI:

I have kicked off the project by starting to wrap around boost/shared_ptr to get the fundamental types under garbage collection. This is really just a place holder to prove the project has actually started and nothing more. Next step - a very simple parser. I suspect I will just hand implement the parse as a standard recursive decedent parser because the whole point is that the syntax is very simple indeed.

Here is the first piece of code:

//============================================================================
// Name        : task.cpp
// Author      : 
// Version     :
// Copyright   : Affero GPL V3
// Description : Hello World in C++, Ansi-style
//============================================================================

#include <iostream>
#include <vector>
#include <boost/foreach.hpp>
#include <boost/shared_ptr.hpp>
using namespace std;
namespace task
{

 template<typename CONTAINED>
 struct task_atom
 {
  typedef boost::shared_ptr<CONTAINED> type;
 };

 template<typename CONTAINED> boost::shared_ptr<CONTAINED> make_atom(CONTAINED *t)
 {
  return boost::shared_ptr<CONTAINED>(t);
 }

 template<typename CONTAINED>
 struct task_vector
 {
  typedef boost::shared_ptr<vector<typename task_atom<CONTAINED>::type > > type;
 };

 template<typename CONTAINED> typename task_vector<CONTAINED>::type make_vector()
 {
  return typename task_vector<CONTAINED>::type(new vector<typename task_atom<CONTAINED>::type>());
 }

}
int main(int argc, const char* argv[] )
{
 task::task_vector<string>::type args = task::make_vector<string>();

 for(int i=0;i<argc;++i)
 {
  args->push_back(task::make_atom(new string(argv[i])));
 }
 for(vector<task::task_atom<string>::type>::iterator it=args->begin();it!=args->end();++it)
 {
  cout << **it << endl;
 }
 return 0;
}

DevPartnerJ For Mac OSX Prototype Running!

2013-04-02T17:06:00.000Z

DPJ x64 profiling jedit all on the Mac

This is not a product announcement! However, I am completely and utterly blown away with excitement at seeing the first few breaths of life in DevPartnerJ for Mac OS X.

The software took its first fill breath of air at around 17:40 BST today Tues 2nd April 2013. Whilst it only a few minutes old, there is no doubt that it has been seen performance profiling jedit on a 64bit Java 8 JVM!

This internal development version of Micro-Focus DevPartnerJ
can profile 64 bit Java applications on Mac OS X 10.7

This evenings and weekends project by the Dev Partner team has been a labour of love which I for one love to see working:

My Macbook Pro running a debug version of DPJ x64 in the background and showing a jedit
call graph in the foreground.

My Macbook Pro running a debug version of DPJ x64 in the background and showing a jedit
detailed performance figures in the for foreground.

The thread 'Swim Lanes' view of DPJ profiling jedit in 64 bits on the Mac!

FOSPL - Chapter 2

2013-03-31T12:18:00.001Z

OK, a my F***k Off Simple Programming Language ideas continue.

I seem to have picked up a virus and spent quite a bit of last night running a fever. My brain being what it is - somes of the results were dreams and semi-conscious states considering programming language design.

Now - I was quite pleased with FOSPL chapter 1, but is was clearly too complex. I need something even simpler and with less syntax if possible. All that parenthesis is a bit too Lisp like. So, I took the ideas of 'everything is a closure' and ran with it. In this alternative we have three ways of assigning store positions:

Type 1 means that a takes on the value of b. Remember that all data are immutable, so altering anything about the value stored in a will not change anything about the value stored in b (appart from if it is attached to some mutable resource like a file). There is a special rule that if b is a Do task, a will become the return value of the Do task.

Type 2 means a becomes the execution of closure b. As everything, including constants, are actually closures this means a="hello" and a!"hello" have the same effect; the execution of a constant produces a copy of that constant. The difference is when we use more complex closures:

b={1=p 2=q +}

a=b [ a is now a closure which when executed will return the addition of store positions p and q ]

a!b [ a is now a number which is the addition of store positions p and q.

Type 3 means b is executed as a Do task and the task is places in store position a.

Store Positions And Naked Closures

This syntax gives us a very clear separation of that is a closure and what is a store position. Store positions are always to the left of one of the three assignment operators and closures are always to the right (plus one other alternative - see naked closured below). So the following is legal...

1=100

... because it means the store position 1 is given the value 100.

Similarly, we can miss out the store position and execute a naked closure. The parser will assume anything which does not have an assignment operator is a closure to execute...

println

... executes the println closure and discards the return value.

No Need For Parenthesis

This syntax removes any need for parenthesis; all assignments and logic are unambiguous because everything is done through store positions via the three assignment operators. Numeric store positions lend themselves to easy parameter naming as in 1=10 2=100 + .

Closure Return Values

The last missing bit from this design is how to assign the return value from a closure. I am thinking of going with 'the return value of the last closure in the closure' which works because everything is a closure. Let us consider this...

a={a=1 a}

_=a

println

... this means that we assign a closure to store position a. In the closure we assign 1 to store positon a (which will initially hold the closure - but the stores are copies of one another so changing a inside the closure does not effect the contents of the same named store position outside the closure. The closure then executes a, which is storing a constant and constants are closures so the last closure executed inside the closure is 1 so the return value from it is 1. Assuming the store position read by println is _ then the whole script will cause 1 to be printed to stdout.

FOSPL - Thoughts On A Very, Very Simple Programming Language

2013-03-30T17:07:00.002Z

Pronounced Fossple - it stands for Fuck Off Simple Programming Language.

Many years ago I cam up with the notion of VSPL - very simple programming language. The problem was that it was not simple enough! More recently I took some of the ideas from VSPL and implemented them in SFPL - Sound Field Patch Layout. Now SFPL has proven to be much simpler. It also grew in incorporate a fully immutable programming model. As part of that, it came to have a robust parallel programming model via Do Semantics which is all based around closures.

Sadly SFPL is still too complex. First off, it retains the Reverse Polish notation of VSPL. People hate that (I don't know why but I am happy to accept it). Second, it has built in operators and closures, which work in different ways. Thirdly it has list processing built into the syntax which is required to get it to do almost anything.

How about getting rid of almost all the syntax and having only five notions:

Store positions.
Closures.
Do.
Constants.
Return values

A Closure has a store. A store hold immutable data in named positions.

An example:
Adding 1 and 2 then putting the result to stdout:

out(left(1) right(2) add) println

add and println are closures.
out, left and right are store positions.
1 and 2 are constants.
the closure add looks for store positions names left and right, takes the values from them and adds then 'returning' the result.
println is a closure which reads the value of the store position out and sends it to std out.

Closures (just as in SFPL) get a copy of the store from the context in which then are invoked so altering the store in a closure does not effect the store outside the closue.

addAndPrint({out(add) println})

The above defines a new closure addAndPrint. So we can do something like:

addAndPrint({out(add) println})
left(1)
right(4)
addAndPrint

Which is result in 5 being printed to stdout.

I do appreciate in these examples it is rather wordy. However, we could be less wordy with the store position names:

a(1)b(2)+

Above a more minimal example of adding 1 and 2. Parenthesis causes storage into the preceding named store position.

x1(_{a(1)b(2)+} Do)

x2(_{a(3)b(4)+} Do)

out(x1) println

out(x2) println

Above is an example of doing addition in parallel. Do is a closure which executes in parallel what ever is in the _ store position. Braces define a closure.

Symbolically we appear to be mixing closures and store values freely without showing which is which just the behaviour of them is different. That is because closures are stored as store values as we see in the addAndPrint example. The runtime can work out what for of action of perform based on context.

addAndPrint({out(add) println})

In the above context the store position addAndPrint is having something placed into it.

addAndPrint

But in the here addAndPrint is not having something stored so it must be being read and so it should be executed. This does leave one slight problem which is that we cannot assign one store position from another when the other is a closure because the closure will be executed instead. This can be overcome without any special syntax, just via a closure!

_("addAndPrint") Get

Above we see the _ store position to hold the name of the store position we want to retrieve by value and then execute the Get closure. If we want to make a new store position y which has the same value as addAndPrint we can do this:

y(_("addAndPrint") Get)

Have you noticed that this creates a language with only NO KEY WORDS AT ALL. It is ridiculously simple and yet an powerful, simple, scripting language with a strong, immutable data system and built in parallel semantics.

Motivation

I work on cross platform solutions a lot and I am ever more aware of how important parallel programming is becoming. What I want/need is a very simple, cross platform language for controlling build and installation systems and stuff like that. Most simple scripting languages have immutability and parallel processing added to them as afterthoughts; they still require one to think about threads rather than tasks.

FOSPL can be implemented using much of the technology from SFPL but in a very small footprint running in Java and so be of use on all platforms Java runs on. I see it kind of like a shell for the 21st century.

I am not sure if I will actually do the work - might be fun.

The other challenge is if I should actually do it in Java or in C++.

C++/Eclipse On Mac OSX

2013-03-29T20:08:00.001Z

Eclipse editing C++ on Mac OSX (false colour)

XCode is what you are 'supposed' to use - but what if you are working on a very cross platform project in C++?

"I tried XCode. I tried to create a new make file project with it and it immediately crashed. If it had at least tried for a few moments - things might have been different - but we fell out and now XCode is dead to me. (OK - I have been watching far too much Sopranos recently and picked up a taste for the over dramatic). Anyhow - I "wacked XCode and went over to the Eclipse crew."

So, I have installed Eclipse Helios for C++. It is pretty cool and I will describe how import an existing Makefile project into Eclipse. However, the reality is that something else has stepped in and taken my interest. It was just like trying to chat up a new girl friend when our first love from college turns up and looks more gorgeous than ever!

IF YOU CAME HERE ONLY TO FIND OUT ABOUT ECLIPSE ON OSX FOR C++

JUST SCROLL

[Not that I would know - I married my college girlfriend, I am still married to her and she does look more gorgeous than ever!]

And so does vi!

MVIM - is amazing. I learned vi back when I used a dial up modem to program remote UNIX machines (like the venterable Cray YMP). There was no other editor which was good enough to easily write FORTRAN and was always available. OK, a lot of people were big Emacs fans. However, it took more bandwidth, was not everywhere and the key strokes just never cam naturally to me. The other 'big advantage of Emacs' was/is supposed to be Lisp. Now - I realise that a lots of people love Lisp; I don't love it, shall we leave it at that?

Anyhow - I digress. It suffices to say that mvim (MaxVIM) has pretty much stolen eclipse's thunder for me. The command line tools like gdb, find, mvim and grep so completely replace the functionality of IDEs for C++ that it is hard to keep up the effort of using the IDE.

Vi is a bit like the Lee-Enfield. Shockingly
effective given the earliness of its design.
(Image is wiki-commons)

Note: this is not the case for Java where the IDE is fantastically powerful. No C++ IDE comes close to the power available to Eclipse/Java because of the nature of the Java programming language and type system.

BACK TO ECLIPSE

Now back to Eclipse. For cross platform development make is the only way to go. This does not include Windows. The elephant in the room is that any project with Windows and *nix in it will have to have two parallel build systems or be cripples. Visual Studio's build system and IDE are great (really, really great) but 100% Windows only. So, here I am looking at make based projects.

The first step (assuming you have Googled Eclipse IDE C++ Mac)

I thought this might be the option to import my
makefile project - but no!

I tried using the import (File>Import) and then "Existing code as Makefile Project" but I think that is for existing code which does not yet have a makefile. I was completely unable to get it to work with my code and the only reward for trying was the following stack trace:


Alexanders-MacBook-Pro:eclipse alexanderturner$ java.lang.reflect.InvocationTargetException
 at org.eclipse.ui.actions.WorkspaceModifyOperation.run(WorkspaceModifyOperation.java:121)
 at org.eclipse.jface.operation.ModalContext$ModalContextThread.run(ModalContext.java:121)
Caused by: org.eclipse.core.internal.resources.ResourceException: Invalid project description.
 at org.eclipse.core.internal.resources.Project.checkDescription(Project.java:162)
 at org.eclipse.core.internal.resources.Project.assertCreateRequirements(Project.java:52)
 at org.eclipse.core.internal.resources.Project.create(Project.java:274)
 at org.eclipse.core.internal.resources.Project.create(Project.java:256)
 at org.eclipse.cdt.core.CCorePlugin$3.run(CCorePlugin.java:766)
 at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:1975)
 at org.eclipse.cdt.core.CCorePlugin.createCDTProject(CCorePlugin.java:758)
 at org.eclipse.cdt.core.CCorePlugin.createCDTProject(CCorePlugin.java:748)
 at org.eclipse.cdt.managedbuilder.ui.wizards.NewMakeProjFromExisting$1.execute(NewMakeProjFromExisting.java:83)
 at org.eclipse.ui.actions.WorkspaceModifyOperation$1.run(WorkspaceModifyOperation.java:106)
 at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:1975)
 at org.eclipse.ui.actions.WorkspaceModifyOperation.run(WorkspaceModifyOperation.java:118)
 ... 1 more

So I tried "Existing Projects Into Workspace - that did work. It found the code and make file and I selected it and went onto the dialogue below in which I went for MacOSX GCC:

Turns out this is the way to do it - Eclipse found the makefile
and run with it.

After a longish wait (don't get impatient and assume it has failed - like I did) Eclipse manage to import the project, kick off a build and produce a build output. It did an OK job of working out that there were warnings in the make output. The first build through seemed to have some sort of issue where clicking on a warning did not bring up the source; however, this ultra-useful feature worked as expected on later builds.

And here we have it - editing C++

Well - it is an ide - here are some pluses and minuses:

+ Fn^F3 works for find definitions.
+ Find all references works.
- Code annotations and warnings show the wrong bit of the output from make so are not as helpful as one might like.
+ Find/Search work OK.
+ Build calls make as one might expect.
- Setting up build targets is fiddly.
- Clicking on errors/warnings takes you to the text.
+ Auto indent is OK.
+ Unresolved inclusions are annotated.
+ Build output is parsed and colour coded.
+ Syntax errors are annotated.
+ Ctrl^Space competition suggestions works.
+ FIXME and TODO are parsed and show up in the tasks pannel.
- It is big - over 350Meg when only running a few minutes.
- Really difficult (I have not managed yet) to use the debugger.

All in all - it is pretty good. I am not sure why I am not raving about it - maybe time will tell. Maybe I love Eclipse for Java so much it is hard to live with the limitations it has for C++? Maybe it is just my infatuation with vi's Mac make over?

Memory Profiling Complex 64bit Java Applications

2013-03-23T08:51:00.003Z

This image is public domain.

Here I demonstrate using DevPartner Java edition 64bit to profile a large and complex Spring application for memory.

Memory profiling is not just leak detection or heap size but being able to see the lifetimes, inter-relations and references between objects, objects and the heap and even objects and the stack.

Please note - at the start of the video I mention that I will explain where the byte arrays are being used. This did not make it into the video - but I did mention it. They are part of the system for serialising float arrays to disk.

This video gives a brief introduction to this huge area which is of great importants especially with the growth of massively parallel systems like Hadoop.

x86/64 Cross Compiling On Suse 10.3 And the i486 Problem

2013-03-22T12:16:00.000Z

Another 'I did this do you don't need to' post on cross platform devel'!

I've been working Open Suse 12.2 for a few weeks now and it came time to back port the code to Suse (SEL) 10 Sp3. Cross platform devel' on Open Suse seemed to take a bit of fiddling. Maybe because I have been through that or maybe because it is an earlier product, but on SEL 10.3 things were a bit more straight forward.

BTW - I am super please that my client was happy to only go back as far at 10.3. That has gcc 4.1 with the atomic builtins - no more assembler :)

OK - step one after a clean install of the OS is to go to the 'computer' icon in the bottom left of the screen and from there click on 'Install Software'.

Install Software is on the right of the 'Computer' pannel.

Next we need to simply search the gcc. Then select the packages I have shown below. This will give gcc and g++ - who needs any more than that?

Install gcc

The installer should work out and add any dependencies required:

Let the installer add in dependencies.

The symptoms of glibc 32bit not being installed

But then your build might still not work. This is because the required dynamic loading library is missing by default even when the 32 bit cross compiling option is available in gcc/g++. The symptom of this can be seen in the 'before and after' listing from /usr/lib. Note that on Suse, 32 but libraries and shared objects end up in /usr/lib and the 64 bit ones in /usr/lib64.

user/lib before and after installing x86 glibc.

The solution is it install the 32 bit devel' version of glibc. This can be done by searching for 32bit in the software installer.

Install glibc 32bit

If you need to check what is already installed - Yast can do this for you with the Software Management console:

Yast - software management.

I final note on those atomic builtins. They do not link by default with Suse 10.3. To get them to link you need to pass -march=i486 to gcc.

Happy developing!

MIT Desktop Metaphor Replacement Effort A Sad Joke

2013-03-19T09:24:00.001Z

Yes - there is is - top seller of the category - a machine
which boldly states 'No OS is the new OS'

No Interface Is The New Interface

Guess which laptop is the Amazon.com best seller today and has been for 150 days? The Chrombook! I bought one of these (and older model) when they first came out in the UK. The user experience then was nothing like as good as it is now and I was concerned that the Chromebook might die. Well - the answer to my question 'Dawn or Dusk' was dawn. The Chrombook has gone from strength to strength. Even my old one is much better due to all the enhancements to web based user experience.

Chromebooks do not have a operating systems in the way we think of them these days. Most devices, phones, laptops, even some TVs have an operating system which control the user experience. We, as users of the device, are aware of the operating system. Most of the time on a Chrome book, we are not aware of the operating system at all. The user experience is controlled by what ever website we happen to be on at that moment.

Despite its limitations, I burn hours on my Chromebook second only to my work computer. I don't even use my Mac Book Pro Retina for surfing the web in the evening - no - the Chromebook does that.

NO OS WORKS!

This screen shot is of an American site and as such comes
under fair use.

This is way MIT's attempts at creating a new metaphor for human operating system interaction are so wide of the mark:
http://www.technologyreview.com/featuredstory/401304/the-next-computer-interface/

This whole subject is irrelevant. It is sad to see such clever people even thinking in these terms. The idea of a personal computer (note - I don't mean a PC - any such device) having more than a trivial effect on the user experience is 100% 20th century thinking.

It is not the desktop metaphor which is outdated - it is the operating system metaphor which has had its day.

All computing will be web based linking back to cloud back ends long before these people have had any impact one our experience. It is the operating system metaphor which is dead - not the desktop metaphor. The funny thing is that Gate clearly saw this in the 90s but could not work out how to make money from it. Remember web pages embedded into the desktop and the web unification ideas in Win98?

Java 8 Lambdas - 10 Things That Are Wrong With Them

2013-03-18T15:00:00.000Z

Looking cool is not always good for your health.
Lambda's are cool, but will they been
good for the health of Java?
This image is Creative Commons - see here.

This is a personal view of things I am not so keep on when it comes to Java Lambdas. DO NOT GET ME WRONG - I like Java Lambdas and am very glad they have been added -> but ...

Lambdas have one huge benefit for any language - locality of implementation. Java anonymous inner classes already provide this but with a syntax which is so verbose as to all but eradicate the benefit. With lambdas Java has achieved true locality of implementation where operational code can be placed right into the parameters of method calls etc. Also, with method and constructor references, Java 8 has an amazingly light weight syntax for chaining functionality together.

However, we programmers always have our own likes and dislikes. I have compiled a list of 10 things I don't like about Java 8 lambdas. This is 100% a personal view and I would like to point out that I struggled to find 10!

Cannot close around mutable local variables.

I know some people think this is a good thing because it avoids adding new threading race conditions to the language. For me, the experience of closing around the current scope by reference in with C++11 lambdas has been so positive that I was really sorry to see one cannot do so in Java 8. A local in the outer scope needs to be final (or effectively final - i.e. execution path tracing can show it not to change) before it can be accessed from within the lambda. See point 6 below as to why this is restriction makes even less sense than at first sight
Use of InvokeDynamic.

I am not happy with this - it seems over engineered and leaves open the paradigm of implementing more and more at runtime rather than compile time. I really hate the idea Java will start to become more dynamic as a language; this is largely because I, personally, hate dynamic languages.
No auto keyword.

Defining a lambda on the fly and using it without having to explicitly define the type is a major help to syntactic brevity and implementation locality (type inference - not duck typing**). Not having this feature in Java is a big problem and it is further made worse by not having any way of defining a true generic function type (as we can with C++11 templates).
No control over what they close around.

Java 8 lambdas close around the entire state. They always close by value. These makes isolation and encapsulation much less rigid than in - you guessed it - C++11.
Potential mine field for state.

Item 4 is part of this problem but it is not all of it. Lambdas implement interfaces which can (now) have default implementations which can then lead off to static state and generally allow very bad programming practice. Having said that, all my previous comments seem to be about C++ which makes a special effort to allow every single bad programming practice available simultaneously.
Final is not const.

Lambdas are great for parallel programming - but only if their state is immutable so that inter-thread communication is not required after the initial scatter. BUT - Java cannot enforce this because of its 'everything is a reference' model of objects. Just marking a an object reference used by a lambda as final does not mean it cannot be mutated - it means the reference cannot be changed. So, enforcement of final is left up to the programmer. Why can't there be a REALLY_VERY_FINAL_INDEAD key word (or something like - err - const) which enforces true immutability?
Poorly communicated.

Until I found http://www.lambdafaq.org/ I was stumped to find any good docs on Java 8 lambdas. It is as though it is all being kept a secret.
Tooling nightmares.

Delaying implementation to runtime and uncoupling it from JVM specification is a tooling nightmare. Profilers and debuggers are going to have to wade through all this complex engineering without a standard to help them. WRONG!
Operator blight.

Not one, but two new operators help add to a snow storm of symbols driving Java away from its easy to read roots. Callable c = () -> () -> { System.out.println("hi"); }; - barf - brain death imminent. An example using method references and lambda definition at the same time would be even better for screwing with the mind completely.
Mixin abuse, Java becomes Ruby (ultra-barf).

OK - this last one is not strictly about Lambdas but has been introduced as a consequence of Lambdas or at least to support them and it is quite a big problem. Default implementations on interfaces look a lot like mixins. We can just add stuff to a class by adding implements to it. Oh - it is just too horrid to contemplate. Multiple mixing inheritance in Java is a lot like dioxin the in water supply - sooner or later it will lead to grieve.

** If it looks like a duck and sounds like a duck then we can assume it is a duck - and it plus everything todo with it can be shot so we can get back to type safety.

C++ 'Static Constructor' Trick For time critical unordered_map

2013-03-18T13:29:00.001Z

Having fun with containers!
Creative Commons - see here.

C++ does not have static constructors; however there are ways for making behave as though it does. This can make unordered_map behave consistently and deterministically - which is great for realtime or instrumentation systems.

I have been performing a lot of 32 to 64 bit porting over the last 2 years. One of the biggest issues with this comes from idiomatic legacy code. The 32 bit platforms where dominant for so long, many developers who wrote code which is now legacy or at least old never programmed anything else. This has lead to a 'pointers are 32bit integers' view of the world which is just wrong headed. The snag is that, for more than 20 years, it also happend to work.

So, it is for this kind of work that I have ended up needing a very fast indeed way of mapping from true pointers to 32 bit integers. Such a mechanism means the logic code which treats pointers as integers can remain in place (unless pointer arithmetic is being done - in which case there is no option but to do a full port) whilst the pointer code can simply move from int to size_t and ptrdiff_t.

Why unordered_map is ideal

I am working with older code (pre C++11) and so I am using tr1 templates, but C++11 std templates do exactly the same thing. unordered_maps make the perfect (ish - see later) way of translating form pointers to integers. The reason being that there are template specialisations for pointers and integers in the unordered_map standard and when an unordered_map is not resizing, both read and write operations are zeroth order scaling (ish - see later).

But - there is a little issue

Did you notice my cavet? 'when an unordered_map is not resizing'. If you measure the insertion time of an unordered_map as you insert many elements into it, the observed scaling is nlog(n). This is because every so often the map must resize its self to have more buckets to hold records. This is a complete copy operations and so scales a n but increasingly less frequently resulting in an over all nlog(n) complexity for insertion.

That can be a killer, huge issue

In time constrained systems like profilers or real times systems, unordered_maps offer a greater menace than just nlog(n) scaling. The time to insert an element is (effectively) non deterministic. A profiler should measure the time for an operation which is being profiled with a consistent overhead. Clearly the profiler will add an overhead, but that can be removed if it is consistent and/or deterministic. Similarly, for a real time system, the time constraints can be tailored and the performance tuned to avoid missing time cutoff points. But, if a call sometimes take 10ns and sometimes 10ms, we are faced with either making the time constraints on the system much longer than we would like or allowing more missed cutoff points than we would like. Consistency and deterministic behaviour are just MUST HAVE attributes in many situations.

A work around

The rehashing problem can be completely removed if we know the maximum number of elements we will ever have in the unordered_map. We can use the ::rehash member to pre-size the map ( as demonstrated in the code snippet below). However, we want to achieve that before the map is ever used. That can be guaranteed by performing it from a constructor of a global object. This pattern relies on the unordered_map construction happening before the global rehashing object which can be enforced due to the 'in file order' rule of construction:

static std::tr1::unordered_map<const void*, int> mapPointerToIndex;
static std::tr1::unordered_map<const int,void *> mapIndexToPointer;

// Pre-allocate a bunch of buckets so that
// for small runs, rehashing the map does not cause
// spurious pauses in operation
#define INITAL_BUCKET_GUESS 1048576
class setUpHandlMap{
public:
 setUpHandlMap(){
  mapPointerToIndex.rehash(INITAL_BUCKET_GUESS);
  mapIndexToPointer.rehash(INITAL_BUCKET_GUESS);
 }
};

setUpHandlMap staticInitializerOfMaps;