Thinking about it in a machine learning setting, let’s say that we have a dataset containing cats and dogs, and their respective names. We would like to classify an unknown pet as either a cat or dog, given only its name. We can use Bayes rule to help us.

First, let’s start off by understanding where the rule comes from. We can use conditional probability identities to get some results.

The probability the the pet is called Rover and is a dog, is the probability the the pet is called Rover given that it is a dog, multiplied by the probability that it is a dog.

Similarly, we have:

And now, because the left hand sides of both of those equations are the same, we have:

Now for the magic – divide both sides by the probability that the pet is called Rover, and — Boom! — Bayes rule.

Why is this good and useful? Because it allows us to find the probability of a label given a constraint, using known probabilities from the training data!

Replacing dog with the variable class (or label), and Rover with the variable feature, we have derived the following:

i.e. a way of determining the class of an item given one (or more) feature constraints. How does this help with a classification machine learning problem? Well, we could calculate the probability of each class given the constrains, and select the class with the highest recorded probability.

Let’s finish off the pet example. We have the training data in the table below. We are given a mystery pet, which we know is called Rover. We wish to predict whether it’s a cat or a dog using only its name and the probabilities derived from this table.

Now we have , and . Of the 5 dogs, two are called Rover, so we have . Of the 5 cats, one is called Rover, so we have . Finally, of all the pets, three are called Rover, so . We now have everything required to make a prediction.

For a cat:

And for a dog:

So our model has made a prediction that the pet is a dog – which sounds good to me!

OK, so can we trick it?

From the pet dataset above, it might seem that dog was given the highest probability because there are more dogs called Rover than cats called Rover. OK, so can we fool it by introducing a new category, fish, which are all called Rover?

Let’s try, by using the above dataset, but adding a single fish which belongs to a joker who decided to call it Rover. That means that we now have , , and .

Notice how does not change — given a dog, there is still a probability of 0.4 that it will be called Rover. Similarly, remains the same. Now we can also do the same for a fish — there’s only one fish, and its name is Rover. Finally we have

So the question is, will it trick the system? Let’s run the numbers.

For a dog:

For a cat:

For a fish:

So no! Even though a fish is most likely to be called Rover, there are many more dogs, so there are many more chances for the pet to be a dog than there are for it to be a fish. The model still correctly (according to intuition, at least) predicts that the pet is a dog.

And this is what leads to the often contradictory and surprising results from the theorem. You know the ones – results like if a doctor tests for a rare disease (say, 1 in 10000 people), and the test is 99% accurate, and you test positive, chances are high that you actually do not have the disease. That’s because the disease is so rare, that the number of opportunities for a false positive outweighs the number of opportunities for a true positive.

But the problem is exactly that. I’ve been sitting and wishing, not learning and doing. True, I did a course on Coursera a few years ago, but that hardly cuts the mustard. So I’ve decided that now it’s my time. I would really like to get into it, and so here’s my commitment to myself to do it.

I don’t mean just watching videos and reading books, or even creating pretty visualisations or learning how to throw something together using Scikit Learn. I want to know what’s going on so that I can make good and informed decisions, communicate them, and (perhaps one day!) become a data scientist.

One of the best ways that I know to be sure that I know material, and can talk about it confidently, is to talk about it. So that’s what I’ll be doing. Talking about it. Writing about what I’ve learnt, am learning, insights that I’ve had. Consolidating my knowledge. Who knows, there may be some cool visualisations too.

What I must remember along this journey (huge note to self) is that if this is going to be successful, I need to just do it. The aim is not to write lecture notes or a book. This is reinforcement learning (pardon the pun), and it’ll only work if I reinforce the learning – i.e. just write. As such, if you’re reading this, please don’t expect any kind of order!

To get started, I’ll be working my way through An Introduction To Machine Learning.

I figured that this is also a good excuse to practice my skills and write it out.

We want to find the general solution of the equation:

Start off by dividing through by

then complete the square

rearrange to leave the square on the left

simplify the RHS

take the square root of both sides

simplify the RHS

make the subject

and, boom – the quadratic formula!

If I could go back in time and tell my younger self one thing, it would be to play with maths – don’t just do the homework. It’s all very well being able to follow a process (and essentially being a calculator), but knowing and understanding where something comes from unearths the true beauty of maths.

I put it together on codepen (which is my new favourite tool, by the way) so feel free to fork it and see what you can do with it!

It takes a little while to draw, but it’s quite interesting seeing it building up.

The javascript code

var WIDTH = 800,
	HEIGHT = 500,
	PARTICLES = [],
	STARTING_COUNT = 600,
	MAX_PARTICLES = 600,
	DAMPING = 0.05,
	CURRENT_FRAME = 0,
 PADDING = 10;

var canvas = document.getElementsByTagName ('canvas')[0];
canvas.width = WIDTH;
canvas.height = HEIGHT;

var ctx = canvas.getContext ('2d');

function random (min, max) {
	return (Math.random() * (max - min) + min);
}


function _particleTimer () {
	addParticle (random(0, WIDTH), random(0, HEIGHT));
	if (PARTICLES.length < STARTING_COUNT) setTimeout(_particleTimer, 0);
}


function init () {
	_particleTimer();
	animate();
}


function addParticle (x, y) {
	PARTICLES.push ({
		position: {
			x: x || random(0, WIDTH),
			y: y || random (0, HEIGHT)
		},
		force : {
			x: 0,
			y: 0
		},
		vel : {
			x: 0,
			y: 0
		}
	});
}

function animate () {
	while (PARTICLES.length > MAX_PARTICLES) PARTICLES.shift ();


	CURRENT_FRAME++;
  

	
	
	var force = {x : 0, y: 0};
	for (var i = 0; i < PARTICLES.length; i++) {
		var p1 = PARTICLES[i];
	

		for (var j = i + 1; j < PARTICLES.length; j++) {
			var p2 = PARTICLES[j];

			// length vector
			force.x = p2.position.x - p1.position.x;
			force.y = p2.position.y - p1.position.y;

			var magnitude = mag (force);

			var actingDistance = 50-magnitude;

			if ((actingDistance > 0) && (magnitude > 0)) {
				var red = 0;
				var green = 255;

				var color = green - actingDistance * (green / 2);
				if (color < 0) color = 0;
				if (color > 255) color = 255;

    if (color < 115 && color > 10) {
			  	ctx.save ();
			  	ctx.strokeStyle = 'hsla(' + color + ', 100%, 50%, 0.05)';
				  ctx.strokeWidth = 1;
			  	ctx.beginPath ();
			  	ctx.moveTo (p1.position.x, p1.position.y);
				  ctx.lineTo (p2.position.x, p2.position.y);
			  	ctx.stroke();
				  ctx.closePath();
			  	ctx.restore ();
    }

				force.x *= DAMPING * actingDistance / magnitude;
				force.y *= DAMPING * actingDistance / magnitude;
				p1.force.x -= force.x;
				p1.force.y -= force.y;

				p2.force.x += force.x;
				p2.force.y += force.y;
			}
		}
		draw (p1);
		update (p1);
	}
 if (CURRENT_FRAME > 1500) {
   alert('finised drawing!');
   document.getElementById("message").innerHTML = 'Done!';
 } else {
   setTimeout(animate, 0);
 }
}

function mag (vector) {
	return Math.sqrt (vector.x * vector.x + vector.y * vector.y);
}

function update (particle) {
	with (particle) {
		vel.x += force.x;
		vel.y += force.y;
		vel.x *= 0.8;
		vel.y *= 0.8;

		position.x += vel.x;
		position.y += vel.y;

		force.x = 0;
		force.y = 0;

		if (position.x > WIDTH - PADDING) {
			position.x = random(PADDING, WIDTH - PADDING);
		}
		if (position.x < PADDING - 3) {
			position.x = random(PADDING, WIDTH - PADDING);
		}
		if (position.y > HEIGHT - PADDING) {
			position.y = random(PADDING, HEIGHT - PADDING);
		}
		if (position.y < PADDING-3) {
			position.y = random(HEIGHT, HEIGHT - PADDING);
		}
	}
}

function draw (particle) {
	ctx.save ();
 ctx.globalCompositeOperation = 'xor';
	ctx.translate (particle.position.x, particle.position.y);
	ctx.fillStyle = "rgba(255,255,255,0.1)";

 ctx.fillRect (0, 0, 1, 1);
	ctx.restore ();
}

init();

1. we overuse the word awesome
2. it would be cool to use this as an excuse to use the Twitter streaming API

So I fired up npm, installed ntwitter and within minutes had simple program that printed out to the console time there was a tweet (with a location) that had the word ‘awesome’ in it. I then got it to store them in a MySQL database as well, and excitedly got to work hacking together a PHP script to read the tweets from the last hour and output them as JSON.

It was an amazing moment when I loaded up the page and saw it working properly for the first time. Unsurprisingly, the place where awesome is mentioned most often is in the US, but it’s interesting to watch the red moves across the country through the night.

Visit the awesometer!

The processing code

// canvas colour
int background_color = color(70, 70, 70);
// particle colours
color[] colors = {
      color (227, 211, 31), // yellow
      color (227, 31, 123), // pink
      color (31, 188, 227) // blue
    };
// colour of the stroke around the particles
color stroke_color = color(0, 70); // black, 70% alpha

// should the mouse draw particles when moved?
Boolean INTERACTIVE = true;
// should the canvas be cleared on each particle iteration
Boolean CLEAR = false;

//-----------------------------------------------------------------//
//-----------------------------------------------------------------//
//-----------------------------------------------------------------//
//-----------------------------------------------------------------//
ArrayList particles;

// reset on mouse click
void mouseClicked () {
  setup();
  mouseMoved();
}

// save on press spacebar
void keyPressed () {
  if (keyCode == RETURN || keyCode == ENTER) {
    save("Colourful entrails " + year() + "-" + month() + "-" + day() + " " + hour() + ":" + minute() + ":" + second() + ".png");
  }
}

// add particles on mouse move
void mouseMoved () {
  if (!INTERACTIVE) return;
  int count = int(random(2, 5)); 
  while(count-- > 0) {
    addParticle(new PVector(mouseX, mouseY));
  }
}

void setup() {
  size(800, 400, P2D);
  blendMode(ADD);
  stroke(stroke_color);  
  background(background_color);

  particles = new ArrayList();

  // create a particle which'll go around making particles if it's not interactive  
  if (!INTERACTIVE) {
    Particle generator = new Particle(new PVector(width/2, height/2));
    generator.size = 10000;
    while(generator.isAlive()) {
      int count = int(random(2,5));
      generator.update();
      while (count-- > 0) {
        addParticle(generator.position);
      }
    }
  }
}

// add a new particle to the particles arraylist with specified position
void addParticle (PVector position) {
  // clone the vector 
  PVector position_clone = new PVector(position.x, position.y);
  particles.add(new Particle(position_clone));
}

void draw() {
  // keep track of dead particles - they'll be removed after the loop
  ArrayList dead = new ArrayList();

  if (CLEAR) {
    fill(background_color);
    rect(0, 0, width, height);
  }

  // go through each of the particles and update/draw
  ListIterator it = particles.listIterator();
  while (it.hasNext()) {
    Particle particle = (Particle) it.next();
    // only draw if it's alive
    if (particle.isAlive()) {
      particle.update();
      particle.draw();
    } else { // otherwise add to dead list
      dead.add(particle);
    }
  }
  // remove all dead particles
  particles.removeAll(dead);
}

// a tad messy, building it up as i go along
class Particle 
{
  PVector position;
  PVector velocity;
  PVector force;
  float size;
  color c;
  int life;

  Particle (PVector p) {
    life = 0;
    position = p;
    size = random(30, 50);
    velocity = new PVector(random(2, 5), 0);
    rotate2D(velocity, random(0, TWO_PI));
    c = colors[int(random(colors.length))];
    force = new PVector(0, 1);
  }

  void update () {
    life++;
    if (life == 20) {
      velocity.mult(1.1);
    }
    if (life < 20) {
      size *= 0.99;
    } else {
      size *= 0.95;
    }

    position.add(velocity);
    velocity.rotate(random(-QUARTER_PI, QUARTER_PI));
  }
  void draw () {
    pushMatrix();
    translate(position.x, position.y);
    fill (c);
    ellipse(0, 0, size, size);
    popMatrix();
  }
  Boolean isAlive() {
    return size >= 1;
  }
}

void rotate2D(PVector v, float theta) {
  float xTemp = v.x;
  v.x = v.x*cos(theta) - v.y*sin(theta);
  v.y = xTemp*sin(theta) + v.y*cos(theta);
}

Anyway, I charged up my camera battery and thought that I’d start taking pictures of space. Yesterday I got a couple of blurry overexposed pictures, but today I tweaked the settings a little and ended up with the picture shown above. I’m really impressed.