Heaton Research

Preferred language for AI and Data Science?

jeffheaton — Tue, 02 Sep 2014 17:32:08 +0000

Java build tool of choice.

jeffheaton — Sun, 12 Jan 2014 00:48:35 +0000

Have you ever backed a Kickstarter project?

jeffheaton — Mon, 20 May 2013 13:26:56 +0000

Encog 3.2 Machine Learning Framework released to beta

jeffheaton — Fri, 17 May 2013 13:28:55 +0000

Both the .Net and Java versions of the Encog Machine Learning Framework have been released to beta. The 3.2 version adds many features in the area of evolutionary algorithms. Specifically enhanced support for NEAT Neural Networks, HyperNEAT Neural Networks, Genetic Programming and Genetic Algorithms.

There was a previous Java beta release, however for simplicity I am labeling both of these as beta-2.

https://code.google.com/p/encog-java/downloads/list

https://code.google.com/p/encog-cs/downloads/list

Encog 3.2 Beta-1 for Java Released

jeffheaton — Thu, 18 Apr 2013 01:50:56 +0000

Encog 3.2 beta-1 is released. Now I am on to Encog C#. Also want to create a few more examples to address some FAQ's. You can download Encog 3.2 from the following URL.

https://code.google.com/p/encog-java/downloads/list

Some of the major new features in Encog 3.2 include:

Much improved NEAT, with multi-threading
HyperNEAT
Genetic Programming
Code generation for NinjaTrader
Code generation for MT4

There will be more articles coming on the above features. There are examples for each already.

With regards to Artificial Intelligence are you solution, implementation or curiosity focused.

jeffheaton — Thu, 28 Mar 2013 13:06:50 +0000

Iterations and Epochs are Evil

jeffheaton — Wed, 20 Feb 2013 10:41:47 +0000

Many machine learning training algorithms have the concept of an iteration or an epoch. Both terms mean the same thing. An iteration is simply on pass through the training algorithm. At the end of each iteration the accuracy of the machine learning algorithm is evaluated. This process is continued until the accuracy is sufficient.

The training iteration is something of a black box. What happens during a training iteration is largely dependant on what type of machine learning algorithm you are training. However, there are typically several steps and the machine-learning algorithm is not actually taught anything until the final step of the iteration.

The problem with this approach is that it often does not work terribly well with multi-core architecture. Additionally, it works even worse with a multi-computer grid. The reason for this is the threads must bottleneck at the end of an iteration. To use OpenCL terminology, the end of an iteration is a fence. Meaning all threads must reach that point before we continue. Then once we do continue, all of the threads must start up again. It would be as if we had to clear the entire global airspace once a day to complete the daily iteration. This is very wasteful, in terms of getting all of the aircraft down and them back up into the air again.

Not every aircraft can land instantly. Some are probably fairly far from an airport. Some are probably in the middle of a large ocean. So have to wait for every aircraft (thread) to finish its current task (a flight). Then we update something, and let the entire aircraft take off again. If you can pardon the pun, this approach "would not fly!"

Yet for a multi-threaded iteration based training algorithm this is exactly what you see. Look at my CPU usage history below. This is from a hyper-threaded i7 quad core. See the "icicles"? Can you guess where my program hit an iteration? Correct! At each icicle. In this image it is not that bad. There is quite a bit of training data. The longer an iteration takes, the more efficient this approach is. However notice on a small training set. The icicles are much larger!

Performance has frozen over!

A much better approach is to have a never-ending supply of small work-units. This can be challenging. And it often means changing the underlying algorithm to some degree. As an example of this we will look at Encog's Genetic Programming algorithm.

Genetic Programming works by creating a population of "programs" or "equations", referred to as genomes. Each of these genomes is evaluated and the top genomes mate and mutate to produce the next generation. In genetic programming iterations are often called generations. The typical approach for a non-species based genetic algorithm is as follows.

Create a random population of genomes
Evaluate every genome in the population
Add some percentage of the top genomes directly into the next generation (elitism)
Randomly choose to mutate or crossover
For mutation choose one parent from the population via tournament, then mutate it and add the child to the next generation
For crossover choose two parents from the population via tournament, then breed a new offspring that is added to the next generation
Does the next population have the desired population level? If not, go to step 3
Replace current population with new population
Continue back to step 2 until the evaluation has a good enough score

The iteration ends in step 8. This is an iteration. Steps 4 through 7 can be done in parallel. But then the whole process must synchronize at step 9. An icicle!

Humans like iterations! There are many children in the USA right now in Fifth Grade (assuming you are not reading this in the summer). They will all enter Sixth grade at approximately the same time. Typically sometime in August or September. That is not terribly natural.

An example more from nature is generations. This might be a Western concept, but I identify myself as "Generation X". My parents are baby boomers. Children in school now are either Generation Y or Millennials. When did Generation X stop? Who was the very last baby that was born into generation X? No one knows exactly. This is an attempt to place human conceived labels onto a natural process that cannot so easily be labeled.

We need a similar process for Genetic Programming. Encog GP makes use of the following process that is constantly running inside of a pool of threads equal to the core count of the machine.

Randomly choose to mutate or crossover
For mutation choose one parent from the population via tournament, then mutate it producing a child
For crossover choose two parents from the population via tournament, then breed a new child
For each child, choose an existing population member via anti-tournament to kill. The child replaces this "unlucky" genome.

Some thread synchronization is very necessary for this to happen. Particularly where genomes are entering and exiting the population. However, there are no icicles with this approach. The CPU is pegged solid!

You might be asking, "What happened to elitism". Elitism is where the top genomes are passed directly onto the next generation. This is actually not a natural process. The fact that genomes are killed by anti-tournament accomplishes much of the same thing.

AI Debugging

jeffheaton — Sun, 27 Jan 2013 13:49:01 +0000

The fact that Encog corrects for many of my dumb mistakes can make debugging interesting. Most machine learning algorithms contain multiple sub-algorithms to ultimately converge. Consider NEAT, or any GA for that matter. You have mutation and crossover working. Both going at the same goal (convergence) just by different means. Even if I destroy the crossover operation to the point of being counterproductive (like I just did by mistake), it still converges. Mutation just steps up to the plate and works harder. Of course overall convergence time suffers, but it still fundamentally "works". There are other examples of this. LMA is another great example, mess up either the Newton or Gradient Descent and the other side steps up. This is why when I make changes a real "warning sign" that I have to look out for is if all of a sudden more iterations are needed for something, even if it actually "works".

Updates on Encog NEAT and HyperNEAT Progress

jeffheaton — Mon, 21 Jan 2013 01:55:34 +0000

Over the last few weeks NEAT and HyperNEAT have been the focus of my Encog efforts. Support for NEAT has existed in Encog for some time. However, it has never worked terribly well. There were many bugs that were fixed during Encog 3.2. Additionally, there have been many performance improvements to NEAT.

HyperNEAT is the current area of work. HyperNEAT builds directly on top of NEAT. In a nutshell, HyperNEAT works by evolving neural networks to create neural networks. The final output from HyperNEAT is a neural network called a CPPN. The CPPN is used to create a regular NEAT neural network capable of solving the problem you are training for. This two-phased approach can allow HyperNEAT to train for very high-dimensional problems. These same high-dimensional problems would take an eternity to train in regular NEAT.

At this point I have both NEAT and HyperNEAT working relatively well in Encog for Java. There are more improvements to be made before I consider it done. However, it is getting close. The performance is not nearly where I would like it to be. Currently Encog NEAT is single threaded. I will add multi-threaded last. My goals at this point are as follows.

Finish last remaining HyperNEAT issues
Work on some core performance issues in NEAT/HyperNEAT
Port from Java to C#
Multi-Thread Java
Multi-Thread C#

I really want to have Encog 3.2 finalized before porting to C#. This limits the amount of “reporting” that I need to do as Encog 3.2 evolves. However, the multi-threading is very different on C# and Java. I do not want to shoehorn a Java threading model into C#. C# has the very cool Parallel class that I am trying to use through Encog for multithreading. Java works more using a thread pool.

JLatexMath Tutorial for Rendering LaTex Formulas in Java

jeffheaton — Thu, 03 Jan 2013 10:43:25 +0000

LaTex is a complete typesetting system that can be used on a variety of computing platforms. I make use of LaTex to produce the PDF versions of my ebooks. I also produce the “master” PDF that the printed copies of my books are created from. However, Latex also has another very common use. Latex is a very common means of encoding mathematical formulas for eventual typesetting.

MediaWiki makes use of LaTex to enter mathematical formulas. This is the Wiki software used by Wikipedia. I also make use of MediaWiki to run the Encog wiki. As a result, I find my self frequently making use of LaTex formulas.

Encog is now adding Genetic Programming (GP). GP allows you to evolve programs to provide solutions to an objective function. These programs can range from mathematical formulas to a Turing Complete mini-language, complete with storage and branching. I decided that LaTex would be a great option for visualizing the formulas produced by Encog GP in the Encog Workbench.

To do this I need the ability to render LaTex in Java. I do not require the ability to render the entire of LaTex in Java. I don’t need to typeset entire documents. I just need to be able to render a LaTex mathematical formula. After several Google searches I landed on JLatexMath. This package is open source, and used by a number of other open source projects. It is also available in a Maven Repository. You can find JLatexMath here.

http://forge.scilab.org/index.php/p/jlatexmath/

Unfortunately, documentation and examples are very scarce for JLatexMath. TO get started, I first created a simple example. Because examples seem to be in short supply for JLatexMath, I will present this example/tutorial in this article. You can find my current example in GitHub here. You can download a .ZIP at this location that contains everything you need.

https://github.com/jeffheaton/jlatexmath-example

The most important part is to make sure you include the JAR that contains JLatexMath. JLatexMath is completely contained in this JAR file. At the point I wrote this article, the JAR was named jlatexmath-0.9.6.jar.

You can see a screenshot of my example program here.

package com.jeffheaton.latex;

import java.awt.BorderLayout;
import java.awt.Color;
import java.awt.Container;
import java.awt.Graphics;
import java.awt.Graphics2D;
import java.awt.GridLayout;
import java.awt.Insets;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import java.awt.image.BufferedImage;

import javax.swing.JButton;
import javax.swing.JFrame;
import javax.swing.JLabel;
import javax.swing.JOptionPane;
import javax.swing.JPanel;
import javax.swing.JScrollPane;
import javax.swing.JTextArea;

import org.scilab.forge.jlatexmath.TeXConstants;
import org.scilab.forge.jlatexmath.TeXFormula;
import org.scilab.forge.jlatexmath.TeXIcon;

public class LatexExample extends JFrame implements ActionListener {
	
	private JTextArea latexSource;
	private JButton btnRender;
	private JPanel drawingArea;

	public LatexExample() {
		this.setTitle("JLatexMath Example");
		this.setSize(500, 500);
		Container content = this.getContentPane();
		content.setLayout(new GridLayout(2, 1));
		this.latexSource = new JTextArea();
		
		JPanel editorArea = new JPanel();
		editorArea.setLayout(new BorderLayout());
		editorArea.add(new JScrollPane(this.latexSource),BorderLayout.CENTER);
		editorArea.add(btnRender = new JButton("Render"),BorderLayout.SOUTH);		
		
		content.add(editorArea);
		content.add(this.drawingArea = new JPanel());		
		this.btnRender.addActionListener(this);
		
		this.latexSource.setText("x=\\frac{-b \\pm \\sqrt {b^2-4ac}}{2a}");
	}

	public void render() {
		try {
			// get the text
			String latex = this.latexSource.getText();
			
			// create a formula
			TeXFormula formula = new TeXFormula(latex);
			
			// render the formla to an icon of the same size as the formula.
			TeXIcon icon = formula
					.createTeXIcon(TeXConstants.STYLE_DISPLAY, 20);
			
			// insert a border 
			icon.setInsets(new Insets(5, 5, 5, 5));

			// now create an actual image of the rendered equation
			BufferedImage image = new BufferedImage(icon.getIconWidth(),
					icon.getIconHeight(), BufferedImage.TYPE_INT_ARGB);
			Graphics2D g2 = image.createGraphics();
			g2.setColor(Color.white);
			g2.fillRect(0, 0, icon.getIconWidth(), icon.getIconHeight());
			JLabel jl = new JLabel();
			jl.setForeground(new Color(0, 0, 0));
			icon.paintIcon(jl, g2, 0, 0);
			// at this point the image is created, you could also save it with ImageIO
			
			// now draw it to the screen			
			Graphics g = this.drawingArea.getGraphics();
			g.drawImage(image,0,0,null);
		} catch (Exception ex) {
			ex.printStackTrace();
			JOptionPane.showMessageDialog(null, ex.getMessage(), "Error",
					JOptionPane.INFORMATION_MESSAGE);		
		}

	}

	public static void main(String[] args) {
		LatexExample frame = new LatexExample();		
		frame.setVisible(true);
	}

	@Override
	public void actionPerformed(ActionEvent e) {
		if( e.getSource()==this.btnRender ) {
			render();
		}
		
	}
}

Most of the work is being done in the render method. There, I obtain the LaTex formula "source code" and render it to an image. This image is then displayed to the JFrame that makes up the application's main window.