Multitouch Dev

What if Keyboards NEVER existed? Part 1 of Many.

2011-06-01T04:36:00.000-07:00

Wow! I have been busy, although it may not seem so, seeing as how I haven't updated this in forever, but really, I assure you, have been quite busy working away. So I love retro things, whether it is an arcade, comic, or even computers. I've spent countless hours working on low level Linux kernel development, hacking android, working on the x86 port of it and tuio drivers / different multi touch implementations on Linux, and worst of all, I've spent most of my time trying to write new or rework existing window managers and UI toolkits for multi touch.

Obviously we have some great multi touch frameworks out there. The big problem though, is that none of it is native. I could go on and on forever about everything there is out there but they all have one commonality, and one ultimate downfall: they are all based and running on top of a keyboard and mouse system.

By the way I just want to throw in there that I absolutely love where android and the tablet scene is headed, unfortunately, that and almost all tablet technologies are based around single pointer emulation, i.e. your finger is the mouse click.

Now some of you may be saying, this guy has a really disorganized thought process and this article is all over the place, others will have noticed I’ve hit three main points so far. Let’s recap:

1. I love old things.

2. Everything is native keyboard / mouse (or mouse emulation) based

3. I love where tablet computing is headed

It is now time to tie those three things together.

What if the keyboard never existed? What if the mouse never happened? I got to thinking, if we had to start all over, how would we do it now? How would the operating system and how we interact with computers have developed? Not only that, what if they developed around the notion of directly interacting with the information and screen from day one? So that is what I have been doing. Some of you are going to be all like, way cool, and others like, that’s way dumb. I personally think it is great. So what am I doing exactly? Well, I’ve built a nice little 16 bit computer with LCD and touchscreen overlay, but I’m going to take it one step back further. I’m going to build an 8 bit native multi touch computer using the good ol’ z80. I’m actually not sure how well this will work, and if it will even have enough processing power to be able to drive the screen and do the gesture work in a time responsive fashion, but that is part of the fun and challenge of the 8 bit world, and even if it doesn’t succeed, at the very least it is going to force me to optimize and trim down much of the code.

On to the next part, I’m done with UI tool kits, specifically the realm of widgets, buttons etc. Done. That is the bane of modern computing and the biggest thing holding back multi touch application development as far as I am concerned. If you think about it, even the concept of the widget, the button, text box etc., is based around the concept of a single mouse click and the keyboard. So how much sense does it make that we are trying to do that with multi touch frameworks, that is make standard widget toolkits. It doesn’t make sense at all; they are two different realms of operations. Think of it like this. The current desktop is like dry land and the mode of transportation is a car with wheels, and the car is the widgets. The other realm is the tablet / multi touch scene and that is like water, and the mode of transportation ought to be a boat, but we are pulling in land technology, we are trying to use a car, trying to use widget based UI to navigate! This is obviously a no go. We need a boat.

Think about it like this as well: There is an infinite amount of information out there, and an infinite amount of ways to display it, and just as many ways to interact with it. So why are we going to limit ourselves to working with the data and displaying it with a single set of simple widget tools? What needs to happen instead is two parts; one part has already taken shape somewhat in the tablet scene. Apps. Apps? Yes, Apps, apps in the sense no two apps look or interact the same. That many apps are based around that specific data / function set, so in many ways, apps have already achieved widget independence for the most part, but not on the desktop, I’m talking desktop power, commercial grade application, power user apps. Not novelty checkers and Sudoku / tide table apps. The second part being that create an easy way for a person to PAINT, or render whatever kind of UI, basically custom widget for that application in that context, whatever works for the BEST interaction with the data. Think early WPF, but don’t think of all the problems with it. If programmers can easily and quickly render whatever they want and interact with what they renders, how they want, then you don’t need a widget framework / toolkit, and aren’t bound by its ridiculous constraints.

Anyways, I’d like to think some of you can see what I’m trying to get at and where this post is headed. This is just one of many posts in a series I will be doing on this over the couple weeks. Don’t expect to see me posting much code or hardware specs because, one it isn’t done, and two 99% of people would find it useless and or confusing and it would be defeating the purpose of this multi-part series, which is for me to explain my idea and thought process behind this course of development.

As always, I love feedback and constructive criticism etc. and most kinds of reader input.

Expanded Thought Map

2010-11-08T19:19:00.001-08:00

Here is an expanded thought map, as you can see I've added some pages of documentation to a few nodes, I'm going to put up the interactive map today when I get off work, which will let you explore it more.

Why a hardware standard?

2010-11-06T18:52:00.000-07:00

I had a comment and a couple emails asking as to what the point of standards are, especially the point of a hardware standard, let us take a short moment to explore this, hopefully after you read this post the entirety of reason will become self evident.

If you take the time to look, everything has standards, the lcd screen in front of you, the hdmi cable connecting your xbox and tv, the cd you put in your drive, everything has a standard, standards promote quality and insure interoperability between components. Lets take a look at how standards between the three branches feed themselves:

Software Standard:

The idea is to creat the worlds first native multitouch operating system based on the linux kernel. The operating system will run and work properly on all certfied hardware.

App Dev Platform:

The most successful and capable software is properitary, adobe, autocad, maya, etc. This is because sales gives the funding to develop them. The applications in turn cause the operating system to be successful. A successful operating system promotes the following of a hardware standard.

Hardware Standard:

The idea is that in the future there will not just be one manufacturer of multitouch tables / boxes, just as there is not one manufacture of mobile devices / touch screens / pc's. By creating a set of standards, if manufacters comply with them then they will be gaurnteed to work with the software. As hardware is purchased it provides funds to further develop the technology and promte growth.

As you can see standards are a good thing, each branch feeds back into the next fostering growth and development.

Thought Map

2010-11-06T05:33:00.000-07:00

Here is my thought map for Multitouch Specification Project. I will add to this and continue to develop it. This is pretty much the bare, bare bones skeleton of it. As you can see it it is split up into the 3 main branches I discussed in my last post plus a new administrative branch I will develop to see oversight of all three primary branches.

I used vym to make this, you should be able to find it in most distro's software repositories.

If you want to add onto the thought map or suggest changes let me know, better yet, download it from this link, and email me a copy of your changes.

Update? A BIG UPDATE.

2010-10-26T04:41:00.000-07:00

Alright so I have not been up keeping with this blog as well as I would have liked. I am glad though that it seems to have served many people very well! I have received a couple comments and many private messages requesting the live CD's. Here is what is going on with that, and how it ties into everything else:

1. By the time I finalized a live CD that I determined was bug free and ran on all the hardware / virtual hardware that I could get my hands on Ubuntu 10.10 came out.

2. Ubuntu 10.10 broke a few dependencies and other things and is largely untested.

3. Ubuntu itself is a problem in regards to the long term efforts of my project.

4. What is my project?

5. Linux in general on the desktop.

Alright, you can read lots of articles about this, Linux for the most part is awesome when it comes to the server environment, and even with modern desktop distros such as 10.10 Ubuntu, which in my opinion right out of the box is very comparable to even a mac, the desktop Linux has largely failed due to its lack of applications. As good and amazing as some open source software is the simple fact is nothing besides GIMP has come even close to its commercial counterpart, and debatable GIMP still can't touch a well trained Photoshop user. I realize I will probably get a lot of hate mail for this, but it is the truth. Unless you have the millions of dollars , time, and resources to develop something like auto-cad inventor, or Maya, it will never compare.

Back to the point, even with amazing as Ubuntu 10.10 is out of the box, it will and the rest of desktop Linux is failing because of application support. This got me thinking, open source is a great thing for an Operating System but is for the most part counterproductive to software. Also open source doesn't necessarily have to be free of charge either.

Where am I going with this? I think it is fairly self evident that eventually the keyboard and mouse will disappear. The obvious replacement is touch screen which has in the land of mobile computing, largely if not entirely replaced the keypad, just look at iPhone's and Android's. Eventually touch, and naturally multi touch will take over the desktop interface.

As we all know there is really no standard what so ever when it comes to multi touch. There is not even a hardware standard. There is also no real application base out there either. Linux actually has a chance for a massive market gain here. To be the pioneer where others are not yet reaching. All of the building blocks are here they just need to be put together. One must choose an operating system to build these future applications on. I'm not going to get into the details of this one but Linux is the obvious choice here. As far as the hardware is concerned, if you are reading this I am sure you are all to familiar with LLP, FITR, DSI, and every other derivative out there.

I feel like I am getting lost in the sauce here and just rambling, okay to collect all of this into a sensible post:

Here is what I want to do. I want to develop an open hardware solution, a Linux headless TUIO server which will pretty much be an embedded device with the camera attached directly to it; and a Linux distribution with an OpenGL interface with standard application development platform, think Visual Studio and WPF for all Windows developers reading this.

I want to build a team of people that will work on all three of these aspects.

The reason for the open hardware solution, there needs to be a standard for multi touch hardware, so pretty much we will have to pick a technology and stick with it. If you don't understand why there needs to be a standard, I don't even know what to say about that...

The distribution, this is kinda a toughie. If you think about it no current operating system in existence is really suitable for multi touch development. All current operating systems are based around keyboard and mouse input. If Linux were to drop out to the command line and not start the x server on a multi touch table that has no keyboard and mouse it would be pretty hard to recover the system. And no, we are not going to making a new operating system. There are ways for us to work around this problem. As far as basing this new distribution off of a current one, there is not much point, the only reason for that would be package, dependency and application support, but seeing as how all these applications and even the GUI is based around a mouse and keyboard input it is completely useless the multi touch community. All we really need is a bare bones Linux distro with XServer on it and OpenGL support. Remember that a large portion of desktop users spend their time on the internet so the distribution will have to pull all the required pieces to support things such as flash player etc. Other than that, it will be pretty much a blank slate for us to roll with. We need the WPF / Visual Studio style standard because the new technology needs an amazing development platform if companies will ever develop applications for it.

So hardware is never free, but if we get the standard down, we can make a table / slate style design that can be mass manufactured, something that people can pick up at a best buy.

Linux based and open source operating system, free of charge installed on the hardware.

Standardized development platform for companies to develop software for. Software that people will pay for.

I have seen a lot of something in the multi touch community that I really do not like. So much time and argument is being put into creating a gui that is best for or which one is the most correct for multi touch. Something that I have learned from my time in the military is that a timely decision is better than a decision that took a very long time but is more "correct". In the military it costs people their lives. The same principle applies here. A timely decision must be made. That's why who ever joins me on this project, we will come up with a set of standards and stick with it. People will use whatever you give them. They will make it work, and things evolve towards the better over time. Look at windows 95 compared to windows 7. I imagine our first distribution will be more like windows 95 than windows 7. If we keep debating and "researching" critical time will be lost and some one else will come in and take the multi touch reigns.

As many of you can see I am proposing a sort of hybrid thing here. Open standards and an open source operating system, but something that will have the support and backing of privatization.

If you want to join this team, and project send me a comment, email or message, point is contact me some how and let me know. I look forward to hearing from you all some more. and hope you enjoyed my latest post.

Live CD's

2010-09-14T18:27:00.000-07:00

Good news everyone, I know I promised live cd's, and I am running a bit behind, but as I mentioned earlier I can't find a cost effective solution for hosting multi-gig iso files. As a result I will be offering live cd's via the mail, I plan on being able to offer them hopefully within the next week.

Another thing I would like to mention, I turned ccv into a background process and turned the gui into a webpage. This would allow you to run ccv on a bare-bones pc and simply connect it to the multitouch pc via Ethernet. You can then control the bare-bones via the web page dished out by apache. I will upload screen shots around the same time I offer the live cd's. It is still a little buggy, especially in the regards of streaming the video, but I really like the idea of a independent tuio server that is platform independent. This tuio server is currently the main focus of my project and I will be putting off the open gl multitouch gui tutorial off for at least another month as there are a few more bugs I need to work out before I offer that as a tutorial.

Back Again!

2010-08-31T09:29:00.000-07:00

All right so I have been away for a couple weeks now but I am back!

I see a couple of you have had success and some are having problems. I am going to look at the problems and see if I can't resolve them.

Also here is what is coming up soon:

1. I am going to remaster this into a live-cd so that you can just download the iso and have a working fresh install of ubuntu 10.04 plus CCV.

2. The next tutorial will follow my development of an openGL based windowing system.

This windowing system will incorporate a tuio listner so that everything will be multitouch, even the login after you press the power button on your computer. :)

STEP 6: Compiling CCV on Ubuntu 10.04

2010-08-21T03:06:00.000-07:00

Now we are going to compile CCV for Ubuntu 10.04.

Make sure you DO NOT start here. You need to start at the beginning of the blog and follow from there, if you try to do this tutorial BEFORE this one, your ps3 eye will NOT work with CCV.

The following is a modified tutorial based off of x29a's post on the nui group forums.

Enter the following commands:

sudo apt-get install subversion
sudo su
cd /opt
svn co http://nuicode.svnrepository.com/svn/tbeta/trunk/tbeta/Linux/ ccv-Linux
cd /opt/ccv-Linux/scripts/ubuntu
chmod +x *.sh
./install_codeblocks.sh
./install_dependencies.sh
apt-get install libpoco-dev
apt-get install gpp
cd /opt/ccv-Linux/
cd apps/addonsExamples/Codeblocks_8_linux/

Now Download the following zip file. It contains the files I modified so that it will build on ubuntu 10.04: http://mt-dev.googlecode.com/files/codeblocks.tar

Unzip it to your Documents folder.

Back in your terminal enter the following command:

nautilus

Navigate to /opt/ccv-Linux/apps/addonsExamples/Codeblocks_8_linux
and delete the following files:

Community Core Vision.cbp
Community Core Vision.depend
Community Core Vision.layout
Makefile

Now navigate to where you unziped my file. Select and cut the 4 files and paste them in the /opt/ccv-Linux/apps/addonsExamples/Codeblocks_8_linux folder.

Close Nautilus. Back in the terminal enter the following commands:

cd /opt/ccv-Linux/libs/fmodex/lib
rm libfmodexp.so libfmodex.so
ln -s libfmodex-4.22.00.so libfmodex.so
ln -s libfmodexp-4.22.00.so libfmodexp.so

cd /opt/ccv-Linux/apps/addonsExamples/Codeblocks_8_linux/bin/libs
rm libfmodexp.so libfmodex.so
ln -s libfmodex-4.22.00.so libfmodex.so
ln -s libfmodexp-4.22.00.so libfmodexp.so

cd /opt/ccv-Linux/
codeblocks apps/addonsExamples/Codeblocks_8_linux/Community\ Core\ Vision.cbp

Now in codeblocks click Build --> Build.
You may get some errors, just click OK and ignore them.
CCV should now have successfully built. Close Codeblocks and go back to your terminal and enter the following commands:

cd /opt/ccv-Linux/apps/addonsExamples/Codeblocks_8_linux/bin
gedit start.sh

In gedit copy and paste the following:

#!/bin/sh
echo $(pwd)
export LD_LIBRARY_PATH=$(pwd)/libs/
./Community_Core_Vision

Click File --> Save. Now close gedit. Now enter the following commands into your terminal:

chmod +x start.sh
cd data
gedit config.xml

In the Camera settings change UseCamera to 1. Click File --> Save. Close gedit.

To Start CCV enter the following commands in your terminal:

cd /opt/ccv-Linux/apps/addonsExamples/Codeblocks_8_linux/bin
./start.sh

Congrats! You are now running CCV with the ps3 eye! if the camera feed is not from your ps3 eye but from another camera on your computer don't fear! Go back to the terminal in which your ran /start.sh and scroll up till you see the list of video capture devices, take note of which device your ps3 eye is and change the device number in config.xml to that and restart CCV with start.sh! Thats it! Hope this helps all those out there who want to develop MT on linux with CCV and a ps3 eye. Please comment if my instructions don't work anywhere in this blog.

STEP 5: Testing the ps3 eye with VLC.

2010-08-21T02:33:00.000-07:00

It is time to test out our ps3 eye!

First off I want to say that I am using a wonderfully modified ps3 eye made by Nolan at Peau Productions. I have a 780nm band pass filter, which is why my vlc stream will appear a very dark orangish / red.

Click on Applications --> Sound & Video --> VLC Media Player

You should see the following:

Now on VLC, click on Media --> Open Capture Device

If your computer, like my laptop has a built in video camera, your ps3 eye will not be /dev/video0 it will be /dev/video1 assuming your ps3 eye is the only other webcam you have hooked up to your computer.

Under the video device name put in your ps3 eye. In my case it looks like this since my laptop has a built in webcam:

Press play and see your ps3 eye working!

STEP 4: Getting the ps3 eye to work!

2010-08-21T01:23:00.000-07:00

The following is based off the tutorial from kaswy's page.

I have however made a couple small changes that are required. Open up your terminal and enter the following commands.

sudo -i
apt-get install kernel-package libncurses5-dev wget bzip2 build-essential
cd /usr/src
apt-get install linux-source

Now type in:

ls

You will now get a list of everything in the directory. Look for the .tar.bz2 file.
<your version> is the number that is between the 2.6. and the .tar.bz2. in my case the file is: linux-source-2.6.32.tar.bz2 so my <your version> is 32

Now type in the following command, but replace the <your version> with the proper number.

tar --bzip2 -xvf linux-source-2.6.<your version>.tar.bz2
ln -s linux-source-2.6.<your version> linux
cd linux

In my case it would look like this:

tar --bzip2 -xvf linux-source-2.6.32.tar.bz2
ln -s linux-source-2.6.32 linux
cd linux

Continue with these commands:

wget http://kaswy.free.fr/sites/default/files/download/ps3eye/0.5/ps3eyeMT-2.6.31-10-generic.patch
patch -p1 < ps3eyeMT-2.6.31-10-generic.patch
make oldconfig
cp /usr/src/linux-headers-$(uname -r)/Module.symvers /usr/src/linux
make modules_prepare
make SUBDIRS=drivers/media/video/gspca modules
cp drivers/media/video/gspca/gspca_ov534.ko /lib/modules/$(uname -r)/kernel/drivers/media/video/gspca/
depmod
modinfo gspca-ov534

You should see the following after entering the last command:

root@mt-dev-laptop:/usr/src/linux# modinfo gspca-ov534
filename: /lib/modules/2.6.32-24-generic/kernel/drivers/media/video/gspca/gspca_ov534.ko
license: GPL
description: GSPCA/OV534 USB Camera Driver(kaswy mod for MT use V0.5)
author: Antonio Ospite
srcversion: 2153E3BE52298A39177E46C
alias: usb:v1415p2000d*dc*dsc*dp*ic*isc*ip*
depends: gspca_main
vermagic: 2.6.32.15+drm33.5 SMP mod_unload modversions 586
parm: videomode: = xx //Set the videomode(see doc) (int)
parm: autogain: = [0|1] //Autogain (agc,aec,awb) (bool)
parm: gain: = [0..63] //main gain (int)
parm: exposure: = [0..255] //Exposure (int)
parm: redblc: = [0..255] //Red Balance (int)
parm: blueblc: = [0..255] //Blue Balance (int)
parm: sharpness: = [0..63] //Sharpness (int)
parm: vflip: = [0|1] //Vertical flip (bool)
parm: hflip: = [0|1] //Horizontal mirror (bool)
root@mt-dev-laptop:/usr/src/linux#

The important part is the description where it has v0.5 and kaswy's name.

Go ahead and enter the following commands:

modprobe -r gspca-ov534
modprobe gspca-ov534 videomode=01 autogain=0 exposure=200 hflip=1

Now exit back to the normal account by typing:

exit

This completes the first step to getting the ps3eye working properly.
On to modifying and installing unicap

Enter the following commands:

cd
mkdir unicap
cd unicap
wget http://unicap-imaging.org/downloads/unicap-0.9.5.tar.gz
tar -xvzf unicap-0.9.5.tar.gz
wget http://kaswy.free.fr/sites/default/files/download/ps3eye/unicap/unicap-gspca.patch
patch -p0 < unicap-gspca.patch
cd unicap-0.9.5
./configure
make
sudo make install

Unicap should now be working.

STEP 3: Getting VLC.

2010-08-21T00:57:00.000-07:00

The next thing we need to do before anything else is get VLC. Vlc is a very versatile media player.

Click on Applications --> Accessories --> Terminal

now enter the following command:

sudo apt-get install vlc

STEP 2: Install Ubuntu and Start Up.

2010-08-21T00:51:00.000-07:00

So now that you have your Ubuntu cd, throw it in your disk drive and install it. Be careful though! If you are going to install this on a computer that already has another operating system on it you can easily erase EVERYTHING including the operating system itself.

Follow the instructions on the screen. I decided to create a user name of mt-dev for this instillation.

The computer will restart when the instillation is complete. Go ahead and log in. Connect yourself to the internet.

NOTE: This is the most important step in everything we are going to do: The very first time you log in, do not install any new programs, do not install any updates if the package manager asks you to. DO NOT INSTALL ANYTHING, unless it is in this blog, in the right order.

STEP 1: Get Ubuntu.

2010-08-20T20:13:00.000-07:00

The first step is to get the most current copy of Ubuntu. The reason I have chosen Ubuntu is because it is very user friendly and now days one of the most supported distro's out there.

We will be working with version 10.04.

You can get the latest version here:

http://www.ubuntu.com/desktop/get-ubuntu/download

An important note: Get the 32 bit version. I have not yet attempted this with 64 bits. That will come in the future.

Once you have the .iso file downloaded, burn it to a disk. If you don't know how to do this, follow step 2 on :

http://www.ubuntu.com/desktop/get-ubuntu/download

-----

For those of you like myself who prefer to download large files as a torrent, you can get the torrent here:

http://releases.ubuntu.com/10.04.1/ubuntu-10.04.1-desktop-i386.iso.torrent

-----

Now I am assuming that since you are interested in developing MT Apps and Systems for Linux, you have some kind of Linux background. This is the only thing I am going to assume through out this blog. If you can't get a fresh install of Ubuntu 10.04 on your system than there is no hope for you. Also, I am not responsible for any data loss or damages when you try to install a fresh copy of Ubuntu on your computer.

Purpose of this blog.

2010-08-20T20:04:00.000-07:00

I've been working with multi touch systems for a while now. First off I would like to say I am a strong supporter of the open source community and believe that being open helps resolve issues and drive development. Development would be the key word here. The people at nui group have a great thing going, but I feel like the development on the Linux platform has been severely hindered and is falling behind. I would like to see effort being put into developing multitouch applications and not just people trying to spend hours and hours just getting CCV or the ps3 eye to work, and then to just simply give up out of frustration. So, I will start this blog off with how to get CCV and your ps3eye working, and CCV and the ps3eye working together on Linux. Here is where things are different from the other tutorials available on the forums, I will be documenting how to do this from a fresh install of Ubuntu.