Most of the time what I'm working on when I need a proxy has something to do with SOAP and there is actually a MITM proxy in SoapUI but it doesn't have support for HTTPS. If you are working with SOAP and can use HTTP then SoapUI is a great tool. If that doesn't fit the bill then Paros proxy does a good job.

Paros proxy is pretty easy to set up, just download the binary distribution and run it. The only thing I wanted to change was the port that Paros listened on and that can be done by going to the Tools menu option then Options and then selecting Local proxy:

From the description of this option you can see the Paros listens for both HTTP and HTTPS on the same port. Tell Java where to find the proxy server either on the command line with the following system settings:

-Dhttp.proxyHost=localhost -Dhttp.proxyPort=8081 -Dhttps.proxyHost=localhost -Dhttps.proxyPort=8081

or in code using the System.setProperty call:

    System.setProperty("http.proxyHost", "localhost");
    System.setProperty("http.proxyPort", "8081");
    System.setProperty("https.proxyHost", "localhost");
    System.setProperty("https.proxyPort", "8081");

At this point if you try to use the proxy with HTTPS you are going to get an error because the proxy doesn't have the correct certificate or a matching hostname for the proxied service. Luckily you can tell Java to ignore all SSL certificate problems and any hostname miss-matches with the following code:

// The following tells Java to ignore certificate problems
TrustManager[] trustAllCerts = new TrustManager[]
{
  new X509TrustManager()
  {
    public java.security.cert.X509Certificate[] getAcceptedIssuers()
    {
      return null;
    }

    public void checkClientTrusted(java.security.cert.X509Certificate[] certs, String authType)
    {
    }

    public void checkServerTrusted(java.security.cert.X509Certificate[] certs, String authType)
    {
    }
  }
};

try
{
  SSLContext context = SSLContext.getInstance("TLS");
  context.init(null, trustAllCerts, new java.security.SecureRandom());
  HttpsURLConnection.setDefaultSSLSocketFactory(context.getSocketFactory());
}
catch (Exception e)
{
}

// This tells Java to not worry about hostnames matching
HttpsURLConnection.setDefaultHostnameVerifier(new HostnameVerifier()
{
  public boolean verify(String string, SSLSession ssls)
  {
    return true;
  }
});

Make sure the above block of code is executed before any SOAP calls need to be made.

Now if you start Paros proxy and run your application you should see the SOAP requests in their raw form. If you are debugging SOAP messages these raw requests can be exported to text files and then loaded into SoapUI for further processing if need be.

Some of what follows is exactly the same as my post about installing CentOS 5.5 on EC2 with the stock kernel. Once you get the hang of it you can install just about anything Linux based to EC2. I've broken this post into two parts to try and separate the generic transfer information from the specifics of an example. The first part that goes over the basics of what needs to be done to transfer any VirtualBox or VMWare box to EC2. The second part is an example of transferring an Ubuntu Server install to from VirtualBox to EC2.

The following steps should work for any Linux OS but the main sticking point is that the kernel needs to be compatible with EC2 (see the anouncment for a list of some distros that have compatible kernels and remember you can always compile by hand as well).

1. Make sure your VirtualBox or VMWare install has a kernel that is able to boot on EC2.
2. Make sure you have grub installed and set up your /boot/grub/menu.lst file to point to the correct Xen kernel and initrd file. See the Ubuntu instructions that follow for an example menu.lst.
3. Make sure you have your /boot directory in the correct place. If your disk is just one big partition you shouldn't need to do anything (use the hd0 PV-Grub kernel in the last step when booting). If you have a partition for /boot you will need to make sure it is the first partition on the disk and you will also need to copy the directory into /boot/boot so that the menu.lst file would be in /boot/boot/grub/menu.lst after the copy. For an example of this see the Ubuntu instructions that follow (use the hd00 PV-Grub kernel in the last step when booting).
4. Export the image to a raw form. This includes the partition table and everything. For VMWare you can use qemu-img to convert the VMDK to a raw image with the following command:

      qemu-img convert -O raw vmware-image.vmdk myosimage.raw
   

   Or if you are using VirtualBox you would use the VBoxManage command (I'm currently using VirtualBox 3.2.6 and since the VBoxManage command has changed before that might be important for those reading this at a later date):

      VBoxManage internalcommands converttoraw myosimage.vdi myosimage.img
   

   In both cases you should be able to run fdisk against the resulting raw image file and see a partition table:

      fdisk -lu myosimage.img
   

5. Create an EBS volume that is the same size or larger than the raw disk image created in step 4 and then attach it to a running EC2 instance:

   ec2-create-volume -z us-east-1a -s 2
   ec2-attach-volume volume-id -i instance-id -d /dev/sdh
   

   Note: The instance you attach the volume to is just a place used to copy the image file onto the volume. There is no need to be too picky about the type of instance it is as long as you have access to the dd command that should be all you need.

6. Transfer the exported VirtualBox or VMWare disk image to the running EC2 instance created in step 5 and then copy it to the volume with the following dd command (this example assumes the volume is attached to the /dev/sdh device):

   dd if=myosimage.img of=/dev/sdh bs=10M
   

   Note: It may make sense to compress your raw image before transferring it since any free space will be represented in the raw file and will compress down a lot.

7. Make a snapshot of the volume:

   ec2-create-snapshot -d "Volume Description" volume-id
   

8. Register the snapshot as a new AMI:

   ec2-register -n "AMIName" -d "AMI Description" --root-device-name /dev/sda2 -b /dev/sda=snap-id:2:true
   

9. Boot the new AMI using the correct PV-Grub kernel. The correct kernel will depend on how your partition structure. For more information see the AWS PV-Grub documentation. The main thing to know is that if you have /boot on its own partition use the hd00 kernel otherwise use the hd0 kernel:

   ec2-run-instances -z us-east-1a -g your-group -k your-keypair -n 1 --kernel pv-grub-kernel-id ami-from-step-19
   

The following is a specific example of converting an existing Ubuntu 10.04 LTS Server installed on VirtualBox to a bootable EC2 AMI. Note that steps 1 to 4 of the following create my "existing" Ubuntu Server so skip those if you have something that already exists. The last portion of this will look just like the above generic steps:

1. Use the Ubuntu Server ISO to install Ubuntu on a fresh VirtualBox VM. For this example I used 512M of memory and a 2G disk. This is what my VirtualBox setup looked like:
   
   

   

2. Because I wanted /boot to be on its own partition I used the manual partition creation option:
   
   

   

   You don't have to do this but I wanted these instructions to align with my previous post for CentOS 5.5. In this case I made the first partition mount at /boot and the second partition was everything else mounted at / and the result looked like:
   

   

   Notice that I didn't include any swap, that is something you probably want to put on the EC2 instance's ephemeral storage.

3. For Ubuntu you are prompted to install other software after the base install and setting up a user. Make sure you install OpenSSH server:
   
   

   

4. When prompted to install grub answer yes.
5. At this point you should have a bootable Ubuntu system either from following the previous steps or from an existing bootable install. You will need to install the libuuid-perl package so that the EC2 compatible Xen kernels will be installable:

      sudo apt-get install libuuid-perl
   

6. Grab a Xen kernel and initrd that are compatible with EC2:

   wget http://ftp.debian.org/debian/pool/main/l/linux-2.6/linux-base_2.6.32-20_all.deb
   wget http://ftp.debian.org/debian/pool/main/l/linux-2.6/linux-image-2.6.32-5-xen-686_2.6.32-20_i386.deb
   
   sudo dpkg -i linux-base_2.6.32-20_all.deb linux-image-2.6.32-5-xen-686_2.6.32-20_i386.deb
   

   Notes: I was unable to find a compatible kernel that was apt-get installable from Ubuntu and that is why I grab the two above from the Debian site. Also, If you want to continue to boot the system outside of EC2 then make sure to edit the /boot/grub/grub.cfg file so that it has the correct kernel set to boot by default.

7. Create a menu.lst file that points to the correct kernel and initrd that were just installed:

      sudo vi /boot/grub/menu.lst
   

   For this example mine contains:

   default 0
   timeout 1
   title UBEC2
           root (hd0,0)
           kernel /boot/vmlinuz-2.6.32-5-xen-686 root=/dev/xvda2
           initrd /boot/initrd.img-2.6.32-5-xen-686
   

   Notes: The grub root command is pointing to hard disk 0 and partition 0 because I put the boot directory on the first partition. The root parameter for the kernel is pointing to /dev/xvda2 and not /dev/sda2 because the kernel patch to disable XSAVE changes the names of the devices.

8. Copy everything from /boot into /boot/boot, this is just the easiest way of making things work since EC2 PV-Grub looks for the menu.lst file /boot/boot/grub/ when you specify the hd00 kernel:

   sudo cp -Rp /boot/ /boot/boot
   

9. Shut the VirtualBox system down and extract the hard drive image with the following command:

   VBoxManage internalcommands converttoraw ~/.VirtualBox/HardDisks/YourHardDiskName.vdi /tmp/myosimage.img
   

   Notes: The raw image that results will be the full size of the disk as it was seen by VirtualBox so if you have a lot of free disk space on your image you will probably want to compress it before you transfer it to EC2. At this point the commands are exactly as they are for the generic instructions above.

10. Start a temporary EC2 instance and transfer your image:

    ec2-run-instances -z us-east-1a -g your-group -k your-keypair -n 1 ami-84db39ed
    

11. Create an EBS volume of the correct size to put your image onto and attach it to the previously created temporary EC2 instance:

    ec2-create-volume -z us-east-1a -s 2
    ec2-attach-volume volume-id -i instance-id -d /dev/sdh
    

12. Transfer the image to the volume:

    dd if=myosimage.img of=/dev/sdh bs=10M
    

13. Create a snapshot of the volume:

    ec2-create-snapshot -d "Volume Description" volume-id
    

14. Register the snapshot as an AMI:

    ec2-register -n "AMIName" -d "AMI Description" --root-device-name /dev/sda2 -b /dev/sda=snap-id:2:true
    

15. Boot the AMI:

    ec2-run-instances -z us-east-1a -g your-group -k your-keypair -n 1 --kernel pv-grub-kernel-id ami-from-step-19
    

    Note: Because the partition table contains /boot on hd00 I used the kernel named aki-4c7d9525 for the east region.

Once you get the hang of transferring the image these steps aren't as complicated as they seem. It may help to imagine the extracted disk image as a physical hard drive that you are moving around. The main pain points are selecting the correct PV-Grub kernel and making sure your grub menu.lst file is in the correct location on the partition.

For those who might just be interested in booting a custom kernel using EC2 pv-grub I will try to produce a few more posts with more details on that but for now here are the main things to know:

• The pv-grup kernels named with hd00 will look on the first partition of the registered device in the /boot/boot/grub directory for a menu.lst file. Use this type of kernel if you create want to use a partitioned disk.
• The pv-grup kernels named with hd0 will look on the registered device in the /boot/grub directory for a menu.lst file. Use this type of kernel if you don't have a partition on your disk.
• You won't get anything meaningful back from the boot attempt if your grub menu.lst file is in the wrong place or is not valid. See the end of the post for what a pv-grub error message looks like and some tips on what to do if you see it.
• The kernel you use does matter but the current mainline Linux kernel (2.6.35) contains everything you need except for a small change to turn off XSAVE. The main thing to know is that not every distribution may have made the change needed to work on EC2.
• I have tried non-Linux kernels to no avail. See the end of the post for a little more information.

A lot of what follows is similar, both steps and concepts, to the "from scratch" section of my post on Fedora 12 on EC2 using a root EBS. I've also bundled all the instance building commands up into one script (centos5.5.sh). If you want to use that script then do 1 and 2 of what follows, make sure to change the password used for root in the script and then pick back up at 18. The following steps should not be taken as the only way to do this but more of a recipe:

1. Start an EC2 instance that has yum on it to be used as a setup box. A RedHat based box, Fedora or CentOS will work best unless you want to install yum. For the following steps I used a Fedora 8 based EC2 node.

   ec2-run-instances -z us-east-1a -g your-group -k your-keypair -n 1 ami-84db39ed
   

2. Create a new EBS volume to install to and map it to the running instance from step 1. Your volume should be greater than 2G for a base install. I mapped this new volume to the /dev/sdh device on the setup machine so you will notice that in the following steps (if you are using the script you will want to make sure you map to /dev/sdh as well):

   ec2-create-volume -z us-east-1a -s 2
   ec2-attach-volume volume-id -i instance-id -d /dev/sdh
   

3. Create a partion table using fdisk on the volume you are going to install to.

   I created both a /boot and / partion on /dev/sdh1 and /dev/sdh2 respecivly. I also made the /dev/sdh1 partition active so it is exactly as it would be if it had been installed on a real machine.

   Note that this step is optional but I am going to include it because I think it makes for a more natural setup and is more in line with what you would get if you did a VirtualBox install and then transfered the image.

4. Format your partition(s) and mount them into /mnt. For me that was done with the following:

   echo "y" | mkfs.ext3 /dev/sdh1
   echo "y" | mkfs.ext3 /dev/sdh2
   mount /dev/sdh2 /mnt
   mkdir /mnt/boot
   mkdir /mnt/dev
   mkdir /mnt/proc
   mkdir /mnt/etc
   mount /dev/sdh1 /mnt/boot
   mount -t proc none /mnt/proc
   

5. Create a base device setup for the new instance:

   for i in console null zero ; do /sbin/MAKEDEV -d /mnt/dev -x $i ; done
   

6. Create a base fstab file in /mnt/etc/fstab. The following is the one I used:

   /dev/sda1               /boot                   ext3    defaults 1 1
   /dev/sda2               /                       ext3    defaults 1 2
   none                    /dev/pts                devpts  gid=5,mode=620 0 0
   none                    /dev/shm                tmpfs   defaults 0 0
   none                    /proc                   proc    defaults 0 0
   none                    /sys                    sysfs   defaults 0 0
   /dev/sdc1               /mnt                    ext3    defaults 0 0
   /dev/sdc2               swap                    swap    defaults 0 0
   

7. Create the yum repo configuration, prepare for the yum install and then install the base OS onto the new volume.
   
   

   The following is the yum configuration file I used:

   [main]
   cachedir=/var/cache/yum
   debuglevel=2
   logfile=/var/log/yum.log
   exclude=*-debuginfo
   gpgcheck=0
   obsoletes=1
   reposdir=/dev/null
   
   [os]
   name=CentOS 5.5 - i386 - OS
   mirrorlist=http://mirrorlist.centos.org/?release=5&arch=i386&repo=os
   enabled=1
   
   [updates]
   name=CentOS 5.5 - i386 - Updates
   mirrorlist=http://mirrorlist.centos.org/?release=5&arch=i386&repo=updates
   enabled=1
   

   The following command will install the base of Cent OS 5.5 into /mnt (note that I created the above config file as /tmp/yumec2.conf):

   yum -c /tmp/yumec2.conf --installroot=/mnt -y groupinstall Base
   

8. Install sshd, grub, the Cent OS Xen kernel and then clean the repo to free up disk space:

   yum -c /tmp/yumec2.conf --installroot=/mnt -y install openssh-server
   yum -c /tmp/yumec2.conf --installroot=/mnt -y install grub
   yum -c /tmp/yumec2.conf --installroot=/mnt -y install kernel-xen.i686
   
   yum -c /tmp/yumec2.conf --installroot=/mnt -y clean packages
   

9. Disable DNS checks and allow root to log in via SSH:

   echo "UseDNS no" >> /mnt/etc/ssh/sshd_config
   echo "PermitRootLogin yes" >> /mnt/etc/ssh/sshd_config
   

10. Set up networking by creating the /mnt/etc/sysconfig/network file. The contents for this example are:

    NETWORKING=yes
    

    As well as the /mnt/etc/sysconfig/network-scripts/ifcfg-eth0 file. The contents for this example are:

    DEVICE=eth0
    BOOTPROTO=dhcp
    ONBOOT=yes
    TYPE=Ethernet
    USERCTL=yes
    PEERDNS=yes
    IPV6INIT=no
    

11. I'm not sure if this is needed still but in the past there have been some /dev file missing on boot so I always add the following to the startup script to make sure they are available. The first two are the random devices and the last three are where the ephimeral drive is usually mapped:

    echo "/sbin/MAKEDEV /dev/urandom" >> /mnt/etc/rc.sysinit
    echo "/sbin/MAKEDEV /dev/random" >> /mnt/etc/rc.sysinit
    echo "/sbin/MAKEDEV /dev/sdc" >> /mnt/etc/rc.sysinit
    echo "/sbin/MAKEDEV /dev/sdc1" >> /mnt/etc/rc.sysinit
    echo "/sbin/MAKEDEV /dev/sdc2" >> /mnt/etc/rc.sysinit
    

12. Change the root password for the new instance. This is optional as you could create scripts to download your SSH key from EC2 but for these instructions setting the root password is the easiest:

    chroot /mnt
    pwconv
    passwd
    exit
    

13. Change the network settings so that the NetworkManager is off and network is on

    chroot /mnt chkconfig --level 2345 NetworkManager off
    chroot /mnt chkconfig --level 2345 network on
    

14. Disable a few things that are enabled by default but won't do any good for an EC2 instance:

    chroot /mnt chkconfig --level 2345 avahi-daemon off
    chroot /mnt chkconfig --level 2345 firstboot off
    

15. The stock CentOS Xen initrd doesn't load the Xen block or net drivers and those are required to boot. I unpacked the installed initrd and created a modified version by hand using the following commands (note that as soon as the CentOS Xen kernel version changes this will stop functioning):

    cp /mnt/boot/initrd-2.6.18-194.8.1.el5xen.img /mnt/boot/initrd-2.6.18-194.8.1.el5xen.img.orig
    mkdir /tmp/initrdextract
    cd /tmp/initrdextract
    gzip -dc /mnt/boot/initrd-2.6.18-194.8.1.el5xen.img | cpio -id
    cp /mnt/lib/modules/2.6.18-194.8.1.el5xen/kernel/drivers/xen/blkfront/xenblk.ko lib
    cp /mnt/lib/modules/2.6.18-194.8.1.el5xen/kernel/drivers/xen/netfront/xennet.ko lib
    chmod -x lib/xenblk.ko
    chmod -x lib/xennet.ko
    cat <<EOL > init
    #!/bin/nash
    
    mount -t proc /proc /proc
    setquiet
    echo Mounting proc filesystem
    echo Mounting sysfs filesystem
    mount -t sysfs /sys /sys
    echo Creating /dev
    mount -o mode=0755 -t tmpfs /dev /dev
    mkdir /dev/pts
    mount -t devpts -o gid=5,mode=620 /dev/pts /dev/pts
    mkdir /dev/shm
    mkdir /dev/mapper
    echo Creating initial device nodes
    mknod /dev/null c 1 3
    mknod /dev/zero c 1 5
    mknod /dev/urandom c 1 9
    mknod /dev/systty c 4 0
    mknod /dev/tty c 5 0
    mknod /dev/console c 5 1
    mknod /dev/ptmx c 5 2
    mknod /dev/rtc c 10 135
    mknod /dev/tty0 c 4 0
    mknod /dev/tty1 c 4 1
    mknod /dev/tty2 c 4 2
    mknod /dev/tty3 c 4 3
    mknod /dev/tty4 c 4 4
    mknod /dev/tty5 c 4 5
    mknod /dev/tty6 c 4 6
    mknod /dev/tty7 c 4 7
    mknod /dev/tty8 c 4 8
    mknod /dev/tty9 c 4 9
    mknod /dev/tty10 c 4 10
    mknod /dev/tty11 c 4 11
    mknod /dev/tty12 c 4 12
    mknod /dev/ttyS0 c 4 64
    mknod /dev/ttyS1 c 4 65
    mknod /dev/ttyS2 c 4 66
    mknod /dev/ttyS3 c 4 67
    echo Setting up hotplug.
    hotplug
    echo Creating block device nodes.
    mkblkdevs
    echo "Loading jbd.ko module"
    insmod /lib/jbd.ko
    echo "Loading ext3.ko module"
    insmod /lib/ext3.ko
    echo "Loading xenblk.ko module"
    insmod /lib/xenblk.ko
    echo "Loading xennet.ko module"
    insmod /lib/xennet.ko
    mkblkdevs
    echo Scanning and configuring dmraid supported devices
    resume /dev/sdc2
    echo Creating root device.
    mkrootdev -t ext3 -o defaults,ro /dev/sda1
    echo Mounting root filesystem.
    mount /sysroot
    echo Setting up other filesystems.
    setuproot
    echo Switching to new root and running init.
    switchroot
    EOL
    find ./ | cpio -H newc -o | gzip > /mnt/boot/initrd-2.6.18-194.8.1.el5xen.img
    cd -
    

16. Install grub on the new instance, move the boot directory into a subdirectory and create a grub menu.lst file that points to the CentOS kernel and initrd file:

    chroot /mnt grub-install /dev/sdh
    
    mkdir /mnt/boot/boot/
    mv /mnt/boot/* /mnt/boot/boot/ 2> /dev/null > /dev/null
    

    Put the following in /mnt/boot/boot/grub/menu.lst (note that as soon as the CentOS Xen kernel version changes this will be incorrect):

    default 0
    timeout 1
    title CentOS5.5
         root (hd0,0)
         kernel /boot/vmlinuz-2.6.18-194.8.1.el5xen root=/dev/sda2
         initrd /boot/initrd-2.6.18-194.8.1.el5xen.img
    

    Note that this goes in /mnt/boot/boot/grub and that isn't the normal spot you would expect it in. This is where the AWS EC2 pv-grub expects to find the file on the first partition and moving the boot directory around just keeps everything in line with those expectations.

17. Make sure everything is written to disk and unmount the volume. At this point you have a CentOS 5.5 install that is almost ready to boot.

    sync
    umount /mnt/proc
    umount /mnt/boot
    umount /mnt
    

18. Make a snapshot of the volume you just installed to, you will need to volume ID that came from step 2:

    ec2-create-snapshot -d "Volume Description" volume-id
    

19. Use the snapshot from step 18 along with the ec2-register command to register your instance:

    ec2-register -n "AMIName" -d "AMI Description" --root-device-name /dev/sda2 -b /dev/sda=snap-id:2:true
    

    There are a number of things to take note of with the above command:

    1. Running this command will result in output something like: IMAGE ami-a5ae9bb
    2. The -b option can now assign a snapshot to a block device, the options in this example tell EC2 to generate 2G of space for the snapshot and to delete the volume it creates from the snapshot if the instance terminates. If you plan to use an instance long term you should replace that true at the end with a false to keep EC2 from deleting the volume when the instance terminates.
    3. Notice that the -b option is assigning the snapshot to the device and not to a partition of the device, that is /dev/sda instead of /dev/sda1. You can still assign a snapshot directly to a partition but now you can also assign a block device to a raw partitioned disk. Because I created the partition table earlier the snapshot is the raw disk device here.
    4. Also note that we are missing the kernel configuration option. As of this post using it with a pv-grub kernel causes the register command to fail. It isn't a big issue but just keep that in mind when you fire the AMI up otherwise it won't boot with the correct pv-grub kernel.

    
    

20. Start an instance of the fresh CentOS 5.5 install. One key thing here is picking the correct pv-grub kernel to boot from. There are currently 4 different kernels at each location, see the Enabling User Provided Kernels in Amazon EC2 document for a full list of kernels in each availability zone. In this case because the root disk was created with a partition table I used the "ec2-public-images/pv-grub-hd00-V1.01-i386.gz.manifest.xml" kernel to boot with (on US-East-1 that is kernel id aki-4c7d9525). For example:

    ec2-run-instances -z us-east-1a -g your-group -k your-keypair -n 1 --kernel pv-grub-kernel-id ami-from-step-19
    

Tips on debugging the boot process

If your instance won't boot you can use the ec2-get-console-output command to get the console output created from the pv-grub boot process. If your console output ends up like the following there are a number of things you may have done wrong.

• You may have selected the wrong kernel and it is trying to boot from a non-existant partition. Make sure you are using the correct pv-grub kernel hd0 vs hd00.
• You forgot to install grub or installed grub in the wrong place. Make sure you have either /boot/grub/menu.lst or /boot/boot/grub/menu.lst
• You have a bad menu.lst file. One mistake I made was giving a boot item a title with a space in it. Make the menu.lst as simple as you can until you get it to boot.

    Xen Minimal OS!
  start_info: 0xb10000(VA)
    nr_pages: 0x6a400
  shared_inf: 0x002f9000(MA)
     pt_base: 0xb13000(VA)
nr_pt_frames: 0x9
    mfn_list: 0x967000(VA)
   mod_start: 0x0(VA)
     mod_len: 0
       flags: 0x0
    cmd_line:  root=/dev/sda1 ro 4
  stack:      0x946780-0x966780
MM: Init
      _text: 0x0(VA)
     _etext: 0x621f5(VA)
   _erodata: 0x76000(VA)
     _edata: 0x7b6d4(VA)
stack start: 0x946780(VA)
       _end: 0x966d34(VA)
  start_pfn: b1f
    max_pfn: 6a400
Mapping memory range 0xc00000 - 0x6a400000
setting 0x0-0x76000 readonly
skipped 0x1000
MM: Initialise page allocator for e6c000(e6c000)-0(6a400000)
MM: done
Demand map pfns at 6a401000-7a401000.
Heap resides at 7a402000-ba402000.
Initialising timer interface
Initialising console ... done.
gnttab_table mapped at 0x6a401000.
Initialising scheduler
Thread "Idle": pointer: 0x7a402008, stack: 0x6a030000
Initialising xenbus
Thread "xenstore": pointer: 0x7a402478, stack: 0x6a040000
Dummy main: start_info=0x966880
Thread "main": pointer: 0x7a4028e8, stack: 0x6a050000
"main" "root=/dev/sda1" "ro" "4"
vbd 2048 is hd0
******************* BLKFRONT for device/vbd/2048 **********

backend at /local/domain/0/backend/vbd/2111/2048
Failed to read /local/domain/0/backend/vbd/2111/2048/feature-barrier.
Failed to read /local/domain/0/backend/vbd/2111/2048/feature-flush-cache.
12582912 sectors of 0 bytes
**************************
vbd 2051 is hd1
******************* BLKFRONT for device/vbd/2051 **********

backend at /local/domain/0/backend/vbd/2111/2051
Failed to read /local/domain/0/backend/vbd/2111/2051/feature-barrier.
Failed to read /local/domain/0/backend/vbd/2111/2051/feature-flush-cache.
1835008 sectors of 0 bytes
**************************

    [H
    [J

    GNU GRUB  version 0.97  (1740800K lower / 0K upper memory)

       [ Minimal BASH-like line editing is supported.   For

         the   first   word,  TAB  lists  possible  command

         completions.  Anywhere else TAB lists the possible

         completions of a device/filename. ]

grubdom>
    [9;10H

Booting non-Linux OSes with EC2

I have attempted both FreeBSD and NetBSD in particular with no luck.

FreeBSD is tricky because it really wants to use its loader and while you can do that with the grub chainloader command it results in a grub error from EC2 about needing to load the kernel before booting:

root (hd0,1)

 Filesystem type unknown, partition type 0xa5

chainloader +1

Error 8: Kernel must be loaded before booting

Press any key to continue...

I was also able to try a modified version of FreeBSD that should boot without the loader but with that I get an error claiming the kernel isn't bziped:

root (hd0,1,a)

 Filesystem type is ufs2, partition type 0xa5

kernel /boot/loader

xc_dom_probe_bzimage_kernel: kernel is not a bzImage
ERROR Invalid kernel: xc_dom_find_loader: no loader found

xc_dom_core.c:523: panic: xc_dom_find_loader: no loader found
xc_dom_parse_image returned -1

Error 9: Unknown boot failure

Press any key to continue...

For NetBSD the result is actually a completely blank console log so I assume it causes some catastrophic failure that keeps the EC2 system from even being able to pull back a log.

Google has released source for the VP8 codec as the WebM project. The WebM project will be an open alternative in the HTML5 video tag codec space and being backed by Google, who will use it for YouTube, will give it a fighting chance. WebM is starting to be integrated into FFMpeg but there are still some patches that need to be applied. While there are some guides to how to build FFMpeg for Linux with VP8 WebM support I wanted to get it compiled for Windows.

First off this won't be done using a Windows install but instead using Virtual Box with a fresh install of Ubuntu 10.04 desktop. It is just faster and easier that way. So go ahead and grab Virtual Box and get Ubuntu installed on it before you continue.

I used this cross compiling X264 using Ubuntu as a base for part of my script. The entire script follows but if you want to make sure you get a good copy you can download it:

mkdir ffmpeg
cd ffmpeg

BASEDIR=`pwd`
MINGBASE=$BASEDIR/win32-webm

sudo apt-get install pkg-config yasm subversion cvs git-core gcc-mingw32

cat > mingw32-setup.sh << EOF
#!/bin/sh
export CC=i586-mingw32msvc-gcc
export CXX=i586-mingw32msvc-g++
export CPP=i586-mingw32msvc-cpp
export AR=i586-mingw32msvc-ar
export RANLIB=i586-mingw32msvc-ranlib
export ADD2LINE=i586-mingw32msvc-addr2line
export AS=i586-mingw32msvc-as
export LD=i586-mingw32msvc-ld
export NM=i586-mingw32msvc-nm
export STRIP=i586-mingw32msvc-strip

export PATH="/usr/i586-mingw32msvc/bin:$PATH"
export PKG_CONFIG_PATH="$MINGBASE/lib/pkgconfig/"
exec "\$@"
EOF
chmod +x mingw32-setup.sh

mkdir $MINGBASE $MINGBASE/src $MINGBASE/lib $MINGBASE/include $MINGBASE/share $MINGBASE/bin

cd $MINGBASE/src

wget -qO - http://downloads.xiph.org/releases/ogg/libogg-1.2.0.tar.gz | tar xzf -
cd libogg-1.2.0
$BASEDIR/mingw32-setup.sh ./configure --enable-static --disable-shared --host=mingw32
make
DESTDIR=$MINGBASE make install prefix=
cd ..

wget -qO - http://downloads.xiph.org/releases/vorbis/libvorbis-1.3.1.tar.gz | tar xzf -
cd libvorbis-1.3.1
LDFLAGS="-L$MINGBASE/lib" CFLAGS="-I$MINGBASE/include" $BASEDIR/mingw32-setup.sh ./configure --enable-static --disable-shared --host=mingw32
make
DESTDIR=$MINGBASE make install prefix=
cd ..

wget -qO - ftp://sourceware.org/pub/pthreads-win32/pthreads-w32-2-8-0-release.tar.gz | tar xzf -
cd pthreads-w32-2-8-0-release
make GC-static CROSS=i586-mingw32msvc-
cp libpthreadGC2.a $MINGBASE/lib
cp *.h $MINGBASE/include
cd ..

wget -qO - http://zlib.net/zlib-1.2.5.tar.gz | tar xzf -
cd zlib-1.2.5/
$BASEDIR/mingw32-setup.sh ./configure
sed -i"" -e 's/-lc//' Makefile
make
DESTDIR=$MINGBASE make install prefix=
cd ..

git clone git://review.webmproject.org/libvpx.git
cd libvpx/
git checkout fd0d7ff4c155b94d3f322addc7b66234b6908cc6
cat > /tmp/libvpx.x.patch << EOP
diff -rupN libvpx.orig/configure libvpx/configure
--- libvpx.orig/configure	2010-07-11 14:49:18.703230816 -0400
+++ libvpx/configure	2010-07-11 14:50:33.747300030 -0400
@@ -436,9 +436,9 @@ process_detect() {
         }
     fi
     check_header stdio.h || die "Unable to invoke compiler: \${CC} \${CFLAGS}"
-    check_ld <<EOF || die "Toolchain is unable to link executables"
-int main(void) {return 0;}
-EOF
+#    check_ld <<EOF || die "Toolchain is unable to link executables"
+#int main(void) {return 0;}
+#EOF
     # check system headers
     check_header stdint.h
     check_header pthread.h
diff -rupN libvpx.orig/vp8/common/postproc.c libvpx/vp8/common/postproc.c
--- libvpx.orig/vp8/common/postproc.c	2010-07-11 14:49:18.939266476 -0400
+++ libvpx/vp8/common/postproc.c	2010-07-11 14:51:03.855300662 -0400
@@ -481,7 +481,7 @@ int vp8_post_proc_frame(VP8_COMMON *oci,
     }

 #if ARCH_X86||ARCH_X86_64
-    vpx_reset_mmx_state();
+//    vpx_reset_mmx_state();
 #endif

     if (flags & VP8D_DEMACROBLOCK)
diff -rupN libvpx.orig/vp8/common/systemdependent.h libvpx/vp8/common/systemdependent.h
--- libvpx.orig/vp8/common/systemdependent.h	2010-07-11 14:49:18.947263304 -0400
+++ libvpx/vp8/common/systemdependent.h	2010-07-11 14:51:25.019253115 -0400
@@ -10,12 +10,12 @@

 #include "vpx_ports/config.h"
-#if ARCH_X86 || ARCH_X86_64
-void vpx_reset_mmx_state(void);
-#define vp8_clear_system_state() vpx_reset_mmx_state()
-#else
+//#if ARCH_X86 || ARCH_X86_64
+//void vpx_reset_mmx_state(void);
+//#define vp8_clear_system_state() vpx_reset_mmx_state()
+//#else
 #define vp8_clear_system_state()
-#endif
+//#endif

 struct VP8Common;
 void vp8_machine_specific_config(struct VP8Common *);
EOP
patch -p1 < /tmp/libvpx.x.patch
$BASEDIR/mingw32-setup.sh ./configure --disable-examples --disable-mmx --disable-sse --disable-sse2 --disable-sse3 --disable-ssse3
$BASEDIR/mingw32-setup.sh make
DESTDIR=$MINGBASE make install prefix=/
cd ../..
mv usr/local/lib/* lib
mv usr/local/include/vpx/ include/
cd src

wget -qO - http://webm.googlecode.com/files/ffmpeg-0.6_libvpx-0.9.1.diff.gz | zcat - > /tmp/ffmpeg-0.6_libvpx-0.9.1.diff
wget -qO - http://www.ffmpeg.org/releases/ffmpeg-0.6.tar.gz | tar xzf -
cd ffmpeg-0.6
patch -p0 < /tmp/ffmpeg-0.6_libvpx-0.9.1.diff

LDFLAGS="-L$MINGBASE/lib" CFLAGS="-I$MINGBASE/include" ./configure --target-os=mingw32 --cross-prefix=i586-mingw32msvc- --arch=x86 --prefix=$MINGBASE --enable-memalign-hack --enable-gpl --enable-avisynth --enable-version3 --enable-postproc --enable-libvpx --enable-runtime-cpudetect --disable-network --disable-devices --disable-encoder=vorbis --enable-libvorbis
make

cp ffmpeg.exe $BASEDIR

Just transfer that script to the user you create when you installed Ubuntu and run it. You will need to provide your password once so that the system can install a few tools but after that everything is done automatically. At the end you will have exe files in the directory you ran the script from.

Transfer the exe files that are created to a windows machine and then transcode a video like this:

Please note that you need to include the audio codec information above to correctly encode the video. If you don't include rate values you will get an error telling you the libvorbis can't be initialized.

When the transcoding is done you will have a WebM video. Grab your closest HTML5 video player like VideoJS and you are ready to go.

For a demo video encoded using FFMpeg created with the above script check out this WebM demo video. You will need to use one of the following browsers to view the demo (all of these are as of July 13 2010, support will eventually be added to the main build):

• Chromium dev channel build
• Firefox 4 beta
• Opera
• IE 9 when the user has the Direct Show DLLs installed

I have tested the demo video on each of the above, except for IE9 that doesn't have WebM support as of yet, and it seems to work pretty well.

• boot.fedoraproject.org – A small bootable image to start the install from.
• IntelliJ IDEA – IntelliJ IDEA community edition. I'm glad to see my favorite IDE make it in along with Netbeans and Eclipse even if it is just the stripped down version.
• Python 3 – Running along side Python 2.
• Nouveau DisplayPort – Support for NVIDIA cards with a DisplayPort.
• Radeon DisplayPort – Support for Radeon cards with a DisplayPort.
• NetworkManager Command Line – A command line interface for NetworkManager. As much as I find myself hating NetworkManager maybe this will help.
• Gnome 2.30 – The latest stable version of Gnome.
• KDE 4.4 – The latest stable version of KDE.

Check out the Fedora 13 feature list for all the major features in this release.

I'm continuing with the trend of using PreUpdate to do the upgrade:

1. Make sure you are using a Fedora 12 install. If you need instructions on getting to Fedora 12 then start with upgrading from Fedora 11 to Fedora 12.
2. yum clean all
3. yum update
4. Make sure you have plenty of space free in the /boot partition. I learned this lesson the last time I upgraded.
5. I always back up my xorg.conf file since because it seems to disappear sometimes.
6. preupgrade-cli "Fedora 13 (Goddard)" – My Fedora 12 to Fedora 13 upgrade weighed in at 1.0GB
7. reboot and wait

No dependency issues when upgrading this time making it two in a row. I was tempted to try out the Nouveau NVIDIA drivers this time around but the driver from NVIDIA are not too hard to upgrade now so I haven't found a real reason. If you don't want to compile the NVIDIA drivers by hand you can get them from atrpms, make sure to remove the Nouveau drivers first. The Fedora 13 version of Virtualbox was also needed.

I did run into a small issue with xorg after the upgrade. The NVIDIA driver requires the ignoreABI flag be sent to the X server on startup. I was able to modify startx but I always forget how to modify GDM. It turns out that you can also just put the flag in the xorg.conf file like this:

Section "ServerFlags"
        Option      "ignoreABI" "true"
EndSection

If you prefer the yum upgrade option here are the steps for that as well:

1. yum clean all
2. yum update
3. rpm -Uvh http://mirrors.kernel.org/fedora/releases/13/Fedora/i386/os/Packages/fedora-release-13-1.noarch.rpm http://mirrors.kernel.org/fedora/releases/13/Fedora/i386/os/Packages/fedora-release-notes-13-3.fc13.noarch.rpm
4. yum clean all
5. yum -y update
6. You may need to resolve dependencies and then do another yum -y update or in my case I had to use –skip-broken to resolve some issues with packages I've installed from atrpm.
7. reboot

I needed to use the –skip-broken flag on yum this time but I didn't have to resolve any dependency issues. The resulting update was about 1.2G so it took a little while to apply.

You have almost certainly seen faceted searching or faceted browsing already. A lot of online retailers now use facets in their online stores. Here are a few examples:

• An Amazon search for java programming
• A Best Buy search for flat panel TVs
• A New Egg search for Laptops

In each of the above cases you will find the left hand navigation displaying the facets. Here is what the Amazon search looks like:

One of the ways facets are useful is the ability to drill down into the results and create filters based on the facets. In the Amazon example I went from looking for anything that was related to "java programming" to only things related to books:

Some sites build a breadcrumb of facets you have selected as you drill down into the results. This lets you remove facets by just clicking one of the crumbs. The search for laptops on Newegg above provides an example of breadcrumb navigation:

Now that you have some examples of what faceted search looks like we can move on to creating the same using Sphinx. First we need some type of data to work from and in these examples I'm using the Stack Overflow CC data dump. The following steps should get you a database ready to index by Sphinx if you want to follow along with the same data:

1. Download the latest Stack Overflow CC data dump.
2. Create a MySQL database that will be used to hold the data.
3. Grab the schema I created to hold the data and load it into your new database.
4. Extract the Stack Overflow CC data dump into a directory.
5. Use the Ruby load script to load the XML files from the data dump into the database you created.

If you haven't already, now is the time to install Sphinx. I'm using version 0.9.9 for the following examples. I've created a sample configuration file that will get you started:

source stack_overflow_src
{
	type				= mysql

	sql_host			= localhost
	sql_user			= root
	sql_pass			=
	sql_db			= yourdbname
	sql_port			= 3306

	sql_query			= SELECT post.id AS post_id, post.owner_id AS owner_id, UNIX_TIMESTAMP(post.created) AS date_added, post.post_type_id AS post_type_id, post.score AS post_score, post.view_count AS post_view_count, post.favorite_count AS post_favorite_count, NOT ISNULL(post.accepted_answer_id) AS has_accepted_answer, GROUP_CONCAT(distinct(post_to_tag.tag_id)) AS tag_ids, title, body_text, GROUP_CONCAT(comment_text SEPARATOR '\n') AS comments, GROUP_CONCAT(distinct(tag.name) SEPARATOR ', ') as tags from post LEFT OUTER JOIN comment ON post.id = comment.post_id LEFT OUTER JOIN post_to_tag ON post_to_tag.post_id = post.id LEFT OUTER JOIN tag ON tag.id = post_to_tag.tag_id GROUP BY post.id;

	sql_attr_uint			= owner_id
	sql_attr_timestamp		= date_added
	sql_attr_uint			= post_type_id
	sql_attr_uint			= post_score
	sql_attr_uint			= post_view_count
	sql_attr_uint			= post_favorite_count
	sql_attr_bool			= has_accepted_answer
	sql_attr_multi			= uint tag_ids from field

	sql_ranged_throttle	= 0
	sql_query_info		= SELECT * FROM post WHERE id=$id
}

#############################################################################
## index definition
#############################################################################

index so_index
{
	source			= stack_overflow_src

	path			= ./data/so_index

	docinfo			= extern
	mlock			= 0
	morphology		= none
	min_word_len		= 1
	charset_type		= sbcs
	html_strip		= 1
}

index so_indexstemmed : so_index
{
	path			= ./data/so_index_stemmed
	morphology	= stem_en
}

#############################################################################
## indexer settings
#############################################################################

indexer
{
	mem_limit		= 512M
}

#############################################################################
## searchd settings
#############################################################################

searchd
{
	port			= 3312
	log			= /tmp/searchd.log
	query_log		= /tmp/query.log
	read_timeout	= 5
	max_children	= 30
	pid_file		= /tmp/searchd.pid
	max_matches	= 1000
	seamless_rotate= 1
	preopen_indexes= 1
	unlink_old		= 1
}

The above configuration file will index all posts in the database by title, body text, comments and tags as well add fields to each record for owner id, date added, post type, post score, post view count, post favorite count, if the post has an accepted answer and tag ids. The fields that get associated with each record can later be used in filters and for facets. For more information on how to use the configuration to create your index and start the Sphinx search server check out the Sphinx documentation.

You should now be able to test against the index. There is a handy PHP script named test.php included with the Sphinx source that will let you test just about every option available in the Sphinx search API (other language API wrappers are also available in the source but the test script is in PHP). The following is a simple search that will give you back every post related to "C#":

The key to faceted searching with Sphinx is the group by feature. Now try out the same search using the group and group sort options:

In the above case we group by tag ids and sort by the count of documents containing each from the result of the search for "C#".

Faceted searches in Sphinx require multiple search calls to work in conjunction with a normal search. The query will first be run to generate the non-faceted results. The same search query is then run again with a grouping for each facet you want to include in the results.

It is worth noting that doing the group by is not a simple operation for Sphinx to execute and can result in long queries. Take for instance the above search for "C#" where the non-faceted search returns 704556 documents. It will take a lot of work for Sphinx to count the tags for each of those documents. Slow here is 5 or 6 seconds per facet compared to sub-second for the non-faceted query.

A better approach is to only attempt to find faceted results when the query returns a limited enough set. Something generic enough to return hundreds of thousands of results probably won't be helped a lot by facets. In my final example I've limited the faceted search to queries that produce fewer than 1000 results.

I've created a couple simple examples in PHP to help get you started. These examples assume the sphinxapi.php library is in the current directory, you can find it in the Sphinx source API directory along with the test.php file mentioned above. The first example is a stripped down non-faceted search:

<?php

require ( "sphinxapi.php" );

$sphinx_client = new SphinxClient ();

$query = "java war file deploy";

$sphinx_client->SetServer ( 'localhost', 3312 );
$sphinx_client->SetConnectTimeout ( 1 );
$sphinx_client->SetArrayResult ( true );
$sphinx_client->SetWeights ( array ( 100, 1 ) );
$sphinx_client->SetMatchMode ( SPH_MATCH_ALL );
$sphinx_client->SetLimits( 0, 20, 20 );
$sphinx_client->SetRankingMode ( SPH_RANK_PROXIMITY_BM25 );

$results = $sphinx_client->Query ( $query, '*' );

if( $results===false )
{
  print "Query failed: " . $sphinx_client->GetLastError() . ".\n";
}
else
{
  if ( $sphinx_client->GetLastWarning() )
    print "WARNING: " . $sphinx_client->GetLastWarning() . "\n\n";

  print_results($query, $results);
}

function print_results($query, $results)
{
  print "Query '$query' retrieved $results[total] of $results[total_found] matches in $results[time] sec.\n";
  print "Query stats:\n";
  if ( is_array($results["words"]) )
    foreach ( $results["words"] as $word => $info )
      print "    '$word' found $info[hits] times in $info[docs] documents\n";
  print "\n";

  if ( is_array($results["matches"]) )
  {
    $n = 1;
    print "Matches:\n";
    foreach( $results["matches"] as $docinfo )
    {
      print "$n. doc_id=$docinfo[id], weight=$docinfo[weight]";
      foreach( $results["attrs"] as $attrname => $attrtype )
      {
        $value = $docinfo["attrs"][$attrname];
        if( $attrtype & SPH_ATTR_MULTI )
          $value = "(" . join ( ",", $value ) .")";
        else if ( $attrtype==SPH_ATTR_TIMESTAMP )
          $value = date ( "Y-m-d H:i:s", $value );
        print ", $attrname=$value";
      }
      print "\n";
      $n++;
    }
  }
}
?>

This second script does the same group by as the above test script example to create faceted results where the facets are tag ids:

<?php

require ( "sphinxapi.php" );

$sphinx_client = new SphinxClient ();

$query = "java war file deploy";

$sphinx_client->SetServer ( 'localhost', 3312 );
$sphinx_client->SetConnectTimeout ( 1 );
$sphinx_client->SetArrayResult ( true );
$sphinx_client->SetWeights ( array ( 100, 1 ) );
$sphinx_client->SetMatchMode ( SPH_MATCH_ALL );
$sphinx_client->SetLimits( 0, 20, 20 );
$sphinx_client->SetRankingMode ( SPH_RANK_PROXIMITY_BM25 );

$sphinx_client->SetGroupBy( 'tag_ids', SPH_GROUPBY_ATTR, '@count desc' );

$results = $sphinx_client->Query ( $query, '*' );

if( $results===false )
{
  print "Query failed: " . $sphinx_client->GetLastError() . ".\n";
}
else
{
  if ( $sphinx_client->GetLastWarning() )
    print "WARNING: " . $sphinx_client->GetLastWarning() . "\n\n";

  print_results($query, $results);
}

function print_results($query, $results)
{
  print "Query '$query' retrieved $results[total] of $results[total_found] matches in $results[time] sec.\n";
  print "Query stats:\n";
  if ( is_array($results["words"]) )
    foreach ( $results["words"] as $word => $info )
      print "    '$word' found $info[hits] times in $info[docs] documents\n";
  print "\n";

  if ( is_array($results["matches"]) )
  {
    $n = 1;
    print "Matches:\n";
    foreach( $results["matches"] as $docinfo )
    {
      print "$n. doc_id=$docinfo[id], weight=$docinfo[weight]";
      foreach( $results["attrs"] as $attrname => $attrtype )
      {
        $value = $docinfo["attrs"][$attrname];
        if( $attrtype & SPH_ATTR_MULTI )
          $value = "(" . join ( ",", $value ) .")";
        else if ( $attrtype==SPH_ATTR_TIMESTAMP )
          $value = date ( "Y-m-d H:i:s", $value );
        print ", $attrname=$value";
      }
      print "\n";
      $n++;
    }
  }
}
?>

The last example is a more complex and complete faceted search example that I have put on github. This example gives you a full search interface for the Stack Overflow CC data that includes filters and faceted searches. Here is a screen shot of the resulting interface displaying a faceted search result:

If you are interested in more information about faceted searches I found the following articles helpful:

• LinkedIn's Faceted Search
• Using Rails UltraSphinx for Faceted search
• Solr faceted search
• Design Patterns for Mobile Faceted Search: Part I and Design Patterns for Mobile Faceted Search: Part II
• Design Patterns for Faceted Navigation

• iPad-4×3.plist – A 4×3 aspect ratio at 640×480
• iPad-16×9.plist – A 16×9 aspect ratio at 640×360
• iPad-Full.plist – A 4×3 aspect ratio at 1024×576

Note that when you import videos using iTunes the iPad puts them in their own Videos app unlike the iPhone where they show up under the iPod app. You will want to find the Videos icon if you don't already know where it is:

After verifying that the above 4×3 version worked for the iTunes University video I went about testing it on a couple other video formats. I tested the Big Buck Bunny video that I also used for my post on iPad video streaming in both 640×360 and 1024×576 output formats. Both resolutions looked great. I also tried converting a DVD. If you decide to convert a DVD you will probably want to turn on de-interlacing in HandBrake. You do that by first selecting the "Picture Settings" option:

Then the filters tab and then select the type of de-interlace you want (fast, slow, slowest):

If you want to use FFMpeg to do all this you can. The following is a slightly modified version of the streaming command I'm using that will output a high bitrate version of the input video:

Here is a demo of the iPad streaming video created with the segmenter (I tried to show the progressive upgrade happening but it happens very quickly since the iPad is on WIFI, I also show that you can scrub without any issues and if you look in the background you can see the server log displaying entries as the segments are downloaded):

If you want to view the demo yourself I've created a demo for the iPad, iPhone, and iPod Touch. Note that those are two links, one for the iPad version and one for the iPhone/iPod Touch version. The main difference is the size of the video.

I used the open source video Big Buck Bunny (the 1920×1080 ogg version) for the above demos.

If you are interested in more details on what changed read on or skip to the bottom if you want to see what I'll be working towards in future versions of the segmenter.

Over the past few weeks I learned that it is important that each segment starts with an IDR frame. To accomplish this I thought setting the gop so that the segment time % (gop size / frame rate) = 0 would work but I haven't completely convinced myself of this yet. An example of would be a segment size of 10 seconds % (300 gop size / 30 frame rate) = 0. This should insure that each segment starts with the correct i-frame and while I have looked over the resulting segments with a hex editor and believe it works I still get errors when using certain tools on the individual segments that makes me think it isn't working like I think it should. Either way I have included updates to the gop size in the example configuration files. The gop size is controlled by the -g option for FFMpeg and in the examples I have set it so that for a 10 second segment with the given frame rate the gop size makes sure each segment starts correctly. If you want to know more you can dig into the resulting segments and use this forum post on how to extract an i-frame, this list of mpeg headers to verify that each has the correct i-frame at the start.

The Apple streaming tech notes where informative in a few other ways as well. The tech notes contain the supported h264 profiles for each device. For the iPhone/iPod Touch baseline level 3.0 is supported while the iPad supports baseline level 3.1. You can find out more about h264 encoding levels with FFMpeg and also review FFMpeg x264 encoding guide, FFMpeg option mappings and information on encoding with the x264 codec for more information.

Another bit of information disclosed in the Apple tech notes is the existence of a media stream validation tool. This tool can be downloaded from the Apple iPhone developer site and has to be run on OS X. Apple recommends that you use this tool to validate any streams that you create.

One other note that I have recently ran across is that Apple seems to be rejecting native apps that have video streams without a fall back audio stream. The correct way of generating the fallback 64k audio only stream is something that is lacking in the current version of the segmenter but I hope to fix that soon. It won't matter for those using the HTML5 video tag however.

If you are using HTML5 video tags it is important not to use my older posts as a guide for the iPad since I use a now outdated example. Those examples still work on the iPhone but they do not work on the iPad. Here is an updated example that will work on both the iPhone and iPad:

<html>
  <head>
    <title>Video Test</title>
  </head>
  <body style="background-color:#FFFFFF; ">
    <center>
      <video controls>
        <source src="stream_multi.m3u8"/>
      </video>
    </center>
  </body>
</html>

Finally I am currently using the latest version of FFMpeg to do tests and it seems to have become more reliable for me. The current version in git seems to be pretty good at producing usable videos from a variety of source videos.

The following is a quick summary of the recent changes I have made:

• Applied an audio patch for the segmenter from Scott Kidder. This patch skips any video processing if there is no video stream.
• Modified sample configs with newer FFMpeg string, gop sizes, frame rates and other parameters to match the Apple recomended values for iPhone/iPod Touch with aspect ratios of 4:3 and 6:9 on cell or wifi and iPad with aspect ratios of 4:3 and 6:9 on cell or wifi
• Added configuration sanity checks
• Made gems not required if not using those features that need them
• Added fix for deprecation warning from newest version of libavformat
• Made termination work better when killed with SIGINT/ctrl-c
• Fixed some minor issues with index file format

The following are enhancements I'm planning on making:

• I plan on creating a howto on getting everything set up on EC2.
• Dig more into verification that each segment starts with the correct I-frame.
• I'm going to work on refactoring everything so that instead of creating files and then transferring them it streams them all. This makes more sense with doing encryption as just another part of the pipeline.
• Add ability to do encryption. I have a proof of concept working but I decided I need to get everything set up in one continuous stream before integrating encryption into the mix.
• I had thoughtabout using the ruby daemon utils to wrap the segmenter but I decided it probably isn't necessary at this point.

First off you may ask yourself why you would want to use this. The best answer I have for that is that the solution in the other post loads the entire result set into memory from the very start while with this solution you can control just how many rows you load. To get started you will want to take a look at the MySQLi statement set attribute call. This call is will let you modify the underlying cursor type that is used with the prepared statement in two ways that are useful for this issue.

The MYSQLI_STMT_ATTR_CURSOR_TYPE attribute controls the type of cursor used for the results from the prepared statement. The default option is MYSQLI_CURSOR_TYPE_NO_CURSOR which results in no cursor being used at all. The only other option is MYSQLI_CURSOR_TYPE_READ_ONLY and this is the one you will want to use.

The MYSQLI_STMT_ATTR_PREFETCH_ROWS attribute controls the number of rows that are fetched under the covers from the server as more data is needed. The default for this attribute is 1 which means that each fetch of the row causes a round trip to the server. Larger numbers for this attribute will cause more memory to be used to store the resulting rows but fewer round trips to the server. This is the attribute that gives this option more flexibility.

The following is a slightly modified example that was used in the previous post now using both the MYSQLI_STMT_ATTR_CURSOR_TYPE and MYSQLI_STMT_ATTR_PREFETCH_ROWS attributes to efficiently retrieve rows from two different prepared statements at the same time:

<?php

  $db_connection = new mysqli('127.0.0.1', 'user', '', 'test');

  $post_stmt = $db_connection->prepare("select id, title from post where id between 10000 and 10999");
  $post_stmt->attr_set(MYSQLI_STMT_ATTR_CURSOR_TYPE, MYSQLI_CURSOR_TYPE_READ_ONLY);
  $post_stmt->attr_set(MYSQLI_STMT_ATTR_PREFETCH_ROWS, 100);
  $comment_stmt = $db_connection->prepare("select user_id from comment where post_id = ?");

  if ($post_stmt->execute())
  {
    $post_stmt->bind_result($post_id, $post_title);

    while ($post_stmt->fetch())
    {
      $comments = array();

      $comment_stmt->bind_param('i', $post_id);
      if ($comment_stmt->execute())
      {
        $comment_stmt->bind_result($user_id);
        while ($comment_stmt->fetch())
        {
          array_push($comments, array('user_id' => $user_id));
        }
      }
      else
      {
        printf("Comment statement error: %s\n", $comment_stmt->error);
      }

      printf("ID: %d -> %s\n", $post_id, $post_title);
      print_r($comments);
    }
  }
  else
  {
    printf("Post statement error: %s\n", $post_stmt->error);
  }

  $post_stmt->close();
  $comment_stmt->close();

  $db_connection->close();
?>

I'm going to assume you know something about Spring and their MVC configuration in general. If you want a simple MVC example to start with check out their SVN MVC-Basic sample. There is a decent amount of boiler plate work so I've created a project zip file with all the code, a build file and configuration in it (you just need to add the libraries).

First lets start by gathering up all the libraries we are going to need (see the end of the post for the directory layout I used or just unzip the project). Grab a copy of the Spring 3 libraries, at this time the latest version is spring-framework-3.0.1.RELEASE-A.zip and that is what I used for the following example. For the file upload part there are two non-Spring dependencies as well: Apache Commons FileUpload and Apache Commons IO. If you want to use the example project from the zip just copy all the jar files to the lib directory.

Next it is worth a quick look at the information provided in the Spring documentation about MVC multipart. It touches on what needs to be done but leaves out a full example. In documentation you will see that they talk about using a "multipart resolver" and that can be found at the end of web/WEB-INF/spring/app-config.xml:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:context="http://www.springframework.org/schema/context"
       xsi:schemaLocation="
		http://www.springframework.org/schema/beans	http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
		http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd">

    <!-- Scans the classpath of this application for @Components to deploy as beans -->
    <context:component-scan base-package="net.ioncannon"/>

    <!-- Configures Spring MVC -->
    <import resource="mvc-config.xml"/>

    <!-- Configure the multipart resolver -->
    <bean id="multipartResolver" class="org.springframework.web.multipart.commons.CommonsMultipartResolver">
        <!-- one of the properties available; the maximum file size in bytes -->
        <property name="maxUploadSize" value="100000"/>
    </bean>

</beans>

The above app-config.xml configuration file is referenced from the mvc-config.xml configuration file. The mvc-config.xml configuration file doesn't need anything special for multipart uploads so I'll skip listing it here, have a look in the zip file if you are interested in its contents.

Now that the configuration is set the next step is to create the entry form for the upload. I've done this in the view JSP found in web/WEB-INF/views/upload/uploadForm.jsp:

<%@page contentType="text/html;charset=UTF-8" %>
<%@page pageEncoding="UTF-8" %>
<%@ page session="false" %>
<%@ taglib prefix="form" uri="http://www.springframework.org/tags/form" %>

<html>
    <head>
        <META http-equiv="Content-Type" content="text/html;charset=UTF-8">
        <title>Upload Example</title>
    </head>
    <body>
        <form:form modelAttribute="uploadItem" method="post" enctype="multipart/form-data">
            <fieldset>
                <legend>Upload Fields</legend>

                <p>
                    <form:label for="name" path="name">Name</form:label><br/>
                    <form:input path="name"/>
                </p>

                <p>
                    <form:label for="fileData" path="fileData">File</form:label><br/>
                    <form:input path="fileData" type="file"/>
                </p>

                <p>
                    <input type="submit" />
                </p>

            </fieldset>
        </form:form>
    </body>
</html>

A few notes about this view JSP:

• The form needs to have the encoding type set correctly. Look for: enctype="multipart/form-data"
• The file input field needs to be set: type="file"
• In the above I've included a Name field in the model as well as the File field just to show that you can mix fields in the same form
• The definition of the model can be found next and I assume that it is introduced into the form as "uploadItem"

The model is pretty simple and can be found in src/net/ioncannon/model/UploadItem.java:

package net.ioncannon.model;

import org.springframework.web.multipart.commons.CommonsMultipartFile;

public class UploadItem
{
  private String name;
  private CommonsMultipartFile fileData;

  public String getName()
  {
    return name;
  }

  public void setName(String name)
  {
    this.name = name;
  }

  public CommonsMultipartFile getFileData()
  {
    return fileData;
  }

  public void setFileData(CommonsMultipartFile fileData)
  {
    this.fileData = fileData;
  }
}

The Name attribute is just an extra input field and not the name of the file. Notice that the File attribute is of type CommonsMultipartFile. The CommonsMultipartFile type has a number of features that give you information about the uploaded file as well as access to its contents.

The controller brings everything together. The source can be found in the file src/net/ioncannon/controller/UploadController.java:

package net.ioncannon.controller;

import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.ui.Model;
import org.springframework.validation.BindingResult;
import org.springframework.validation.ObjectError;

import net.ioncannon.model.UploadItem;

@Controller
@RequestMapping(value = "/upload")
public class UploadController
{
  @RequestMapping(method = RequestMethod.GET)
  public String getUploadForm(Model model)
  {
    model.addAttribute(new UploadItem());
    return "upload/uploadForm";
  }

  @RequestMapping(method = RequestMethod.POST)
  public String create(UploadItem uploadItem, BindingResult result)
  {
    if (result.hasErrors())
    {
      for(ObjectError error : result.getAllErrors())
      {
        System.err.println("Error: " + error.getCode() +  " - " + error.getDefaultMessage());
      }
      return "upload/uploadForm";
    }

    // Some type of file processing...
    System.err.println("-------------------------------------------");
    System.err.println("Test upload: " + uploadItem.getName());
    System.err.println("Test upload: " + uploadItem.getFileData().getOriginalFilename());
    System.err.println("-------------------------------------------");

    return "redirect:/app/";
  }
}

Once the file is uploaded I'm just dumping out part of the information. Doing something exciting with the file is up to you.

That is all there is to it. Here is a list of what the project structure looks like with everything pulled together including all the libraries, build file, source and configuration files: