We have a virtualized server on the office, using a free VMware ESXi 4.1 solution. We also have the machine connected to a UPS.

Unfortunately, the power lines on the office already have failed us, so we wanted to configure the server to shutdown gracefully all the VMs and then shutdown the host itself.

These were the steps we took to take care of this:

Install vmware tools on all the VMs. Using the vsphere console (I think that’s what its called…) is easy enough:

• right mouse button on top of the VM name > Guest > install VMware tools;
• on windows guests, just follow the wizard
• On linux guests, mount /dev/cdrom, copy the tar.gz to some temporary location, extract it, use the *_install.pl script to install (accept all the defaults, ensure you have previously installed gccs, linux-headers, and such (on ubuntu, build-essential and linux-headers should be enough)

Create a new VM (once again, using the vsphere) with minimal disk space and RAM, to just install the UPS drivers. This VM job is just take care of listen to the UPS connected to the host. On the properties of the VM, make sure you add a serial port, mapping directly to the physical serial port of the host

• On my case, I installed ubuntu server; The UPS we have on the office is a Riello NETDialog, so we downloaded the drivers from the manufacturer (NUT did not have drivers for it) and installed them. Just had to configure the UPS connection to use the serial on /dev/ttyS0

Now comes the nice part: the idea is to make this VM to connect to the host OS and run the appropriate commands to begin the shutdown process. SSH to the rescue… on that VM, just create a pair of SSH keys (ssh-keygen -t RSA, for example) WITHOUT PASSWORD, and push the public one to the host.

• Of course, you have first to enable SSH on the ESXi host first. For that, we just connected a monitor and keyboard to the host temporarily, logged in using root, and enable the something troubleshooting option, which is a fancy name for turning on the SSH server.
• with the ssh public key on the ESXi host, you just have to put it on the well known authorized_keys file
• mkdir .ssh
• cat id_rsa.pub > .ssh/authorized_keys

Done, you should  now have access to the ESXi host without having to enter a key. Which means, you can automate the process of shuting it down when the UPS starts running on battery, by running an arbitrary command through SSH. On this matter, here goes the script we used on the ESXi host to shutdown everything (this came from a blog whose link I can’t remember. I’m trully sorry. The credits for this should not go to me!)

## get all the VMs identifiers
VMID=$(/usr/bin/vim-cmd vmsvc/getallvms | grep -v Vmid | awk '{print $1}')
## loop through all the VMs
for i in $VMID
do
## get their state (turned on, off, whatever)
STATE=$(/usr/bin/vim-cmd vmsvc/power.getstate $i | tail -1 | awk '{print $2}')
## if they are running, turn them off (only works correctly if
## vmware tools are installed on the VMs)
if [ $STATE == on ]
then
/usr/bin/vim-cmd vmsvc/power.shutdown $i
fi
done
## shutdown the host itself
sleep 30
/sbin/shutdown.sh
/sbin/poweroff

• we saved the script above inside /home/shutdownVMs.sh (we had to create the directory first) on the ESXi host

There’s a small problem here: I’m quite new to this VMware things, but from my understanding, the host completely wipes out all the files you’ve just created on reboot. Everything you see from / on the filesystem gets wiped when the machine reboots. So we have to take a few steps to get the changes you’ve made on authorized_keys and the shutdown script inside /home  permanent. These steps include packing up the directories and files you’ve created and editing a file (I’ve seen this on a blog I can’t remenber… sorry):

## pack your files
tar -C / -czf "/bootbank/home.tgz" /.ssh /home
## edit the file /bootbank/boot.cfg and add the new compressed file. In our case,
## the file reads the following:
kernel=b.z
kernelopt=
modules=k.z --- s.z --- c.z --- oem.tgz --- license.tgz --- m.z --- state.tgz --- home.tgz
build=4.1.0-348481
updated=1
bootstate=0

• We’re done on ESXi; now is just configure the UPS monitoring VM, to run a specific command upon AC fail; We’ve done the following:

ssh -i *my_private_key* root@*esxi_host_ip 'sh /home/shutdownVMs.sh'

Conclusion

So, with this, you get a VM listening to your UPS state, and connecting to the virtualization host when the AC fails. The graceful shutdown of all the VMs is only possible thanks to the vmware tools installed on each one of the VMs.

So, I wanted to revert the changes I made (in several commits done) to the repository, creating a commit which would throw away those changes.

I found a possible solution in http://stackoverflow.com/questions/1463340/revert-multiple-git-commits

Assuming you have a history like:

(…) -> A -> B -> C -> D

and you want to revert all the changes done through B, C and D, you can try something like:

$ git reset --hard A
$ git reset --soft @{1}
$ git commit -a

where A is the SHA of the commit you want to revert to.

Creating a Digitally Reconstructed Radiograph (DRR) from a triangle mesh is like shooting rays from some source position to each pixel of the final image. One pixel, one ray. The objective is to find intersection points of those rays with each triangle composing the model. Intersection points enables us to calculate distances to the source and, from these, the distance a ray traverses inside the object’s volume (path length).

Applying this final value to a particular formula — attenuation law — (just summing all distances and multiplying by an attenuation factor, basically) gives us the final intensity for a pixel.

Context

This post relates to the research project I developed during my thesis, which I concluded a few weeks ago. The goal was to develop efficient algorithms for DRR generation from 3D models of human vertebrae. For this purpose, a set of algorithms were implemented under NVIDIA’s CUDA platform, which allowed the implementation in GPU.

The post will not have much detail. But for whom is interested, the complete thesis report can be found here, and the presentation used during the defense is here (or on slideshare). Following is the description of the most important problems treated during the project.

Naive approach?

We can span a thread for each pixel, following what was described above: iterate over all the triangles, apply a ray-triangle intersection test (heavy computation) and, if an intersection exists, calculate the distance to the source. Or we can do this in reverse: span a thread for each triangle (possibly, fewer threads) and do the same computation. However, we have concurrency problems here, because different threads will try to access the same pixel buffers to hold intermediate information. Moreover, using either approach — image order or object order — we still have to blend together all the intersection-source distances, in an orderly fashion. What was described, computes intersections in triangle submission order (unordered). For example, the picture shows a ring-shaped model. To compute its DRR, we would shoot rays to every pixel. One ray is depicted in the picture, having as intersections points P1, P2, P3 and P4. Finding the path length of the ray is just a matter of subtracting and adding distances. However, we need to do this in depth order!! The problem with the described algorithms is that there is no guarantee we will compute those four points in the correct order…

These two approaches are expensive to perform, mainly because of all the intersection tests and distance calculation.

We want to compute (||P2 − P1|| + ||P4 − P3||). Careful with triangle edges!

A faster approach

It is possible to simplify this problem by assuming that the DRR quality is not significantly affected if we don’t bother to compute Euclidean-based distances.

Instead of looking for ray-triangle intersections, we can just compute the (interpolated) depth value of all the fragments falling into a pixel, and accept that each of those values is an approximation of the ray intersection distance to the source. Doing this in CUDA is considerably faster than the other approaches.

Without many details, this can be implemented using somewhat of a triangle rasterization procedure: for each triangle of the model, our goal is to determine all the pixels it covers when projected onto the image plane, and store each fragment’s depth as a integer. To solve this, I followed a scanline approach, mixed with Bresenham line algorithm to avoid floats during part of the algorithm.

The problem with this approach, is that we are spanning a thread for each triangle (object order) and, as said before, we’ll have concurrency problems when trying to save the fragment’s depth on the correct pixel buffer. To avoid this, we can make use of  CUDA atomic operations. The following enables any number of threads to safely write on an array, concurrently,  and storing some value on a different array index.

index = atomicInc(&counter, INT_MAX)
depthArray[index] = Znew // RAW-hazard free!!!! BASED ON [3]

After this stage — where for each pixel we save all the fragment’s depths — we’ll end up with a 3-dimensional grid containing all the projected fragments. But they will not be ordered. For the final stage we just need to order them, and subtract them two by two, accumulating the result. We end up with a bi-dimensional grid — where each value comes from the blending of each pixel depth array — which is our DRR.

The advantage of the described approach is that we avoid complicated math, artifact generation (careful scanline procedure — filling convention!) and we get a complete DRR with one single geometry pass.

DRR from l3 vertebra, taken from a frontal perspective

The topics covered in this post were the subject of my thesis, during this last semester. The document can be seen here. The presentation, used during the defense, can be seen in slideshare.

Keep in mind that this is a blog post, and I did not bother to properly reference previous work or authors. Credits for many of what I explored here is due to other people’s work. Below, is a very incomplete list of previous work.

Any question, comment or advise is welcome.

References

1. Vidal, F. P., M.Garnier, N. Freud, J. M. Létang, and N. John (2009). Simulation of x-ray attenuation on the gpu. In Proceeding of TCPG’09 – Theory and Practice of Computer Graphics, pp. 25–32. Eurographics.
2. Bresenham, J. E. (1965). Algorithm for computer control of a digital plotter. IBM Systems Jour- nal 4(1), 25 –30.
3. Liu, F., M.-C. Huang, X.-H. Liu, and E.-H. Wu (2010). Freepipe: a programmable parallel ren- dering architecture for efficient multi-fragment effects. In I3D ’10: Proceedings of the 2010 ACM SIGGRAPH symposium on Interactive 3D Graphics and Games, New York, NY, USA, pp. 75–82.
4. Moura, D. C., J. Boisvert, J. G. Barbosa, and J. M. Tavares (2009). Fast 3d reconstruction of the spine using user-defined splines and a statistical articulated model. In ISVC ’09: Proceedings of the 5th International Symposium on Advances in Visual Computing, Berlin, Heidelberg, pp. 586–595. Springer-Verlag.
5. Filling convention
6. NVIDIA CUDA platform

Func, Fedora Unified Network Controller, allows us to send commands for remote execution in a set of pre-configured hosts. More, it allows us to choose which machines we want to address using shell globs. This, and more, is done securely over SSL. Func facilitates remote administration.

Func is a Fedora project whose general architecture is based on two parties: Func itself and Certmaster. Certmaster is responsible for authentication and encryption using SSL certificates. Certificates are identity proofs. And, from my knowledge, (correct me) they are little more than a signature (private key encryption of an hash) attached to a cryptographic public key. Certmaster application is composed by a client and a daemon capable of requesting and serving SSL certificates. This enables a master machine to communicate with a slave over SSL.

On top of this comes Func: it is also a built on client-server style and enables administrators to build an infrastructure of slave machines running the daemon funcd. This daemon listens on a port for requests from a certain master machine. In fact, the SSL scheme behind it lets us sleep tight at night knowing that those slaves will only reply to the master and not some fake.

The master is where the all-powerful sysadmin is logged in From that machine, the sysadmin can issue commands for execution by the slaves at the same time. Even cooler, he can address those machines in the following manner:

"*fe.up.pt"

Meaning all the machines in the “fe.up.pt” domain under my control. How cool is that?

Func is python, and has a python API so, besides using Func directly, we can even build our own customized scripts.

Installation…

I tested this in CentOS 5. I think Func and Certmaster are in the EPEL repo.

This, in fact, little more than what can be found in the official Func page.

On the master:

Install Func and Certmaster. Certmaster gets pushed by Func on installation.

Certmaster daemon will be running on the master machine, making it possible for the slaves (called minions) to request certificates.

$ yum install func
$ /sbin/chkconfig --level 345 certmaster on
$ /sbin/service certmaster start

On the slave machines:

On the minions, the funcd will be running, accepting requests from func client and running commands. Besides this, func will try to request certificate signing from the master. For this to work, we’ll need to indicate the master’s address on the certmaster configuration file.

$ yum install func

Edit the /etc/certmaster/minion.conf file and change the address of your master machine. For example:

# configuration for minions
[main]
certmaster = master.fe.up.pt
certmaster_port = 51235
log_level = DEBUG
cert_dir = /etc/pki/certmaster

Now, put the funcd running at boot and start it.

$/sbin/chkconfig --level 345 funcd on
$/sbin/service funcd start

Also, you need to open ports on iptables. The master machine must have port 51235 open, and the minions must have port 51234 open.

Working

Now we have the environment set.

Before starting to send commands to slaves, we need to “sign” them first. Issue the following command from the master to list the slave machines. If everything is ok, you’ll see all the slave machines where you’ve just installed func.

$ certmaster-ca --list

Once you’ve seen the machines that are ready to be signed, you can sign them:

$ certmaster-ca --sign slave1.fe.up.pt
$ certmaster-ca --sign slave2.fe.up.pt
$ ...

Example

Now, you’re ready to send commands. Let’s imagine you want to send a message to all logged users on the slaves, using wall command:

$ func "*.fe.up.pt" call command run "wall 'Hello users! Machine will go down for reboot in 1 minute! Save you work!'"

This is a simple example. You can use more complex globbing to address specific machines. See https://fedorahosted.org/func/wiki/CommandLineGlobbing.

About provisioning

If you want to integrate minion installation with kickstart for example, it’s fairly simple, as you only need to install packages and change a line of the config file.

You can run into some problems if you happen to kickstart a minion machine, both in the master and slave. This is because in the master there will be a CSR inconsistent with the new machine. And in the minion, because funcd automatically retrieves the certificate at boot, funcd will have a wrong certificate, one referring to the previous state of the machine before re installation. You can solve this by cleaning SSL keys and such in the minion and cleaning the certificate request on the master:

# on the new, kickstarted machine
$ service funcd stop
$ rm -rf /etc/pki/certmaster/*
# on the master
$ certmaster-ca --clean <hostname_of_the_kickstarted_machine>
# again, on the slave
$ service funcd start
# on the master again
$ certmaster-ca --sign <hostname_of_the_kickstarted_machine>

All this is just to clean certificates and certificate requests form both ends.

This can be avoided if, immediately before restarting the minion to a re installation,  we clean its respective certificate request in the master with:

$ certmaster-ca --clean <hostname>

This way, when the minion boots up with the funcd daemon running, it will reach a consistent state.

Still, you’ll have to issue the –sign command.

References

• https://fedorahosted.org/func/

He’s called Jordão and there’s a small chance he’s actually a she… I don’t know…

The Cage

Jordão Resting

Jordão facing me

Jordão being held...He does not like it...heads will roll...!

Jordão dating a gal

]]> http://blog.andrecardoso.eu/2010/08/i-haz-bunny/feed/ 0 http://blog.andrecardoso.eu/2010/08/i-haz-bunny/ http://feedproxy.google.com/~r/andrecardoso/bYoR/~3/4xsSjyIhRPY/ http://blog.andrecardoso.eu/2010/08/openafs-encryption-and-kerberos-v/#comments Mon, 16 Aug 2010 01:41:21 +0000 belerophon http://blog.andrecardoso.eu/?p=513 Context

It was our intention to build a secure network, with single sign-on mechanisms. With ‘secure’ we mean secure authentication and transmission of data. Kerberos solves the problem of secure authentication using a shared secret key architecture. In fact, while authenticating principals passwords are never sent over the cables. Authentication is based upon the ability of the principal of decrypting some piece of information.

Kerberos also solves the problem of mutual authentication: each service (server) and consumer must  know for sure that they are communicating with authorized parties and not with fakes. This is cleverly solved using the Needham-Schroeder scheme. This algorithm enables two parties to share a session key, securely. The session key can be further used to protect communications between them. In particular, Kerberos introduces the concept of Authenticators. These are, basically, principal identifiers and timestamps. When a client initiates communication with a service, sends him … well it sends him a butload of things but the important thing is that it sends him a Authenticator encrypted with the session key that both will use. However, the service cannot decrypt it yet, because it has no access to the session key itself. To obtain the session key, he first has to decrypt another part of the message sent by the client, which is the session key encrypted with the key that only he and the Ticket Granting Service share. Now, the service is prepared to decrypt the Authenticator sent by the client and reply to him with a new Authenticator which now contains the original timestamp plus 1. This goes, once again, encrypted using the same session key. This can somewhat be seen in the pictures below. They were taken from the course I’ve attended last year on Computer Systems Security.

Authentication protocol used in Kerberos. Simplified. Taken from the lecture notes of Computer Systems Security: http://web.fe.up.pt/~jmcruz/ssi/acetat/4-autentica.pdf

Service request phase. Simplified. Taken from the lecture notes of Computer Systems Security: http://web.fe.up.pt/~jmcruz/ssi/acetat/4-autentica.pdf

Wrapping up, mutual authentication is achieved, from the service side, by matching the information found in the Authenticator against the information found in the ticket (crafted by the TGS) and the network information. If everything matches, the client is authenticated. The client authenticates the server when it receives from it the proper authenticator in reply to its own.

Problem

By now we have authentication and, as a sub-product of kerberos system, communication encryption. OpenAFS can now enter here as a means to distribute, for example, the home directories of users over the network (network login facilities…). OpenAFS has the advantage of being a natively kerberized application. The problem is the type of encryption supported by it.

If I’m not mistaken, AES is accepted as the strongest encryption algorithm and is also supported by Kerberos. However, by what I know, there are two problems with OpenAFS: it only supports des-cbc-crc:v4 principal keys, which means authentication is done using a weak encryption protocol. The other problem is that communication between afs clients and servers get encrypted using something even weaker: fcrypt.

In fact, this is something that confuses me: I thought that encryption of any two parties “talking” kerberos was done using the session key given by the TGS. But it seems that applications like OpenAFS bypass this, using their own encryption scheme. Or maybe the session key is used only for the mutual authentication part. In this case, what type of encryption protocol is used those messages? This whole thing about encryption leaves me confused…if someone cares to shed some light, it would much appreciated

But as far as I know, in the current network design that we were trying to mount (kerberos, ldap, openafs) the weak link is OpenAFS, because all the other principals can use of AES encryption type.

Nonetheless, this whole scheme of Kerberos plus OpenAFS plus OpenLDAP is just great to whom seeks to secure a smaller network within unsecured networks.

References

• http://docs.openafs.org/Reference/8/asetkey.html
• http://users.surfvi.com/~ota//fcrypt-paper.txt
• http://docs.openafs.org/Reference/1/fs_setcrypt.html
• http://web.mit.edu/kerberos/krb5-1.8/krb5-1.8.3/doc/krb5-admin.html#Supported%20Encryption%20Types
• http://en.wikipedia.org/wiki/Needham-Schroeder_protocol
• http://en.wikipedia.org/wiki/Kerberos_%28protocol%29#Client_Service_Authorization
• http://web.mit.edu/Kerberos/dialogue.html
• http://openafs.org/
• http://web.mit.edu/kerberos/
• http://web.fe.up.pt/~jmcruz/ssi/acetat/4-autentica.pdf

As the previous post with my first presentation seems to have vanished from my blog (VPS problems :S), I leave here the first presentation as well.

Update (31/07/2010):

The term has ended, and I’ve finished the report for the course dedicated to prepare us for the thesis development on the next school term.

So, I leave here the link for the report, to complete the documentation:

ftp://andrecardoso.eu/ftp/PDIS/report.pdf

(It’s far from perfect though:)

Our program who art in memory, Hello be thy Name. Thy Operating System come, thy Commands be done, at the Printer as it is on the screen. Give us this day our daily data, and forgive us our I/O Errors as we forgive those whose Logic Circuits are faulty. Lead us not into frustration, and deliver us from Power Surges. For Thin is the Algorithm, the Application, and the Solution, looping forever and ever.

Return.

(Credits go to the author, whom I don’t know. It is a nice prayer though..:P)

Then, if you look to the “Getting Started Guide for Mac“, they tell you to make sure your gpu is supported and then compile the sample code provided in /Developer/GPU Computing/C/.

This part failed for me, because I kept getting the “no table of contents” error when the linker tried to link against the libraries, namely the libcutil_i386.a.

I’ve managed to compile the samples after googling for this and reading the posts in this nvidia forum. So, following some recommendations found in the forum I opened the file common.mk found in /Developer/GPU Computing/C/common and changed lines 39 and 306 to the following:

39: SNOWLEOPARD = $(strip $(findstring 10.6, $(shell egrep "<string>10\.6\.3" /System/Library/CoreServices/SystemVersion.plist)))
306: LINKLINE  = ar rucvs $(TARGET) $(OBJS)

Note the different version number of the OS and the s switch in the LINKLINE, which forces the creation of an index.

Then I tried to make clean and make again, but the error persisted. This time, I read this and found the solution, which is to run the ranlib command in the directories containing the libraries used during the compilation of the samples:

cd "/Developer/GPU Computing/C/lib"
ranlib *.a
cd "/Developer/GPU Computing/shared/lib"
ranlib *.a

By the way, apparently, the ranlib command creates a table of contents…

This got me rid of the errors.

Hope this helps.

References

• http://developer.nvidia.com/object/cuda_3_0_downloads.html
• http://forums.nvidia.com/index.php?showtopic=79189
• http://forums.nvidia.com/index.php?showtopic=105940&st=20&p=989833&#entry989833