Compute – Nova
Storage – Swift
Image Service – Glance

Here’s some quick steps to get a functioning nova dev install running on a single machine. I am using Ubuntu 10.10 and the instructions should be applicable to Ubuntu 10.04 as well. For the purposes of following through the steps a VM would suffice, but would be nice to have a devoted machine.

Start by following the instructions on the Dev Install page. Download the “nova.sh” script file and run the branch and install step.

./nova.sh branch

will install Bazaar and it’s dependencies if not already installed and download nova trunk, you can specify a branch if you want a branch other than trunk.

./nova.sh install

will install all the required packages for nova. A tty linux image is also downloaded as part of the install step. You can optionally download a CentOS image from the Dev Install page.
Now that everything is installed, you may want to add some configuration edits to the nova.sh file.
Here’s one I ran into to start with. If you don’t have the nova default volume group you may want to add your locally available volume group if you have lvm setup or add a fake volume driver. Locate a volume group if you have lvm setup locally.

sudo lvm vgs

--volume_group=nova
or
--volume_driver=nova.volume.driver.FakeISCSIDriver

Start all the processes. Before you run the command shore up on your “screen” skills.

sudo ./nova.sh run

This creates the admin user and admin project and creates a local sqlite database and also generates the certificates required for running euca2ools.

Now run euca2ools to check the registered images, by default the script registers the ami-tty image. Now run describe images to list all the available images

# euca-describe-images
 
IMAGE   ami-tty demo/tty        admin   available       public          x86_64  machine aki-tty ari-tty
IMAGE   ari-tty nova/tty-ramdisk        admin   available       public          x86_64  ramdisk
IMAGE   aki-tty nova/tty-kernel admin   available       public          x86_64  kernel

euca-add-keypair mykey > mykey.priv
chmod 600 mykey.priv

# euca-run-instances ami-tty --kernel aki-tty --ramdisk ari-tty --instance-type=m1.tiny --key=mykey
 
RESERVATION     r-exwgaqqh      admin   default
INSTANCE        i-00000001      ami-tty                 scheduling      mykey (admin, None)     0               m1.tiny 2011-02-09 13:57:54.417407      unknown zone

# euca-describe-instances
 
RESERVATION     r-exwgaqqh      admin   default
INSTANCE        i-00000001      ami-tty 10.0.0.3        10.0.0.3        running mykey (admin, dbaas)    0               m1.tiny 2011-02-09 13:57:54.417407      nova

to get the list of running instances and IP

Log in to the created VM

# ssh -i mykey.priv root@10.0.0.3
 
Chop wood, carry water.
 
#

That’s it you have your first vm created with nova.

Note: The project is changing a lot and these steps may not always continue working for you.

Job Templates

1. List all job templates (items created with the JOBCFG setting)

mdiag -j --flags=policy

2. Create a job template

mschedctl -m config --flags=persistent "JOBCFG[<template_name>] <template_parameters>"

Moab configuration

1. List current Moab configuation options

mschedctl -l config

2. Modify a configuration
The -e option will verify the configuration line without actually updating anything. If the --flags=persistent is left out, the configuration change will be lost when Moab is recycled.

mschedctl -m config --flags=persistent "<configuration_line>"

3. Check configuration validity

mdiag -C <config_file>

Reservations

1. Cancel Single Reservations

mrsvctl -r <Reservation Id>

Node Control

1. Offline a node, use state=idle to make the node available

mnodectl -m state=drained  <node_name>

Triggers

1. Delete a trigger

mschedctl -d trigger:<trigger_id>

Moab Adaptive HPC Suite is a complete solution to manage a HPC environment with complete support for workload management, job scheduling and an adaptive OS switcher for Linux & Windows workloads all rolled into one.

Moab Workload Manager is a highly advanced scheduling and management system designed for clusters, grids, and on-demand/utility computing systems. At a high level, Moab applies site policies and extensive optimizations to orchestrate jobs, services, and other workload across the ideal combination of network, compute, and storage resources. Moab enables true adaptive computing allowing compute resources to be customized to changing needs and failed systems to be automatically fixed or replaced. Moab increases system resource availability, offers extensive cluster diagnostics, delivers powerful QoS/SLA features, and provides rich visualization of cluster performance through advanced statistics, reports, and charts.

API

Moab provides an API with various bindings that support Job Control and Job Management functions. C, Perl and Java bindings are currently provided. Moab also has an extensive CLI that can also support XML that acts as a complete interface into Moab.

Enterprise Interoperability

Moab sports an easy to use, pluggable interface that allows integration with third party schedulers — provided the third party scheduler supports a CLI or API for Job Management functions. Moab can schedule, monitor, and manage jobs on external schedulers as well as provide a single view to Administrators. This allows for existing scheduling and resource management infrastructure to be managed through Moab.

Moab can also interface with multiple Resource Managers and schedule jobs on resources contained therein. While Moab makes the scheduling and allocation decisions, the Resource Managers provide Moab with input on current resource availability, but the Resource Manager itself is in charge of orchestrating the actual job staging and job execution. Moab, by default, supports Torque, SGE and Slurm. Other Resource Managers can be added with some integration code.

Moab also coordinates logical resources (database connections, HTTP connections, networks, license managers etc.). Moab can also pull information about a resource from multiple independent sources and aggregate them to provide a single status.

Multiple instances of Moab can be layered in a hierarchical fashion as well as be setup as peers with the ability to integrate the monitoring of the different peers/instances.

High availability and fail over

The Moab architecture provides complete support for High availability and there is no single point of failure. A fail over or backup instance of Moab can be created that would automatically take control if the primary instance fails. All jobs continue to run under their respective Resource Manager as well.

Internationalization and Localization

Moab does not support any form of Internationalization or localization.

Product Requirements

Moab can run on a minimal and shared hardware. The same applies for Torque and its agents. For ideal performance, however, the Moab server needs to be on a standalone node.

Job Definition

Moab supports creation of ad-hoc and predefined jobs through job templates, but Moab lacks some functionality in defining job templates for later reuse. Job templates support standard job parameters and resources that could be requested for a job and used in job matching. Creation/deletion of job templates can only be done through the configuration or CLI — it is not supported through the GUI. Moab supports the specification of various resource parameters during job submission: nodes, memory, cpu, generic resources, wall time, node features, start time, etc. Moab supports options for passing in runtime parameters to jobs, but they are limited in scope when used with job templates. Job Templates may have been updated with a later version, I didn’t have a chance to play around with the latest version.

Job Management

Moab provides all the basic job management functions such as start, stop, cancel, hold, restart, suspend/resume. All of these functions are available through the GUI, CLI, and API. Moab can capture the execution status of a job and use it for conditional branching in workflow jobs. It can also provide the user with the exit status.

Job Monitoring

Job status and stages can be monitored through the GUI, API and CLI. The job logs can only be viewed through the web interface or by logging into the scheduler node. The list of resources utilized by a job can be monitored through the GUI, API and CLI. Jobs can be viewed based on the following filters: user, groups, compute nodes, OS or architecture, queues and many more.

Platform and OS Support

Moab Server: Linux/Solaris on x86 and 64-bit
Resource Managers: Torque (Linux/Solaris), SGE (Linux/Solaris/Windows)
Desktop GUI: Any OS that has a JVM
Web GUI: Any modern web browser (client) and any OS that Java supports (server)

Prioritization and Queues

Moab has a full set of features for job prioritization. It supports priorities based on credentials, resources, usage, and job attributes. Priorities of jobs can be changed while the job is queued and user priorities can be provided at runtime. The effective priority of a job is the sum of the priorities of the various individual factors. Moab also automatically increases the priorities of jobs based on their queue time to avoid starvation.

Moab supports the concept of queues which act as a conduit for jobs into the system. Queues can be assigned resources and settings and provide functionality for prioritization, throttling, and effective utilization of the available resources. Authorization can be attached to the queues by providing credentials that may have access to the queues.

Reporting Capabilities

Moab has some reporting capabilities within the Java client. There are some basic charts on usage like total processors used, total wall time, average queue time of jobs, and number of jobs executed during a period in time. More advanced data can also be generated as reports that can be exported as PDF. All the data is available and can be exported into an external reporting system. The reporting capabilities with the Java Client are not very robust and lack proper visualization.

Reservation

Moab provides advanced capabilities for reserving resources for any period of time. Moab supports three types of reservations; single one time reservation, multiple recurring reservations, and infinite reservation. Moab guarantees the availability of the reserved resources when a reservation is started. The advanced reservations enable Moab to backfill jobs, provide deadline based scheduling, and QoS support. Recurring jobs can make use of reservations to guarantee availability of resources. Moab also provides support to kick-off jobs when a reservation starts.

Resource Management

Moab has a pluggable architecture for interfacing with external Resource Managers. Using Job templates, certain resource requirements can be matched based on the job properties. New resources can be defined, and depending on the specific Resource Manager, various actions such as adding/removing resources can take place. Resources can be assigned to queues or grouped based on custom attributes that can be attributed to a resource. Moab supports a feature to offline a resource for maintenance. That will prevent Moab from utilizing that resource for scheduling decisions.

Scalability and Performance

Moab is designed to run thousands of jobs per hour across thousands of nodes. Due to unavailability of resources, this could not be verified, the maximum I tested was with 64 nodes. Moab supports various configurations to serve the needs of the environment. Moab can be configured for long running jobs, or high throughput jobs, thereby providing appropriate response times and latency based on the configuration.

Scheduling

Moab has an extensive set of scheduling algorithms. It can schedule batch jobs, parallel jobs, and service workload. Moab’s support for parallel jobs relates directly to the parallel support provided by the Resource Managers. Torque and SGE both support parallel and array jobs.

Time based recurrence jobs are supported in Moab through the use of Triggers and Reservations. Using Reservations guarantees availability of resources and job start time — which might be critical for recurrence jobs. It also has very fine grained Calendar support but a lot of that functionality can only be achieved through the Moab configuration settings. This might limit users on their flexibility in calendar based scheduling. File arrival jobs are not supported directly in Moab, but the functionality can be achieved using triggers.

Moab supports various event based scheduling in the form of triggers. Some of the supported events are: threshold limits in reservations, jobs hold/preempt scenarios, and scheduler events. Moab can schedule workflows (which can be a group of jobs). Moab supports workflow creation through templates, API and custom processes.

Security

Moab defers to the local system for authentication; hence LDAP or other authentication systems can be utilized. Moab provides fine grained authorization. However, it can only be configured within Moab. The complete framework (GUI, CLI and API) all utilize this one unified authentication and authorization mechanism. It supports 5 roles which can be customized to suit your authorization needs.

The communication between the GUI and Moab Scheduler is through SSH tunelling. Interaction between the Torque agents is through SSH and SCP but keys need to be enabled for password less access.

SLA/QoS based Scheduling

SLA and QoS is built into Moab’s core scheduling. SLA and QoS are provided through reservations, job backfilling, and for service workloads by automatically increasing the number of service instances to meet a specified SLA.

User Interface

Moab provides a thick Java Client and a Web Portal. The Web Portal is intended for end users. As such, it only provides job submission, monitoring, and the ability to create reservations. The Java Client has more functionality and is geared towards operations. The supported features are: Job submission, monitoring, resource monitoring, management of reservations, credentials, triggers, setting node features, creation of partitions, and reporting.

Dynamic Service Management

Moab has a feature to manage service workloads. A web farm or application server cluster is an example of a service workload. Moab can manage the size of the service clusters based on performance parameters that can be passed back from the service. It administers this workload through dynamic jobs. Moab, based on performance information and SLA for the services, can automatically grow/shrink the service nodes. This allows Moab to meet SLA needs while maximizing the utilization of the resources.

Node Provisioning / Hybrid environments

Moab can interface with xCat or VMWare for node provisioning. Moab, using this functionality, can manage a hybrid workload environment consisting of Linux and Windows jobs. Moab switches between either platforms based on requirements of the jobs in the queue.

Job Templates

Moab’s Job Template functionality seems to be new and not well integrated with the core job submission. For instance, a job script specified in the template does not get executed when the template is invoked. The main reason is that Moab’s view of the job template is just as a template and not as a complete job definition. The templates can only be created through the Moab configuration or the CLI.

Workflow definitions

The main problem is that there are several ways to define a workflow and not one consistent method that would work as expected. There’s no support for creating a workflow through the GUI, and it also lacks support to pass in a workflow as XML (which could, ideally, be generated from an external system). The core concept of workflows and job dependencies seems to be well defined in the core system, however the ability to create and execute one is missing.

Workflow submission

Submission of workflows using templates is not a good idea as each workflow template will have to be linked with an account or some other parameter. Dynamic workflows can be created using Triggers and System Jobs, but there’s no facility to create and submit such workflows.

GUI

The functionality available in the Java client is not full featured. Certain job management functions don’t work from the GUI. Only a limited subset of Reservations and Triggers functionality is supported from the GUI. Job Templates and workflows cannot be created/deleted from the GUI.

The iPad is stirring up a lot of interest in the area and I believe we will have some nice apps that solve the purpose. One such app is from a company called Zinio. Zinio has been around for nearly 10 years, and already has a fairly good amount of digital magazines available for subscription. It however did not have the 2 or 3 magazines I wanted the most, National Geographic, National Geographic Traveler and Wired. It just started digitized  issues on Traveler, and a first issue of National Geographic, which apart from the regular magazine content has a section of interactive content. This first edition is free and can be accessed at “Water: Our Thirsty World“. Wired is getting an app ready for the iPad and there’s a demo of such an app for Sports Illustrated in the videos below.

What I’m looking for in interactive digital magazines:

• The above videos sum up most of what I want, being able to interact with the content.
• I’m looking for additional images and related videos for an article.
• Extra content and interactive maps is what I’m expecting from say Traveler and National Geographic.
• The 360 product views as shown in Wired is neat.
• Additionally I should be able to access additional content provided as part of the article within the context of the app. Links to external sites also needs to be accessible seamlessly. This won’t happen with the iPad though – no multitasking.

Why I want interactive digital magazines:

• I intend to keep my best magazines with me for a long time. Ex. Two years ago I did a flush out of all my magazines, but kept National Geographic, just didn’t want to send it for recycling. They do tend to pile up year after year. With digital versions, I wouldn’t face that problem.
• More and more external web resources are referenced in the content in magazines, I want to be able to do a lookup on those references right when I’m reading them. In most cases when I’m reading a magazine, I’m not in front of the system. Well you’d say reading the interactive magazines on a laptop/pc would give you that convenience, but that’s where devices like the iPad come into play and would be handy.
• No more need to recycle/trash your old magazines, you keep them all.
• Go Green. More trees spared.
• With interactive content, more information can be relayed than your traditional magazine. A counter argument is, don’t websites provide the interactivity you are looking for. Yes and no, the web does provide the information, but not the interactivity and curated, edited content that magazines like National Geographic or Wired provide. Their respective sites don’t provide that, but that would be another medium for the interactive magazines.
• The ability to provide more media (images/video) and starters for more research into the topics.
• Instantly available, you get your magazine the day it’s published, no need to wait for it to be shipped to you.

For additional reading on the topic, checkout ReadWriteWeb’s and NYTimes articles.

Here I’m going to detail a quick how-to of getting Grid Engine installed and running on Windows hosts. This is more applicable for Windows XP and Windows Server 2003, some of additional prerequisites required on the Windows hosts are now standard in Windows Server 2008 and Windows 7.

Prerequisites

• Follow the detailed instructions http://wikis.sun.com/display/GridEngine/Microsoft+Services+for+UNIX for an in-depth understanding.
• You must disable Data Execution Prevention (DEP). DEP is not compatible with some parts of SFU and might cause segmentation faults. See http://support.microsoft.com/kb/875352 for more information about DEP. To disable DEP, see http://wikis.sun.com/display/GridEngine/Disabling+DEP. Note this may not be the case for Windows 2008 Server and newer releases.

Installation of SFU

• Download Windows SFU 3.5 http://technet.microsoft.com/en-us/interopmigration/bb380242.aspx
• Create a copy of the passwd and group file from your master node with only entries required for User mapping on Windows. This will be used in one of the installation screens.
• Refer to http://wikis.sun.com/display/GridEngine/Microsoft+Services+for+UNIX#MicrosoftServicesforUNIX-ServicesforUNIXInstallation for detailed screen shots.
• Choose Custom install and choose Remote Services and leave the rest as is
• Enter the correct path of your passwd and group files for User mapping.
• Complete the installation and restart.
• Some notes to take, Interix is the subsystem for SFU. So with SFU installed, both the Win32 and Interix subsystem will run parallel to each other.
• Also note that any windows application can be invoked from with the Interix shell.

Additional but not required

• Install the bootstrap installer from Interop. Follow the directions here. http://www.interopsystems.com/tools/pkg_install.htm
• If your system doesn’t have access to the Internet download the packages from ftp://ftp.interopsystems.com/pkgs/3.5/ to your local machine
• For adding new packages, set PKG_PATH to the packages directory on your local machine.
• Mostly it can be used for adding Bash, because SFU comes with default ksh and csh.
• The binaries are installed in /usr/local/bin

Post Installation of SFU / GE installation prep work

• The instructions here can be followed http://wikis.sun.com/display/GridEngine/Microsoft+Services+for+UNIX#MicrosoftServicesforUNIX-PostSFUInstallationTasks , but the instructions provided below will also be useful.
• Open the Control Panel -> Administrative Tools -> Services and check that Telnet and Remote Shell Service is disabled. We will need to run telnet and RSH from Interix.
• Uncomment the lines containing telnet and shell from /etc/inetd.conf through one of the Interix shells.
• Restart inetd from /etc/init.d/inet start/stop
• Check the Windows Firewall, if its off there’s nothing to do, if its off and group enabled or if enabled do the following:
  • Add Exceptions for TCP port 23 and 514 for telnet and remote shell access required for GE.
  • Also add an exception for the Grid Engine Execution daemon port, in this case 6444.
  • Do a nmap to check that ports 23 and 514 are accessible.
• User name mapping is an important step. This instructions may vary if there’s a domain controller.
• Map the user names from the passwd file we used earlier and Windows users
  • Refer http://wikis.sun.com/display/GridEngine/User+Management+for+Sun+Grid+Engine+on+Windows+Hosts for a detailed understanding of User management on Windows for Grid Engine.
  • Start -> All Programs -> Windows Services for UNIX -> Service for UNIX Administration -> User Name Mapping
  • Depending on your configuration you may have choose NIS or Password and Group Files. Either of those options are fine.
  • Click on the Maps tab, check Simple Maps and choose the appropriate Windows domain name.
  • Under Advanced Maps click on Show User Maps. Click on List Windows Users and List UNIX Users
  • Now just map the Windows user to the appropriate UNIX user name.
  • Make sure the following mapping is present (Windows User) Administrator -> (UNIX User) root
  • Map any other required users.
• Check the users home directories and map them accordingly and create a profile for users if needed.
  • Control Panel -> Administrative Tools -> Computer Management -> Users -> Properties -> Profile

Pre Grid Engine Installation

Following on Windows Host

• Create a directory for Grid Engine installs, typically /opt/gridengine
• Export the following environment variables, add it to /etc/profile.lcl for system wide settings.

  export SGE_ROOT=/opt/gridengine
  export SGE_QMASTER_PORT=6445
  export SGE_EXECD_PORT=6444

• If you are going to be running multiple instances of the execution daemon, then don’t set these environment wide, rather set it for each instance, as each instance needs to use a different port.
• Add the Master node and the current node in /etc/hosts

Following on Master Node

• Run sgepasswd on the Master node and set the passwords for the Windows users. This is used by G! E for shell logins.
  • When you do "sgepasswd user", it sets the password for user on the default domain.
  • Use "sgepasswd -D domain user" to set the password for a user of a specific domain.
• Make sure the Windows host is added as an administrative host.
  • Run "qconf -sh" to check if the Windows host is already an administrative node
  • Run "qconf -ah hostname" to add the Windows host.
• Type "qconf -mconf" and set the execd_params to enable_windomacc=true.
• Add the Windows Admin name
  • qconf -am Administrator
• Generate certificates for the Windows users and also the certificates and keys used for encryption needs to be copied over to the Windows machine
  • This page provides detailed info – http://wikis.sun.com/display/GridEngine/Installing+Security+Features#InstallingSecurityFeatures-CSP
  • The advanced security can also be used for other hosts, but is a must to run jobs on Windows.
  • Generate certificates for a new user
    • $SGE_ROOT/util/sgeCA/sge_ca -user <win_user_name>
  • Follow Step 6 in the above link

NFS Setup

• Use this as guidance. http://wiki.gridengine.info/wiki/index.php/Install_and_configure_Grid_Engine_in_heterogenic_environment_on_Linux_and_Windows_with_MPICH2#! NFS_on_Windows
• Create a soft link from X:\ or any mount you choose. It available in /dev/fs/X.

Grid Engine Installation

• Copy and extract the Grid Engine binaries, common files and the Windows specific binaries to $SGE_ROOT. Optionally a NFS mount could be setup.
• Copy the folder containing in CELL_NAME in most cases default to $SGE_ROOT
• If you get an error with cannot resolve host name, add the windows hostname and IP in the /etc/hosts file.
• Follow the installation instructions at http://wikis.sun.com/display/GridEngine/Printable+Installing+Sun+Grid+Engine+Guide#PrintableInstallingSunGridEngineGuide-HowtoInstallExecutionHosts for installing the Execution daemon.

The Abstract and Full Text can be accessed here. Genome Biology and Evolution