Global users and groups are really useful for handing authentication and authorization at a global level, especially when interacting with things like Gerrit and other global services. Global users can also be used as service accounts within instances, between instances or between projects. There’s a number of downsides to global users though:

1. The global user creation process is laborious.
2. Global users must provide an email address.
3. Global users have a fixed home directory in /home (which is an autofs NFS or GlusterFS mount).
4. Global users can authenticate to all services, even if that’s not necessary or wanted.
5. Global users must have shell rights and must be added to a project to be properly usable for data access inside of a project.
6. Users get confused when told to create Labs accounts to be used as service accounts.
7. If multiple users want to access a global user, it’s necessary for the credentials to be shared, or to have a project admin create a sudo policy.
8. Global users don’t have personal groups, but instead have global groups, which are projects. Limiting access to data for a global user is difficult.

It’s also possible to define system users and groups via puppet and have those applied to instances within a project. There’s one major downside to this though: the changes would need to go through review first, which bottlenecks the process to the Operations team, and muddies up the puppet repository.

With the introduction of the Tools project, as a Toolserver replacement, there was a really strong need for service users and groups that could be handled in a simple way. Our Tools project implementer, Marc-Andre Pelletier, had a novel concept: per-project users and groups.

The concept and LDAP implementation

Marc’s initial concept was to have another set of OUs, like ou=project-people and ou=project-groups, where we’d create sub-OUs per project. We adjusted the concept when I showed Marc how we’re currently handling per-project sudo, though. Rather than using separate top-level OUs, we further extended the directory information tree (DIT) used by the OpenStack projects. Our OU for projects is ou=projects,dc=wikimedia,dc=org and the tools project OU is ou=tools,ou=projects,dc=wikimedia,dc=org. The extension for service groups also uses the sudo extension, and here’s the basic structure:

dn: cn=tools,ou=projects,dc=wikimedia,dc=org
objectClass: groupofnames
objectClass: extensibleobject
objectClass: top
cn: tools
info: servicegrouphomedirpattern=/data/project/%u
member: <member-list-goes-here>

# Single role, allowed to manage all project info
dn: cn=projectadmin,cn=tools,ou=projects,dc=wikimedia,dc=org
cn: projectadmin
roleOccupant: <member-list-goes-here>
objectClass: organizationalrole
objectClass: top

dn: ou=sudoers,cn=tools,ou=projects,dc=wikimedia,dc=org
ou: sudoers
objectClass: organizationalunit
objectClass: top

dn: ou=groups,cn=tools,ou=projects,dc=wikimedia,dc=org
ou: groups
objectClass: organizationalunit
objectClass: top

dn: ou=people,cn=tools,ou=projects,dc=wikimedia,dc=org
ou: people
objectClass: organizationalunit
objectClass: top

Note: for the project definition we have a pretty ugly hack. We’re sticking configuration information into the OU using the info attribute and extensibleobject. We’re doing this so that both cli tools and the web interface can know how to handle service groups. We should define our own schema and extend the object properly, but this was a quick and dirty hack for handling it to meet a hackathon deadline.

Allowing the addition of service groups is easy enough. When a service group is requested, a user and group are added to ou=people and ou=groups respectively. However, this doesn’t make the service groups easily accessible. To solve this, we automatically add a project sudo policy for every service group. Here’s a single service group added to the DIT:

dn: cn=local-example,ou=groups,cn=tools,ou=projects,dc=wikimedia,dc=org
objectClass: groupofnames
objectClass: posixgroup
objectClass: top
gidNumber: 100000
member: <member-list-goes-here>
cn: local-example

dn: uid=local-example,ou=people,cn=tools,ou=projects,dc=wikimedia,dc=org
objectClass: person
objectClass: shadowaccount
objectClass: posixaccount
objectClass: top
uid: local-example
cn: local-example
sn: local-example
loginShell: /usr/local/bin/sillyshell
homeDirectory: /data/project/example/
uidNumber: 100000
gidNumber: 100000

dn: cn=runas-local-example,ou=sudoers,cn=tools,ou=projects,dc=wikimedia,dc=org
sudoOption: !authenticate
objectClass: sudorole
objectClass: top
sudoRunAsUser: local-example
sudoCommand: ALL
sudoUser: %local-example
cn: runas-local-example
sudoHost: ALL

Project members can create service groups. The project member that creates the service group is the initial service group member. Members of the service group are allowed to add other members. All service group members can sudo to the service group without authentication.

The user and group have the same name, and the same uid/gid number. The uid and gid range were meant to be reserved and be allowed to overlap with service groups in other projects. Similarly, the user and group names were prefixed with local- and were to be allowed to overlap with service groups in other projects. The uid/gid range and non-unique user/group names design has changed since initial implementation, but I’ll get into that later.

The instance implementation

We’re using nslcd, which has support for defining multiple OUs for available services (passwd, shadow, group, etc.). To make service groups available on instances in a project, we use a similar approach as per-project sudo. Puppet has a variable defined for the OpenStack project, which is then used when adding the service group specific OUs to nslcd.conf:

# root
base dc=wikimedia,dc=org

# Global users and groups
base passwd ou=people,dc=wikimedia,dc=org
base shadow ou=people,dc=wikimedia,dc=org
base group ou=groups,dc=wikimedia,dc=org

# Per-project users and groups (service groups)
base passwd ou=people,cn=tools,ou=projects,dc=wikimedia,dc=org
base shadow ou=people,cn=tools,ou=projects,dc=wikimedia,dc=org
base group ou=groups,cn=tools,ou=projects,dc=wikimedia,dc=org

That there’s a pretty massive gotcha here: if any of the OUs don’t exist for a service, it breaks the entire service. It’s very important that the OUs exist for all projects in which this will be used.

After-implementation changes and further changes to come

We added service groups prior to the tools project ramp-up, which was in April 2013. Since then we’ve made some modifications from the original design.

The first issue we ran into was with non-globally-unique uids and gids. Users can have a large number of groups, due to multiple project membership and service group membership. We’re providing per-project NFS shares, and NFS has a 16 group limitation defined in its RFC. A way around this is to use the –manage-gids option in rpc.mountd. With manage-gids, the NFS server does the secondary group lookup and ignores whatever the client sends to it. Here’s where the problem with non-unique uids and gids comes in. The server needs to know about all groups in all projects. We sync group info from LDAP into the NFS server’s local groups file, and we can name the groups whatever we want, but for multi-tenancy purposes the uids and gids need to be unique. Also, in general, making the uids and gids unique makes it easier to integrate other services that aren’t natively multi-tenant.

The second issue is with non-unique service group names. It’s easier to integrate service groups into services that aren’t natively multi-tenant if the service group names are unique as well. We’ll be changing the naming scheme from being prefixed with local- to being prefixed with <projectname>- or some variant of that. We have a specific use case in mind, using Gerrit, for this change, but I’ll write a follow-up post about that.

So, what’s new with the wiki?

1. All articles now have discussion pages
2. It’s possible to make PDFs out of individual pages or to create a book (as a PDF or an actual physical book) from collections of articles
3. Uploads are global and can be used in multiple articles
4. Templates can be written using Lua
5. Gadgets can be written using Javascript and CSS and shared with all wiki users
6. Layout of articles can use Twitter’s Bootstrap, thanks to the strapping-mediawiki skin
7. There’s a mobile view, though mobile device detection won’t be enabled until next Wikimedia branch-point release (1-2 weeks)
8. Many more features available in MediaWiki that don’t exist in MoinMoin

Let me know if there’s any issues you run into with the new wiki.

Extending the flatdhcp network can be kind of a pain in the ass, so here’s how I handled it:

Assumptions

• Network before extension:
  • Network CIDR: 10.4.0.0/24
  • Broadcast: 10.4.0.255
  • Netmask: 255.255.255.0
  • Network ID: 2
• Network after extension:
  • Network CIDR: 10.4.0.0/21
  • Broadcast: 10.4.7.255
  • Netmask: 255.255.248.0
  • Network ID: 2

Modify the network

First modify the network via the database:

mysql nova -e "UPDATE networks SET netmask=\"255.255.248.0\",cidr=\"10.4.0.0/21\",broadcast=\"10.4.7.255\" WHERE id=2;"

Add the fixed IPs

Now it’s necessary to add all of the IP addresses in the range into the fixed_ips table. Additionally, the broadcast address in the original range should be modified so that it’s no longer reserved, and the new broadcast address should be marked as reserved.

for i in {1..7}
do
    for j in {0..255}
    do
        mysql nova -e "INSERT INTO fixed_ips SET created_at=\"2012-10-01 19:24:21\",updated_at=\"2012-10-01 19:24:21\",deleted=0,address=\"10.4.${i}.${j}\",network_id=2,allocated=0,reserved=0,leased=0"
    done
done
mysql nova -e "UPDATE fixed_ips SET reserved=0 WHERE address=\"10.4.0.255\""
mysql nova -e "UPDATE fixed_ips SET reserved=1 WHERE address=\"10.4.7.255\""

Restart nova-network and nova-compute services

I tried launching some instances after making this change, and got the following error popping up in my logs:

2012-10-01 20:04:48 TRACE nova.rpc.amqp DetachedInstanceError: Parent instance <Instance at 0x46fa150> is not bound to a Session; lazy load operation of attribute 'instance_type' cannot proceed

In the #openstack channel, zynzel mentioned that it’s because I needed to restart my nova-network service. Actually, I needed to restart all nova-network and all nova-compute services.

I ran into a likely unrelated issue during this as well. My nova-compute services were deadlocked. I’ve actually noticed this in the past as well. Clearing the lock files from /var/lock/nova then restarting the services fixed that issue, though I still need to trace this issue down.

Remove the old gateway addresses

The old gateway addresses, with the /24 CIDR, need to be removed from the bridge and from the routing table on the network nodes.

Restart dnsmasq and nova-network

After removing the addresses, it’s necessary to restart dnsmasq. Kill the processes, then restart nova-network again.

Why doesn’t Nova and Quantum have this functionality?

Neither Nova, nor Quantum seem to have operations to modify a network. It’s not a completely abnormal task to need to extend a network, to re-vlan it, or to change IP ranges. Hell, I still need to add IPv6 to my network and I need to make it multi-network-node; when I created the network neither of these features existed and there’s no way simple way to enable them now.

You can delete and re-create a network, but what happens to IP address assignments? What happens to DNS entries that were created via the DNS plugin for Nova? We really need the ability to modify networks, not just create and delete them.

A user-unfriendly experience

The above steps don’t really seem very difficult, but the actual steps involved assume fairly involved knowledge of the code. When in the middle of doing this, things are stressful and things are failing. It’s pretty user-unfriendly.

Additionally, when I asked about this in the #openstack channel, I was treated like I was stupid for not wanting to muck around with the database. I was told that it was my fault for creating a network that was too small and that it isn’t Nova’s job to fix my mistakes. I was told that I should “use Windows”; meaning that I’m expecting the software to hold my noob-ish hand.

I can muck in the database and trace code, but I think if users are forced to do that then we’re failing from a usability perspective.

Fight for the users

Being an OpenStack user is difficult, currently. Unless you have an OpenStack developer on your team, it’s difficult to even run OpenStack, let alone migrate between versions. Many deployments will run into bugs in the stable version of OpenStack and getting those bugs fixed and moved into the stable branch is difficult.

Even if your team has a developer, the process for getting fixes into stable is more difficult than getting fixes into master. In fact, it’s at minimum twice as hard, since a requirement of getting a fix into stable is that it it must be fixed in master first. Additionally, all fixes require tests. My complaint isn’t necessarily about the process, but about how there’s not much support wrapped around it.

It would be ideal to have a team that helps developers through the process of getting stable fixed. Additionally, the team should fix bugs on behalf of users who can’t fix the bugs themselves.

Support the support team

There’s a number of core infrastructure services that the community relies on: the blog, the wiki, Gerrit, Jenkins, etc. These services are supported by a great team sponsored by community members. Occasionally this team needs additional long-term or short-term resources, though. It’s not always easy to get a community member to sponsor contracts for development and support that don’t directly benefit them.

Additionally, we can’t fully rely on community members for core services. If a community member sponsors the majority of our core services, and later decides to leave the community, then the core services are at risk. We need to ensure we can keep the lights on, at minimum.

Provide resources to solve support issues

For both of the above issues, I’d like the foundation to reserve a portion of its budget to hire employees or contractors, and to buy hardware to help support the users and the support team.

The foundation should, of course, as a first priority encourage community members to provide needed resources, but it should also ensure that any gaps are covered, especially in regards to user engagement and ensuring the lights stay on. These must be top priorities if we want to continue to grow our community.

In the original design, being a project member in specific projects would automatically give you root via sudo and being a project member in a global project would give you shell, but not root. We were handling this through puppet configuration. This was a fairly limiting system. Giving fine grained permissions wasn’t easy. The instances knew which users were a member of a project since the projects were also posix groups; however, they didn’t know which users were in the roles of that project, so there was no fined grained way to handle this.

sudo-ldap to the rescue. With sudo-ldap, we can manage sudo policies in LDAP, and those can be done in a per-project basis. Let me explain how we’re handling this while also ensuring the original assumed design still applies to old projects.

Handling the sudo policies in LDAP

To make sudo work per-project, we need to make a sudoers OU for each project. Projects are located at ou=projects,dc=wikimedia,dc=org. We have an example project at cn=testproject,ou=projects,dc=wikimedia,dc=org. We can create a new sudoers OU for this project, with a default policy (for backwards compatibility):

dn: ou=sudoers,cn=testproject,ou=projects,dc=wikimedia,dc=org
ou: sudoers
objectclass: organizationalunit
objectclass: top

dn: cn=default,ou=sudoers,cn=testproject,ou=projects,dc=wikimedia,dc=org
cn: default
objectClass: sudorole
objectClass: top
sudoCommand: ALL
sudoHost: ALL
sudoUser: ALL

The above creates a sudoers OU underneath the project’s object and creates a default policy for that project that gives all users the ability to run all commands via sudo.

For every pre-existing specific project, I created an OU and a default policy, then for every pre-existing global project I only created the OU, ensuring everything continued working how things worked in the original design. Whenever a project is created the OU and a default policy is also now automatically created with the project.

Configuring sudo on the instances

Now we must configure the instances to pull their sudo policies from this OU. Here’s the puppet template we’re using for /etc/sudo-ldap.conf:

BASE            <%= basedn %>
URI             <% servernames.each do |servername| -%>ldap://<%= servername %>:389 <% end -%>

BINDDN          cn=proxyagent,ou=profile,<%= basedn %>
BINDPW          <%= proxypass %>
SSL             start_tls
TLS_CHECKPEER   yes
TLS_REQCERT     demand
TLS_CACERTDIR   /etc/ssl/certs
TLS_CACERTFILE  /etc/ssl/certs/<%= ldap_ca %>
TLS_CACERT      /etc/ssl/certs/<%= ldap_ca %>
<% if ldapincludes.include?('sudo') then %>SUDOERS_BASE    <%= sudobasedn %><% end %>

The sudobasedn variable is being set as this:

$sudobasedn = "ou=sudoers,cn=${instanceproject},ou=projects,${basedn}"

For a more in-context view, you can clone our repo, or browse it via gitweb.

Managing the sudo policies

In the trunk version of the OpenStackManager extension, I’ve added support for managing per-project sudo. Users must be a member of the sysadmin role to do so.

This release is mostly aimed at performance and usability. Here’s a list of changes:

• Added a project filter. Rather than showing all projects, only projects selected in the project filter will show in the management interfaces. This should make the interfaces contain far less text, and should make interfaces load much faster.
• Refactored the list pages so that styles can be applied to all pages easily. Applied a couple CSS styles globally across all of the pages. For instance, the table text has been changed to be top aligned, to make large tables easier to handle.
• Merged in Platonides’s change for handling SSH keys uploaded in formats other than OpenSSH format. Keys in non-OpenSSH format will automatically be converted, if possible. If a private key, or a key in a bad formatted is uploaded, it’ll be rejected.
• Changed the project section collapsing behavior. Rather than the project title collapsing the project’s section, a “Toggle” action will do so. The project name has been changed back to being a link to the project’s page.
• Projects are now sorted alphabetically everywhere.
• Various fixes related to the PHP aws-sdk.
• Move creation forms to list pages for many management pages, to avoid extra clicks where possible.
• Various memcache support additions and fixes.
• Added a fix to allow user creation through MediaWiki interface.

If you’d like to help develop this extension, I’ve created a development environment in a project in Wikimedia Labs. Find me on #wikimedia-labs on Freenode or email me to get a labs account and access to the project.

Of course, the instance wasn’t actually running anywhere and the reboot command wouldn’t start the instance, because it thought it was running. The logs complained that the instance wasn’t running whether I tried to restart the migration, or reboot. What a full of fail situation.

So, to fix this, I needed to make the instance actually start. In this situation, the database thought the instance was running on host virt2, but the instance’s libvirt files were on virt4. I copied the nwfilter file across to /etc/libvirt/nwfilter, then the domain file across to /etc/libvirt/qemu. I then created the nwfilter, then the domain:

virsh nwfilter-define /etc/libvirt/nwfilter/<instance-nwfilter>.xml
virsh create /etc/libvirt/qemu/<instance-domain>.xml

Once the instance was started, I re-migrated the instance and all was good.

As a side note, I think what caused the migration failure was that I tried to migrate too many instances at the same time from a host that was already slightly overloaded. Of course, this is no excuse for nova to fail.

I’ve been busy enough lately working on our OpenStack infrastructure that I haven’t made an OpenStackManager release in a while. Over the past seven months I’ve continued to make small changes to the software, and the past few weeks I’ve added features I feel deserve another release.

This is a bugfix and features release. Major changes include compatibility for cactus and diablo releases of nova, and 1.18 compatibility for MediaWiki. The changes in this release focused mainly on making workflow easier. Here’s a complete list of changes:

• Added a reboot action for instances
• Made compatibility changes for cactus and diablo nova releases
• Made compatibility changes for MediaWiki 1.18
• Added support for configurable naming attributes
• Added support for adding objectclasses and attributes for users that are missing them
• It’s now possible for MediaWiki to no longer have to create users, only update select user attributes and objectclasses
• Made a bunch of bugfixes regarding security groups
• Added support for wildcard DNS entries
• Added realm and instancename variables to puppet default variables, so that they can be used in puppet runs
• Added support for wiki page creation for Projects
• Added support for configuring default images for instances
• Added support for creating server admin logs per project
• Added support for default security group rules on project creation
• Added dialog to project creation for adding members to projects and roles upon creation
• Added support for managing puppet classes and variables through the interface, rather than LocalSettings.php
• Made a usability change to instance creation: the default security group will always be selected by default
• Added support for including the ssh key fingerprint of an instance in the “your instance is ready” emails
• Made a usability change to only show actions to users if they can perform them
• Lots of other minor bug fixes
• Bugfixes from John Du Hart, Sam Reed and Mark Hershberger

If you’d like to help develop this extension, I’ve created a development environment in a project in Wikimedia Labs. Find me on #wikimedia-labs on Freenode or email me to get a labs account and access to the project.

1. Create an instance and use it to do experimentation with the service.
2. Document the service, along with the installation process on wikitech, after ensuring the service is working properly.
3. Create a second instance. Following the documentation written, puppetize the service.
4. Create a third instance. Ensure the puppetized service runs properly when initialized from scratch.
5. Kill all three instances, and replace the instances in the test cluster.

When a service changes in puppet, follow the above cycle as well.

Using this process, I can be assured the puppet manifests, as written, will allow me to repeatedly install this service.

The problem with NFS home directories, however, is that they are fairly insecure. They can be used between instances to escalate privileges. In our environment, this isn’t a problem for instances within a project. If a user is a member of a project, they have shell on all instances. If they are given sudo access in a project, they are given sudo access on all instances. Between projects, however, is a problem. user-A with root on instance-A in project-A could su to to user-B on instance-A, modify the user’s authorized_keys file, and then have access to project-B if home directories are shared across projects.

To avoid cross-project escalation, each project needs its own set of home directories. This means we can’t simply export /home to the instance’s private network and be done with it. We’ll need to create an exports file, and share different directory trees with specific instances. We’ll also need to mount home directories differently on the instances, depending on the project they belong to.

To do so, we’ll use a combination of puppet, LDAP, autofs, and Nova.

Creating the exports file

To create the exports file we need three things:

1. A list of projects
2. A list of instances within each project
3. A list of home directory locations for each project

The first two can be found via LDAP. The query for a list of projects is: ‘(&(objectclass=groupofnames)(owner=*))’. The query for a list of instances within each project is: ‘(puppervar=instanceproject=<project>)’. Of course, this approach is only usable for people using the OpenStackManager extension for MediaWiki; I’ll mention more portable ways to get this information later in the post.

The third we can extrapolate, we just need a base directory. I chose to use /export/home/<project> for the locations.

I wrote a python script that will pull this information, and create an exports file that looks like this:

/export/home/<project1> <project1-instance1>(rw,no_subtree_check) <project-instance2>(rw,no_subtree_check) <project_instance...>(rw,no_subtree_check)
/export/home/<project2> <project2-instance1>(rw,no_subtree_check) <project2-instance2>(rw,no_subtree_check) <project2_instance...>(rw,no_subtree_check)

Mounting the shares from the instances

Each instance needs to mount the share, depending on its project. There’s a number of ways we can do this, but I like the flexibility of using autofs and LDAP to manage NFS mounts. To add slightly more flexibility we’ll involve the help of puppet as well.

In LDAP, we can create autofs entries by making maps and objects. The following objects add support for /home:

dn: nisMapName=auto.master,<basedn>
objectClass: top
objectClass: nisMap
nisMapName: auto.master

dn: nisMapName=auto.home,<basedn>
objectClass: top
objectClass: nisMap
nisMapName: auto.home

dn: nisMapName=/home,nisMapName=auto.master,<basedn>
objectClass: top
objectClass: nisObject
cn: /home
nisMapEntry: ldap:nisMapName=auto.home,<basedn>
nisMapName: auto.master

We also need to add entries for the specific home directories. Here we are going to invoke a little awesome magic that autofs has: variables. Here’s the entry we are using for all home directories in all projects:

dn: cn=*,nisMapName=auto.home,<basedn>
changetype: add
nisMapEntry: ${SERVNAME}:${HOMEDIRLOC}/&
objectClass: nisObject
objectClass: top
nisMapName: auto.home
cn: *

We only need the one entry, which saves us from having to create and delete entries on creation and deletion of projects. Using this, however, means we need to set the variables. This is where puppet comes in. First, though, let’s look at the node in LDAP:

dn: dc=i-0000005c,dc=pmtpa,ou=hosts,dc=wikimedia,dc=org
objectClass: domainrelatedobject
objectClass: dnsdomain
objectClass: domain
objectClass: puppetclient
objectClass: dcobject
objectClass: top
puppetVar: realm=labs
puppetVar: writable=false
puppetVar: db_cluster=s1
puppetVar: instancecreator_email=rlane@wikimedia.org
puppetVar: instancecreator_username=Ryan Lane
puppetVar: instancecreator_lang=en
puppetVar: instanceproject=testlabs
puppetClass: base
puppetClass: ldap::client::wmf-test-cluster
puppetClass: exim::simple-mail-sender
puppetClass: db::core
puppetClass: mysql::mysqluser
puppetClass: mysql::datadirs
puppetClass: mysql::conf
l: pmtpa
associatedDomain: i-0000005c.pmtpa.wmflabs
associatedDomain: labs-db2.pmtpa.wmflabs
dc: i-0000005c
aRecord: 10.4.0.12

All the above objectclasses and attributes are available for use in puppet. The really important one here is instanceproject=testlabs.

We can set the autofs variables via the OPTIONS variable in the /etc/default/autofs file:

OPTIONS=”-DSERVNAME=<%= nfs_server_name %> -DHOMEDIRLOC=<%= homedir_location %>”

Here SERVNAME and HOMEDIRLOC autofs variables are being set. nfs_server_name and homedir_location are being set via a puppet template. Both are being determined via a manifest:

$homedir_location = "/export/home/${instanceproject}"

nfs_server_name is a hash, based on the project:

$nfs_server_name = $instanceproject ? {
	default => "labs-nfs1",
}

I chose to use a hash based on project so that I can choose to separate the server based on project as well, if needed for performance, or extra security.

Managing user home directories

Everything up to this point is just creating the shares. However, we must maintain the users’ home directories as well. For this, we need to know which users are in which projects, and we need to manage their home directories per project.

I wrote a script to search for users, based on a group (the project), that selectively creates/deletes/renames home directories and authorized_keys files. I should note here that I don’t use nova’s mechanisms for SSH key management, as it isn’t portable between applications. I instead store the keys in the user’s LDAP entry.

There’s a security issue with management of home directories. If a user is added to a project and we create a home directory, with a populated authorized_keys file, then remove the user from the project, but don’t remove their home directory, the user will still have access to the project’s instances. There’s two ways I go about solving this issue:

1. Ensure the user only has access to the instance if they are in the project, using access.conf. In my architecture, when projects are added, they are also made a posixgroup, with a gid. Thanks to this, we can treat the project as a system group in all instances. In access.conf we limit access to the project group.
2. User’s home directories are moved from /export/home/<project>/<user> to /export/home/<project>/SAVE/<user> when they are removed from the project.

Problems with this solution, and future improvements to make

The major shortcoming of this solution is that it isn’t terribly portable. It is dependent on using LDAP, and storing specific information in the LDAP directory. Using the nova tools, or having nova manage the exports on instance creation/deletion would make this a much more portable solution.

Another shortcoming is that it isn’t terribly scalable. The exports file is being created from scratch every single script run (which needs to happen fairly frequently). Ideally, nova would write to a queue, and the NFS instance would add/remove instances from the exports as instances are created/deleted.

Thankfully, I didn’t have a shortage of ideas about how to accomplish this, as shown in my proposal. I decided upon the quick and dirty approach, opting to do one of the more reusable approaches later. I’ll likely add support to nova for this at some time in the future.