So, what’s new with the wiki?

1. All articles now have discussion pages
2. It’s possible to make PDFs out of individual pages or to create a book (as a PDF or an actual physical book) from collections of articles
3. Uploads are global and can be used in multiple articles
4. Templates can be written using Lua
5. Gadgets can be written using Javascript and CSS and shared with all wiki users
6. Layout of articles can use Twitter’s Bootstrap, thanks to the strapping-mediawiki skin
7. There’s a mobile view, though mobile device detection won’t be enabled until next Wikimedia branch-point release (1-2 weeks)
8. Many more features available in MediaWiki that don’t exist in MoinMoin

Let me know if there’s any issues you run into with the new wiki.

Related posts:

1. Fixing a very broken instance live migration manually
2. Helpdesk system and datacenter inventory Semantic MediaWiki prototypes added to my prototype wiki
3. New MediaWiki extension JSBreadCrumbs 0.1 released

Extending the flatdhcp network can be kind of a pain in the ass, so here’s how I handled it:

Assumptions

• Network before extension:
  • Network CIDR: 10.4.0.0/24
  • Broadcast: 10.4.0.255
  • Netmask: 255.255.255.0
  • Network ID: 2
• Network after extension:
  • Network CIDR: 10.4.0.0/21
  • Broadcast: 10.4.7.255
  • Netmask: 255.255.248.0
  • Network ID: 2

Modify the network

First modify the network via the database:

mysql nova -e "UPDATE networks SET netmask=\"255.255.248.0\",cidr=\"10.4.0.0/21\",broadcast=\"10.4.7.255\" WHERE id=2;"

Add the fixed IPs

Now it’s necessary to add all of the IP addresses in the range into the fixed_ips table. Additionally, the broadcast address in the original range should be modified so that it’s no longer reserved, and the new broadcast address should be marked as reserved.

for i in {1..7}
do
    for j in {0..255}
    do
        mysql nova -e "INSERT INTO fixed_ips SET created_at=\"2012-10-01 19:24:21\",updated_at=\"2012-10-01 19:24:21\",deleted=0,address=\"10.4.${i}.${j}\",network_id=2,allocated=0,reserved=0,leased=0"
    done
done
mysql nova -e "UPDATE fixed_ips SET reserved=0 WHERE address=\"10.4.0.255\""
mysql nova -e "UPDATE fixed_ips SET reserved=1 WHERE address=\"10.4.7.255\""

Restart nova-network and nova-compute services

I tried launching some instances after making this change, and got the following error popping up in my logs:

2012-10-01 20:04:48 TRACE nova.rpc.amqp DetachedInstanceError: Parent instance <Instance at 0x46fa150> is not bound to a Session; lazy load operation of attribute 'instance_type' cannot proceed

In the #openstack channel, zynzel mentioned that it’s because I needed to restart my nova-network service. Actually, I needed to restart all nova-network and all nova-compute services.

I ran into a likely unrelated issue during this as well. My nova-compute services were deadlocked. I’ve actually noticed this in the past as well. Clearing the lock files from /var/lock/nova then restarting the services fixed that issue, though I still need to trace this issue down.

Remove the old gateway addresses

The old gateway addresses, with the /24 CIDR, need to be removed from the bridge and from the routing table on the network nodes.

Restart dnsmasq and nova-network

After removing the addresses, it’s necessary to restart dnsmasq. Kill the processes, then restart nova-network again.

Why doesn’t Nova and Quantum have this functionality?

Neither Nova, nor Quantum seem to have operations to modify a network. It’s not a completely abnormal task to need to extend a network, to re-vlan it, or to change IP ranges. Hell, I still need to add IPv6 to my network and I need to make it multi-network-node; when I created the network neither of these features existed and there’s no way simple way to enable them now.

You can delete and re-create a network, but what happens to IP address assignments? What happens to DNS entries that were created via the DNS plugin for Nova? We really need the ability to modify networks, not just create and delete them.

A user-unfriendly experience

The above steps don’t really seem very difficult, but the actual steps involved assume fairly involved knowledge of the code. When in the middle of doing this, things are stressful and things are failing. It’s pretty user-unfriendly.

Additionally, when I asked about this in the #openstack channel, I was treated like I was stupid for not wanting to muck around with the database. I was told that it was my fault for creating a network that was too small and that it isn’t Nova’s job to fix my mistakes. I was told that I should “use Windows”; meaning that I’m expecting the software to hold my noob-ish hand.

I can muck in the database and trace code, but I think if users are forced to do that then we’re failing from a usability perspective.

Related posts:

1. Announcing OpenStackManager extension for MediaWiki
2. OpenStackManager version 1.2 released
3. Sharing home directories to instances within a project using puppet, LDAP, autofs, and Nova

Fight for the users

Being an OpenStack user is difficult, currently. Unless you have an OpenStack developer on your team, it’s difficult to even run OpenStack, let alone migrate between versions. Many deployments will run into bugs in the stable version of OpenStack and getting those bugs fixed and moved into the stable branch is difficult.

Even if your team has a developer, the process for getting fixes into stable is more difficult than getting fixes into master. In fact, it’s at minimum twice as hard, since a requirement of getting a fix into stable is that it it must be fixed in master first. Additionally, all fixes require tests. My complaint isn’t necessarily about the process, but about how there’s not much support wrapped around it.

It would be ideal to have a team that helps developers through the process of getting stable fixed. Additionally, the team should fix bugs on behalf of users who can’t fix the bugs themselves.

Support the support team

There’s a number of core infrastructure services that the community relies on: the blog, the wiki, Gerrit, Jenkins, etc. These services are supported by a great team sponsored by community members. Occasionally this team needs additional long-term or short-term resources, though. It’s not always easy to get a community member to sponsor contracts for development and support that don’t directly benefit them.

Additionally, we can’t fully rely on community members for core services. If a community member sponsors the majority of our core services, and later decides to leave the community, then the core services are at risk. We need to ensure we can keep the lights on, at minimum.

Provide resources to solve support issues

For both of the above issues, I’d like the foundation to reserve a portion of its budget to hire employees or contractors, and to buy hardware to help support the users and the support team.

The foundation should, of course, as a first priority encourage community members to provide needed resources, but it should also ensure that any gaps are covered, especially in regards to user engagement and ensuring the lights stay on. These must be top priorities if we want to continue to grow our community.

Related posts:

1. Thoughts on OpenStack Foundation
2. I’ve been with the Wikimedia Foundation for a second year. Have I met my goals?
3. Building a test and development infrastructure using OpenStack

1. Continue with the Labs project. Finish set up of test/dev Labs, and begin work and make major progress on tool Labs.
   • Partial success: Test/dev Labs is going really well. At the time of this writing we have 99 projects, 174 instances, and 446 users. We have per-project nagios, ganglia, puppet, and sudo. We also have an all-in-one MediaWiki puppet configuration. We currently have one zone with 5 compute nodes, and will mostly triple the capacity of that in the next month. We have another zone coming up in another datacenter that will be 8 large compute nodes. Stability is still currently a concern, and we haven’t come out of closed beta, yet, though. Also, work on Tool Labs is mostly not started. We do have a bots cluster that’s community managed, but we don’t have database replication and don’t have a simple way for tool authors to contribute.
2. Hire a devops contractor for work on Labs.
   • Success: Not only did we hire a devops contractor, we built a larger team. We now have Andrew Bogott (developer), Sara Smollett (operations), Faidon Liambotis (operations) and myself (operations).
3. Build a devops community around the Wikimedia architecture.
   • Success: We had roughly 700 changes pushed into the operations/puppet repository from people who are not operations team members. A number of our larger Labs projects were built by volunteers (bots, deployment-prep, nagios, for instance). Volunteers are members of most of the projects that exist in Labs.
4. Finish the HTTPS project. This will hopefully be complete from the ops perspective by the end of this year.
   • Partial success: HTTPS is fully enabled on all sites, for both IPv4 and IPv6. I’ve listed this as a partial success, because I’d like the default for logged-in users to be HTTPS. Also, I wanted secure.wikimedia.org to redirect properly to HTTPS by now, and haven’t found time to do so.
5. On-board new employees.
   • Success: We brought on a lot of new Operations Engineers last year and I helped on-board nearly all of them. That said, I wish I would have written more documentation on the process as I was doing it.
6. Enable OpenID as a provider and oAuth on Wikimedia (this goal still needs consensus).
   • Partial failure, again: That said, I’ve been pushing for oAuth very strongly internally and it looks like this is now a stated goal of next year! oAuth is crucial to the success of Labs, so I’m very happy this is happening.

What did I accomplish that was outside of my stated goals?

1. Installed Gerrit, moved our operations repositories from SVN to Git and released our puppet repository as open source and cloneable to the world.
2. Assisted the core services team with the migration from SVN to Git.
3. Launched Labs (in October 2011 at the New Orleans MediaWiki hackathon).
4. Wrote the OpenStackManager and OATHAuth MediaWiki extensions.
5. Massively refactored the LdapAuthentication MediaWiki extension.
6. Rewrote a couple IRC bots (ircecho and adminbot).
7. Wrote a new deployment system that may replace our production deployment system.
8. Did the operations portion of the SOPA blackout.
9. Organized the New Orleans MediaWiki hackathon.
10. Organized an OpenStack meetup held at the Wikimedia Foundation offices.
11. Pushed 790 changes into Gerrit.
12. Made 1,100 edits to labsconsole (those edits include project creations, modification of projects, creation/deletion of instances and actual writing of documentation).
13. Got the 100,000th revision in Wikimedia SVN, much to the dismay of others!

What are my goals for next year?

1. Stabilize Labs.
2. Add a second Labs zone in eqiad.
3. Make major progress on Tool Labs.
4. Add a real queue to the Wikimedia infrastructure, for jobs and other needs.
5. Continue building a solid community around Labs.
6. Continue to improve the HTTPS infrastructure.

Related posts:

1. I’ve been with the Wikimedia Foundation for a year. Have I met my goals?
2. Now full time Operations Engineer for Wikimedia Foundation

OATHAuth is an opt-in feature that adds more security accounts in a wiki. It provides two-factor authentication, using your phone as the something you have, and your username/password as the something you know. If you are using iPhone or Android, you can use the Google Authenticator app as a client. There are also clients for most other phones and desktops; Wikipedia has a good list of clients.

If you have an account in Wikimedia Labs, you can enable two-factor authentication via the sidebar now.

As of version 0.1, OATHAuth only works when chained with LDAPAuthentication. Version 0.2 will work in a standalone manner. See the following image gallery for how it’s used:

Related posts:

1. Announcing OpenStackManager extension for MediaWiki
2. Announcing the Plotters extension for MediaWiki
3. Semi-anonymous users in MediaWiki using the LDAP Authentication extension

In the original design, being a project member in specific projects would automatically give you root via sudo and being a project member in a global project would give you shell, but not root. We were handling this through puppet configuration. This was a fairly limiting system. Giving fine grained permissions wasn’t easy. The instances knew which users were a member of a project since the projects were also posix groups; however, they didn’t know which users were in the roles of that project, so there was no fined grained way to handle this.

sudo-ldap to the rescue. With sudo-ldap, we can manage sudo policies in LDAP, and those can be done in a per-project basis. Let me explain how we’re handling this while also ensuring the original assumed design still applies to old projects.

Handling the sudo policies in LDAP

To make sudo work per-project, we need to make a sudoers OU for each project. Projects are located at ou=projects,dc=wikimedia,dc=org. We have an example project at cn=testproject,ou=projects,dc=wikimedia,dc=org. We can create a new sudoers OU for this project, with a default policy (for backwards compatibility):

dn: ou=sudoers,cn=testproject,ou=projects,dc=wikimedia,dc=org
ou: sudoers
objectclass: organizationalunit
objectclass: top

dn: cn=default,ou=sudoers,cn=testproject,ou=projects,dc=wikimedia,dc=org
cn: default
objectClass: sudorole
objectClass: top
sudoCommand: ALL
sudoHost: ALL
sudoUser: ALL

The above creates a sudoers OU underneath the project’s object and creates a default policy for that project that gives all users the ability to run all commands via sudo.

For every pre-existing specific project, I created an OU and a default policy, then for every pre-existing global project I only created the OU, ensuring everything continued working how things worked in the original design. Whenever a project is created the OU and a default policy is also now automatically created with the project.

Configuring sudo on the instances

Now we must configure the instances to pull their sudo policies from this OU. Here’s the puppet template we’re using for /etc/sudo-ldap.conf:

BASE            <%= basedn %>
URI             <% servernames.each do |servername| -%>ldap://<%= servername %>:389 <% end -%>

BINDDN          cn=proxyagent,ou=profile,<%= basedn %>
BINDPW          <%= proxypass %>
SSL             start_tls
TLS_CHECKPEER   yes
TLS_REQCERT     demand
TLS_CACERTDIR   /etc/ssl/certs
TLS_CACERTFILE  /etc/ssl/certs/<%= ldap_ca %>
TLS_CACERT      /etc/ssl/certs/<%= ldap_ca %>
<% if ldapincludes.include?('sudo') then %>SUDOERS_BASE    <%= sudobasedn %><% end %>

The sudobasedn variable is being set as this:

$sudobasedn = "ou=sudoers,cn=${instanceproject},ou=projects,${basedn}"

For a more in-context view, you can clone our repo, or browse it via gitweb.

Managing the sudo policies

In the trunk version of the OpenStackManager extension, I’ve added support for managing per-project sudo. Users must be a member of the sysadmin role to do so.

Related posts:

1. Sharing home directories to instances within a project using puppet, LDAP, autofs, and Nova
2. Why I chose MediaWiki for my OpenStack Manager project
3. Building a test and development infrastructure using OpenStack

This release is mostly aimed at performance and usability. Here’s a list of changes:

• Added a project filter. Rather than showing all projects, only projects selected in the project filter will show in the management interfaces. This should make the interfaces contain far less text, and should make interfaces load much faster.
• Refactored the list pages so that styles can be applied to all pages easily. Applied a couple CSS styles globally across all of the pages. For instance, the table text has been changed to be top aligned, to make large tables easier to handle.
• Merged in Platonides’s change for handling SSH keys uploaded in formats other than OpenSSH format. Keys in non-OpenSSH format will automatically be converted, if possible. If a private key, or a key in a bad formatted is uploaded, it’ll be rejected.
• Changed the project section collapsing behavior. Rather than the project title collapsing the project’s section, a “Toggle” action will do so. The project name has been changed back to being a link to the project’s page.
• Projects are now sorted alphabetically everywhere.
• Various fixes related to the PHP aws-sdk.
• Move creation forms to list pages for many management pages, to avoid extra clicks where possible.
• Various memcache support additions and fixes.
• Added a fix to allow user creation through MediaWiki interface.

If you’d like to help develop this extension, I’ve created a development environment in a project in Wikimedia Labs. Find me on #wikimedia-labs on Freenode or email me to get a labs account and access to the project.

Related posts:

1. OpenStackManager version 1.3 released
2. OpenStackManager version 1.2 released
3. OpenStackManager version 1.1 released

Of course, the instance wasn’t actually running anywhere and the reboot command wouldn’t start the instance, because it thought it was running. The logs complained that the instance wasn’t running whether I tried to restart the migration, or reboot. What a full of fail situation.

So, to fix this, I needed to make the instance actually start. In this situation, the database thought the instance was running on host virt2, but the instance’s libvirt files were on virt4. I copied the nwfilter file across to /etc/libvirt/nwfilter, then the domain file across to /etc/libvirt/qemu. I then created the nwfilter, then the domain:

virsh nwfilter-define /etc/libvirt/nwfilter/<instance-nwfilter>.xml
virsh create /etc/libvirt/qemu/<instance-domain>.xml

Once the instance was started, I re-migrated the instance and all was good.

As a side note, I think what caused the migration failure was that I tried to migrate too many instances at the same time from a host that was already slightly overloaded. Of course, this is no excuse for nova to fail.

Related posts:

1. OpenStack wiki migration
2. Moved site from a shared host to an EC2 instance
3. Adding new hard drives live on an Ubuntu guest with VMware ESX

I’ve been busy enough lately working on our OpenStack infrastructure that I haven’t made an OpenStackManager release in a while. Over the past seven months I’ve continued to make small changes to the software, and the past few weeks I’ve added features I feel deserve another release.

This is a bugfix and features release. Major changes include compatibility for cactus and diablo releases of nova, and 1.18 compatibility for MediaWiki. The changes in this release focused mainly on making workflow easier. Here’s a complete list of changes:

• Added a reboot action for instances
• Made compatibility changes for cactus and diablo nova releases
• Made compatibility changes for MediaWiki 1.18
• Added support for configurable naming attributes
• Added support for adding objectclasses and attributes for users that are missing them
• It’s now possible for MediaWiki to no longer have to create users, only update select user attributes and objectclasses
• Made a bunch of bugfixes regarding security groups
• Added support for wildcard DNS entries
• Added realm and instancename variables to puppet default variables, so that they can be used in puppet runs
• Added support for wiki page creation for Projects
• Added support for configuring default images for instances
• Added support for creating server admin logs per project
• Added support for default security group rules on project creation
• Added dialog to project creation for adding members to projects and roles upon creation
• Added support for managing puppet classes and variables through the interface, rather than LocalSettings.php
• Made a usability change to instance creation: the default security group will always be selected by default
• Added support for including the ssh key fingerprint of an instance in the “your instance is ready” emails
• Made a usability change to only show actions to users if they can perform them
• Lots of other minor bug fixes
• Bugfixes from John Du Hart, Sam Reed and Mark Hershberger

If you’d like to help develop this extension, I’ve created a development environment in a project in Wikimedia Labs. Find me on #wikimedia-labs on Freenode or email me to get a labs account and access to the project.

Related posts:

1. OpenStackManager version 1.2 released
2. OpenStackManager 1.4 released
3. OpenStackManager version 1.1 released

1. Create an instance and use it to do experimentation with the service.
2. Document the service, along with the installation process on wikitech, after ensuring the service is working properly.
3. Create a second instance. Following the documentation written, puppetize the service.
4. Create a third instance. Ensure the puppetized service runs properly when initialized from scratch.
5. Kill all three instances, and replace the instances in the test cluster.

When a service changes in puppet, follow the above cycle as well.

Using this process, I can be assured the puppet manifests, as written, will allow me to repeatedly install this service.

Related posts:

1. Sharing home directories to instances within a project using puppet, LDAP, autofs, and Nova
2. Announcing OpenStackManager extension for MediaWiki
3. OpenStackManager version 1.3 released