Of course, the instance wasn’t actually running anywhere and the reboot command wouldn’t start the instance, because it thought it was running. The logs complained that the instance wasn’t running whether I tried to restart the migration, or reboot. What a full of fail situation.

So, to fix this, I needed to make the instance actually start. In this situation, the database thought the instance was running on host virt2, but the instance’s libvirt files were on virt4. I copied the nwfilter file across to /etc/libvirt/nwfilter, then the domain file across to /etc/libvirt/qemu. I then created the nwfilter, then the domain:

virsh nwfilter-define /etc/libvirt/nwfilter/<instance-nwfilter>.xml
virsh create /etc/libvirt/qemu/<instance-domain>.xml

Once the instance was started, I re-migrated the instance and all was good.

As a side note, I think what caused the migration failure was that I tried to migrate too many instances at the same time from a host that was already slightly overloaded. Of course, this is no excuse for nova to fail.

I’ve been busy enough lately working on our OpenStack infrastructure that I haven’t made an OpenStackManager release in a while. Over the past seven months I’ve continued to make small changes to the software, and the past few weeks I’ve added features I feel deserve another release.

This is a bugfix and features release. Major changes include compatibility for cactus and diablo releases of nova, and 1.18 compatibility for MediaWiki. The changes in this release focused mainly on making workflow easier. Here’s a complete list of changes:

• Added a reboot action for instances
• Made compatibility changes for cactus and diablo nova releases
• Made compatibility changes for MediaWiki 1.18
• Added support for configurable naming attributes
• Added support for adding objectclasses and attributes for users that are missing them
• It’s now possible for MediaWiki to no longer have to create users, only update select user attributes and objectclasses
• Made a bunch of bugfixes regarding security groups
• Added support for wildcard DNS entries
• Added realm and instancename variables to puppet default variables, so that they can be used in puppet runs
• Added support for wiki page creation for Projects
• Added support for configuring default images for instances
• Added support for creating server admin logs per project
• Added support for default security group rules on project creation
• Added dialog to project creation for adding members to projects and roles upon creation
• Added support for managing puppet classes and variables through the interface, rather than LocalSettings.php
• Made a usability change to instance creation: the default security group will always be selected by default
• Added support for including the ssh key fingerprint of an instance in the “your instance is ready” emails
• Made a usability change to only show actions to users if they can perform them
• Lots of other minor bug fixes
• Bugfixes from John Du Hart, Sam Reed and Mark Hershberger

If you’d like to help develop this extension, I’ve created a development environment in a project in Wikimedia Labs. Find me on #wikimedia-labs on Freenode or email me to get a labs account and access to the project.

Related posts:

1. OpenStackManager version 1.2 released
2. OpenStackManager version 1.1 released
3. Announcing OpenStackManager extension for MediaWiki

1. Create an instance and use it to do experimentation with the service.
2. Document the service, along with the installation process on wikitech, after ensuring the service is working properly.
3. Create a second instance. Following the documentation written, puppetize the service.
4. Create a third instance. Ensure the puppetized service runs properly when initialized from scratch.
5. Kill all three instances, and replace the instances in the test cluster.

When a service changes in puppet, follow the above cycle as well.

Using this process, I can be assured the puppet manifests, as written, will allow me to repeatedly install this service.

Related posts:

1. Sharing home directories to instances within a project using puppet, LDAP, autofs, and Nova
2. Announcing OpenStackManager extension for MediaWiki
3. OpenStackManager version 1.3 released

The problem with NFS home directories, however, is that they are fairly insecure. They can be used between instances to escalate privileges. In our environment, this isn’t a problem for instances within a project. If a user is a member of a project, they have shell on all instances. If they are given sudo access in a project, they are given sudo access on all instances. Between projects, however, is a problem. user-A with root on instance-A in project-A could su to to user-B on instance-A, modify the user’s authorized_keys file, and then have access to project-B if home directories are shared across projects.

To avoid cross-project escalation, each project needs its own set of home directories. This means we can’t simply export /home to the instance’s private network and be done with it. We’ll need to create an exports file, and share different directory trees with specific instances. We’ll also need to mount home directories differently on the instances, depending on the project they belong to.

To do so, we’ll use a combination of puppet, LDAP, autofs, and Nova.

Creating the exports file

To create the exports file we need three things:

1. A list of projects
2. A list of instances within each project
3. A list of home directory locations for each project

The first two can be found via LDAP. The query for a list of projects is: ‘(&(objectclass=groupofnames)(owner=*))’. The query for a list of instances within each project is: ‘(puppervar=instanceproject=<project>)’. Of course, this approach is only usable for people using the OpenStackManager extension for MediaWiki; I’ll mention more portable ways to get this information later in the post.

The third we can extrapolate, we just need a base directory. I chose to use /export/home/<project> for the locations.

I wrote a python script that will pull this information, and create an exports file that looks like this:

/export/home/<project1> <project1-instance1>(rw,no_subtree_check) <project-instance2>(rw,no_subtree_check) <project_instance...>(rw,no_subtree_check)
/export/home/<project2> <project2-instance1>(rw,no_subtree_check) <project2-instance2>(rw,no_subtree_check) <project2_instance...>(rw,no_subtree_check)

Mounting the shares from the instances

Each instance needs to mount the share, depending on its project. There’s a number of ways we can do this, but I like the flexibility of using autofs and LDAP to manage NFS mounts. To add slightly more flexibility we’ll involve the help of puppet as well.

In LDAP, we can create autofs entries by making maps and objects. The following objects add support for /home:

dn: nisMapName=auto.master,<basedn>
objectClass: top
objectClass: nisMap
nisMapName: auto.master

dn: nisMapName=auto.home,<basedn>
objectClass: top
objectClass: nisMap
nisMapName: auto.home

dn: nisMapName=/home,nisMapName=auto.master,<basedn>
objectClass: top
objectClass: nisObject
cn: /home
nisMapEntry: ldap:nisMapName=auto.home,<basedn>
nisMapName: auto.master

We also need to add entries for the specific home directories. Here we are going to invoke a little awesome magic that autofs has: variables. Here’s the entry we are using for all home directories in all projects:

dn: cn=*,nisMapName=auto.home,<basedn>
changetype: add
nisMapEntry: ${SERVNAME}:${HOMEDIRLOC}/&
objectClass: nisObject
objectClass: top
nisMapName: auto.home
cn: *

We only need the one entry, which saves us from having to create and delete entries on creation and deletion of projects. Using this, however, means we need to set the variables. This is where puppet comes in. First, though, let’s look at the node in LDAP:

dn: dc=i-0000005c,dc=pmtpa,ou=hosts,dc=wikimedia,dc=org
objectClass: domainrelatedobject
objectClass: dnsdomain
objectClass: domain
objectClass: puppetclient
objectClass: dcobject
objectClass: top
puppetVar: realm=labs
puppetVar: writable=false
puppetVar: db_cluster=s1
puppetVar: instancecreator_email=rlane@wikimedia.org
puppetVar: instancecreator_username=Ryan Lane
puppetVar: instancecreator_lang=en
puppetVar: instanceproject=testlabs
puppetClass: base
puppetClass: ldap::client::wmf-test-cluster
puppetClass: exim::simple-mail-sender
puppetClass: db::core
puppetClass: mysql::mysqluser
puppetClass: mysql::datadirs
puppetClass: mysql::conf
l: pmtpa
associatedDomain: i-0000005c.pmtpa.wmflabs
associatedDomain: labs-db2.pmtpa.wmflabs
dc: i-0000005c
aRecord: 10.4.0.12

All the above objectclasses and attributes are available for use in puppet. The really important one here is instanceproject=testlabs.

We can set the autofs variables via the OPTIONS variable in the /etc/default/autofs file:

OPTIONS=”-DSERVNAME=<%= nfs_server_name %> -DHOMEDIRLOC=<%= homedir_location %>”

Here SERVNAME and HOMEDIRLOC autofs variables are being set. nfs_server_name and homedir_location are being set via a puppet template. Both are being determined via a manifest:

$homedir_location = "/export/home/${instanceproject}"

nfs_server_name is a hash, based on the project:

$nfs_server_name = $instanceproject ? {
	default => "labs-nfs1",
}

I chose to use a hash based on project so that I can choose to separate the server based on project as well, if needed for performance, or extra security.

Managing user home directories

Everything up to this point is just creating the shares. However, we must maintain the users’ home directories as well. For this, we need to know which users are in which projects, and we need to manage their home directories per project.

I wrote a script to search for users, based on a group (the project), that selectively creates/deletes/renames home directories and authorized_keys files. I should note here that I don’t use nova’s mechanisms for SSH key management, as it isn’t portable between applications. I instead store the keys in the user’s LDAP entry.

There’s a security issue with management of home directories. If a user is added to a project and we create a home directory, with a populated authorized_keys file, then remove the user from the project, but don’t remove their home directory, the user will still have access to the project’s instances. There’s two ways I go about solving this issue:

1. Ensure the user only has access to the instance if they are in the project, using access.conf. In my architecture, when projects are added, they are also made a posixgroup, with a gid. Thanks to this, we can treat the project as a system group in all instances. In access.conf we limit access to the project group.
2. User’s home directories are moved from /export/home/<project>/<user> to /export/home/<project>/SAVE/<user> when they are removed from the project.

Problems with this solution, and future improvements to make

The major shortcoming of this solution is that it isn’t terribly portable. It is dependent on using LDAP, and storing specific information in the LDAP directory. Using the nova tools, or having nova manage the exports on instance creation/deletion would make this a much more portable solution.

Another shortcoming is that it isn’t terribly scalable. The exports file is being created from scratch every single script run (which needs to happen fairly frequently). Ideally, nova would write to a queue, and the NFS instance would add/remove instances from the exports as instances are created/deleted.

Thankfully, I didn’t have a shortage of ideas about how to accomplish this, as shown in my proposal. I decided upon the quick and dirty approach, opting to do one of the more reusable approaches later. I’ll likely add support to nova for this at some time in the future.

Related posts:

1. Why I chose MediaWiki for my OpenStack Manager project
2. A process for puppetization of a service using Nova
3. Announcing OpenStackManager extension for MediaWiki

The day of the announcement there was also a governance town hall meeting. The meeting was conducted with attendees sitting in a circle, discussing the foundation formation as a group. It felt the way a community discussion should feel: warm, open, but with a little bit of critical questioning on occasion as well. During this meeting a number of good ideas were put forward about who we should be getting advice from, possible structures, and most importantly, how we’ll create the foundation as a community over the next year. I think this discussion was a great starting point for the rest of the process.

In the next couple weeks we’ll hear how to actively participate in the creation of the foundation. I’m sure however the process works, that it’ll be transparent and fair. I also have a strong feeling that the outcome of this process will be a foundation that can not easily be coerced by a single vendor.

A few topics that I brought up during the town hall discussion concerned the possibility of the foundation being controlled by a single vendor. The first topic was about how roles in the foundation would be filled. Would the foundation employ all of the roles, or would they be appointed from community members? In the latter case, a single vendor can control the foundation by being the one that continuously occupies all of the positions. In the former case, the foundation would control the roles, but would require far more money to operate. My second topic was regarding control and money. How would donations be handled? Would it be possible for one vendor to control the foundation through being the primary sponsor? My third topic regarded the current situation with community roles. Nearly every role is currently filled by Rackspace. Should we limit the number of appointments any one specific organization can have?

I was very happy with the responses to my questions. Initial thoughts about employees vs appointments were that appointments will likely lead to a stronger community and it’s likely best to not have a sprawling foundation. The money question likely comes down to how much can legally be donated. Lastly, the thoughts on limiting appointments was that it is likely unnecessary and that it may hinder community involvement. Limiting appointments was initially tried on the software development policy side of things and no one liked it. It was also mentioned that we should be encouraging participation from every vendor in a way that is ongoing, rather than one-off.

My questions revolved around control. The project is, and will likely still be mostly lead and controlled by Rackspace for the near future. Of course, these questions are also purely theoretical. Rackspace has done an amazing job leading this project and encouraging participation and growth so far, and they haven’t used the project to benefit themselves over other community members. They just happen to currently be the organization that has the largest commitment to the project, since the project is still so young. That’s a great thing. I’m hoping to see the same level of commitment, if not even more from Rackspace after the foundation is formed. I’m also hoping to see much greater participation from other members of the community. I think that diversity will naturally occur over time as more community members add resources from their organizations.

I’d like to thank Rackspace for forming the foundation, and I’d like to thank in advance all the community members that are going to be working together to make this happen. Let’s make the project strong together!

Related posts:

1. I’ve been with the Wikimedia Foundation for a year. Have I met my goals?
2. Building a test and development infrastructure using OpenStack

Our puppet configuration is broken into two environments (production and labs), each environment having a public and private repository. Inside of these repositories our configuration is broken into three main directories: files, templates, manifests. I often jump between repositories, directories within repositories, and branches within different screen windows.

The information I need to know is this:

• The current working directory in each screen window
• When inside a repository, which branch is active
• Whether the branch has modifications or not

Using a combination of screen and bash configuration, I display the current working directory as the title for each screen window. Here’s the bash configuration for this (in .bashrc):

export PROMPT_COMMAND='
if [ $TERM = "screen" ]; then
 MYPWD="${PWD/#$HOME/~}"
 [ ${#MYPWD} -gt 20 ] && MYPWD=..${MYPWD:${#MYPWD}-18}
 echo -n -e "\033k$MYPWD\033\\"
fi
'

Here’s the screen configuration for this (in .screenrc):

hardstatus alwayslastline
hardstatus string "%{.bW}%-w%{.rW}%n %1`%{-}%+w %=%{..G} %H %{..Y} %m/%d %C%a "

Thanks to Vishvanada Ishava, I can tell which branch is active, and whether or not there are modifications in the repository I’m in. Here’s the bash configuration for this (in .bashrc):

function git_branch {
  git branch --no-color 2> /dev/null | egrep '^\*' | sed -e 's/^* //'
}
function git_dirty {
  # only tracks modifications, not unknown files needing adds
    if [ -z "`git status -s | awk '{print $1}' | grep '[ADMTU]'`" ] ; then
        return 1
    else
        return 0
    fi
}

function dirty_git_prompt {
    branch=`git_branch`
    if [ -z "${branch}" ] ; then
        return
    fi
    git_dirty && echo " (${branch})"
}

function clean_git_prompt {
    branch=`git_branch`
    if [ -z "${branch}" ] ; then
        return
    fi
    git_dirty || echo " (${branch})"
}

export PS1='\n\e[1m\u\e[m@\e[4m\h\e[m:\e[7m\w\e[m\[\033[01;31m\]$(dirty_git_prompt)\[\033[01;32m\]$(clean_git_prompt)\[\033[00m\]\n\# \$> '

Here’s a screenshot of what it looks like (and, yes, the name of my laptop is Free-Public-Wifi :D):

Question is, did I meet the goals I set for last year? Here’s my goals from last year with the results:

1. Learn how to deal with Wikimedia infrastructure during an emergency situation, so that we don’t have to constantly wake up the same folks
• Success. I handled a number of outages without needing to page anyone.
2. Build a virtualization cluster for test, development, and volunteer evaluation; this includes migrating virtual machines away from the currently proprietary ESXi based Tesla infrastructure I built while working on the Wikipedia Usability Initiative as a contractor
• Partially complete. Tesla still exists and is still running a number of virtual machines that we use for test and development. A replacement is not up yet. That said, the scope of this goal changed dramatically as I started thinking of what we really needed. The plans for this are far more ambitious now, and so far it’s going well. The architecture is mostly up, we’re starting to give people accounts, and by mid-October it should be fully ready for use. We are having a hack-a-thon in New Orleans that will be primarily focused on this project. Ubuntu Ensemble will be working with us inside of this environment, and so will OpenStreetMap. I gave a keynote about the architecture of this environment at the OpenStack Design Summit and Conference, and a gave a talk at Wikimania about the future of development and operations at Wikimedia Foundation using this environment. I recently set up Gerrit, imported our puppet repository, and released it to the public, as a first major step in the opening of the Labs project.
3. Centralize authentication for cluster resources and integrate other sys-admin based resources to eliminate multiple sets of credentials
• Partially complete. The Labs project needed a fair amount of services, and their authentication needed to be integrated. Subversion user accounts were already being stored in LDAP, so I integrated all Labs services with Subversion LDAP. I switched our puppet repo from Subversion to Git (in Gerrit), so ops code review now  has integrated authentication.
4. Improve (or replace) Central Auth to include OpenID (as a provider and consumer) and oAuth at a minimum
• Skipped. This was one of my ambitious goals that was an “if I got time for it” goals. I didn’t get time for it. This is something we need for Labs, so it’s still on my plate, and will likely be a major goal of next year. We will be discussing this at the New Orleans hack-a-thon.
5. Add more environments to the Selenium cluster I built as part of the Wikipedia Usability initiative, and continue to work with the group creating a Selenium testing framework for MediaWiki
• Dropped. Support for Selenium from the Wikimedia Foundation was dropped a few months ago. We were focusing on unit tests. We were writing very few Selenium tests, and the overhead of maintaining this cluster was basically wasted effort. Because of that we decided to scrap it and support other things instead.
6. Foster and work towards an environment capable of easily integrating volunteer sys-admins
• Success. This is exactly why the scope of the virtualization project expanded so greatly. I think the goals of Labs address this very well. The entire environment is built around the idea of volunteer ops. Also, with the move of the puppet repository into a public Git repository, we can easily take changes from volunteers.
7. Do whatever I’m told to do ;)
• I always do so poorly with this one :D. Overall I’d say this goal has been completed.

What did I do that was outside of these goals?

1. Created an HTTPS cluster and enabled HTTPS in experimental mode on a couple of our larger sites.
2. Moved, upgraded, and puppetized our Subversion set up. I moved it in the middle of a hack-a-thon too ;).
3. Added backups for our miscellaneous database servers
4. Daemonized and puppetized a bunch of things that were being started in screen sessions
5. Packaged a bunch of things that were either being installed unpackaged, or new things we needed that weren’t packaged yet
6. A ton of other small tasks – see our Server Admin Log for this information.

What are my goals for next year?

1. Continue with the Labs project. Finish set up of test/dev Labs, and begin work and make major progress on tool Labs.
2. Hire a devops contractor for work on Labs
3. Build a devops community around the Wikimedia architecture
4. Finish the HTTPS project. This will hopefully be complete from the ops perspective by the end of this year.
5. On-board new employees
6. Enable OpenID as a provider and oAuth on Wikimedia (this goal still needs consensus)

Related posts:

1. Now full time Operations Engineer for Wikimedia Foundation

Package management is more than just saying “I want this package to be this specific version on these sets of systems”; package management gives you an overall view of your system in a number of ways. One view is the security profile and compliance of your entire network of systems. I should be able to quickly determine the CVE compliance of my systems. I want to be able to match a vulnerability to a package, quickly, and ensure all systems are patched for that. I should also have a view of which packages are installed on which systems, and be able to group them into system groups for easy reporting or actions.

Additionally, I want to be able to quickly and easily test packages before deploying them everywhere. I want to have a testing group of systems, a staging set of production systems, and the rest of the systems grouped, so that I can deploy packages in a sane way. I want to be able to stage package updates on a schedule so that I can do rolling package updates.

These two things are possible with Chef and Puppet, but aren’t easy (not even close to easy). These tools aren’t built for things like this. Landscape has a least a minimum set of these features. Spacewalk has all of these features.

Unfortunately, Landscape is proprietary, and Spacewalk doesn’t fully support Debian and Ubuntu.

Canonical, please open source Landscape

I understand that you need to make money somehow, but it’s somewhat hypocritical to run an open source company based on the profits of proprietary software. Red Hat seems to do pretty well without proprietary software, I’m sure you can as well. Red Hat even goes one step further and buys proprietary software just to open source it (see Fedora Directory Server, Dogtag, RHEV-M, etc.).

Don’t take this the wrong way, I’m happy with your open source offerings. The release schedule for Ubuntu, and Ubuntu server is essentially perfect for running Linux networks, and the LTS offering is great from the stability vs bleeding edge point of view; however, the lack of decent package management makes your offering excruciatingly painful to use on any large set of systems.

Fedora, help us with Spacewalk

Spacewalk is a great product. The functionality it offers for Fedora and CentOS (and RHEL via Satellite Server) is essentially perfect. It is by far one of the best management tools around. Please help expand this tool to Debian and Ubuntu.

Spacewalk version 1.4 greatly helps bring this support, but there is still a lot left to do. Of course, as I ask for the help of the Fedora community, I also hope the Debian and Ubuntu communities help with this as well.

The self documenting architecture

Server documentation is always out of date, and it annoys me. Sure, in a virtualized environment you can query the controller to get information about systems, but that’s only good to a point. Usually most controllers aren’t well suited to do documentation, and it kind of sucks to have to query a system to get that documentation. I like to do system documentation in a wiki. I can organize it how I want, and add any additional information that I want; this may not be supported by a controller. I also want to be able to link to my other documentation from my resource pages, or link from other documentation to my resource pages. This means I usually end up documenting my architecture in a wiki and as normal it’s all out of date.

No more! Since the OpenStackManager extension is managing all of the LDAP and OpenStack Nova resources, it can also add documentation for the resources while it’s at it. The extension will take all of the information and add it to a page based on the resource’s ID. The content of the page will be a mediawiki template (Nova Resource), with arguments and values for each piece of data. Here’s the current format of the template:

{{Nova Resource
|Resource Type=instance
|Instance Name=%s
|Reservation Id=%s
|Instance Id={{PAGENAME}}
|Private IP=%s
|Public IP=%s
|Instance State=%s
|Instance Host=%s
|Instance Type=%s
|RAM Size=%s
|Number of CPUs=%s
|Amount of Storage=%s
|Image Id=%s
|Project=%s
|Availability Zone=%s
|Region=%s
|Security Group=%s
|Launch Time=%s
|FQDN=%s
|Puppet Class=%s
|Puppet Var=%s}}

These pages are created in the Nova Resource namespace, so that it’s possible to restrict write access to that namespace. The pages will be updated whenever certain resources are added, configured, or deleted (currently only instances are supported).

An architecture with queryable semantic data

The OpenStackManager extension enables semantic support for the Nova Resource namespace, if Semantic MediaWiki is available. This allows you to add semantic annotations to the Nova Resource template.

By making semantic annotations for all of the resource data, you can then use those annotations in interesting ways. I have some example queries at the reference implementation.

An example use case of this semantic data

Semantic MediaWiki has a bunch of output formats. One really interesting output format is JSON. The first thing that came to mind when I noticed this format was available was: how can I use this on the instances?

I fairly often need to run commands on a number of systems. I use dsh for this, and don’t necessarily like it. I don’t like it because I need to keep the dsh groups updated. This is like documentation. It’s a manual process, and as such, it’s always out of date. Well, since the wiki is documenting the instances as they are created, deleted, and re-configured, then it’s always up to date. Since we have all the instance data semantically annotated, we can also pull that information, and since there is a json export, I can use the json data in scripts on the command line.

As an example, here’s a simple dsh written in python using system groups pulled via semantic queries. First, take a look at the instances we’ll be running this against. Now, let’s take a look at the output:

laner@nova-controller:~$ python ddsh.py -p ganglia "echo hello"
Running "echo hello" on instance "i-00000010.sdtpa.tesla.wmnet"
hello
Running "echo hello" on instance "i-00000011.sdtpa.tesla.wmnet"
hello

Ideas?

This is just a proof of concept of what can be done. I probably won’t actually use this script. I can keep my dsh groups up to date with puppet and likely will. I’m sure I’ll find some really great uses for the semantic data though.

Have any ideas on how to use a system like this? Let me know in the comments!

Related posts:

1. Sharing home directories to instances within a project using puppet, LDAP, autofs, and Nova
2. OpenStackManager version 1.3 released
3. Building a test and development infrastructure using OpenStack

The basics

First of all, security in an environment like this is important. I restrict my login to key based login. I protect my key with a password. I don’t put my key on removable media. I keep a strong password on my key. I use full disk encryption for the system my key is on. I keep an encrypted backup of my key in a safe, located somewhere my computer isn’t kept.

In this environment I always want my working state to be kept. If I’m logged into 10 systems, I want to still be logged into 10 systems when I leave the office, and go home, or when I’m in an airport. I don’t want to have to deal with reconnecting my ssh-agent. I want my agent to always require the key on my computer (it shouldn’t be stored on the shell server). I want my environment to tell me which systems I’m logged into.

Using screen

Using screen handles a lot of the requirements. If you start a screen session on your shell server, you can always reconnect to the same working state you left behind. I like to know which system I’m currently on inside of a screen window, which shell server I’m on, and the date and time in UTC; to do so, set the following in .screenrc:

hardstatus alwayslastline                                                                              
hardstatus string "%{.bW}%-w%{.rW}%n %t%{-}%+w %=%{..G} %H %{..Y} %m/%d %C%a "

The above isn’t really enough to always show you which system you are ssh’d into past the shell server, though. You can use bash to do this; add the following into your .bashrc:

setscreentitletohost() {                                                                               
  if [ "$TERM" == "screen" ]                                                                     
     then                                                                                           
     echo -ne "\033k$HOSTNAME$\033\\"                                                       
  fi                                                                                             
}                                                                                                       

setscreentitletohost                                                                                    

ssh() {                                                                                                
  inargs="$@"                                                                                    
  if [ "$TERM" == "screen" ]                                                                     
  then                                                                                           
     host="${inargs#*@}"                                                                    
     host="${host% *}"                                                                      
     user="${inargs%@*}"                                                                    
     user="${user#* }"                                                                      
     if [ "$user" == "root" ]                                                               
     then                                                                                   
         host="$host#"                                                                  
     else                                                                                   
         host="$host$"                                                                  
     fi                                                                                     
     echo -ne "\033k$host\033\\"                                                            
  fi                                                                                             
  /usr/bin/ssh -A $inargs                                                                        
  setscreentitletohost                                                                           
}

I’d like to be able to detach while still having access to my current ssh agent, for this you need to deal with your ssh agent.

Handling the agent

This is fairly easy. You can use a symlink in bash to point to your current agent (credit for this particular implementation to a Blinded by Tech post):

In .bashrc:

if test $SSH_AUTH_SOCK && [ $SSH_AUTH_SOCK != "/tmp/ssh-agent-$USER-screen" ]                          
then                                                                                                   
    rm -f /tmp/ssh-agent-$USER-screen                                                              
    ln -sf "$SSH_AUTH_SOCK" "/tmp/ssh-agent-$USER-screen"                                          
fi

Now configure screen to always point to this socket. In your .screenrc:

setenv SSH_AUTH_SOCK "/tmp/ssh-agent-$USER-screen"

Summary

Now whenever you ssh into your shell server (while forwarding your agent), you will be able to connect to your shell session, keep your working state, connected to your current agent, and you’ll be able to see which systems you are currently logged into.