Screen allows you to create virtual terminals that you can detach from and attach to at will. Detaching from a screen will leave it running in the background. Attaching to it again allows you to keep executing commands where you were or see the current state of a process you left running. If you are the user who started the screen you can attach to a screen that is already attached to, this is is how the shared screen is accomplished.

To set it up, one user logs in to the server and, using su or sudo su, changes to either the second users account or an account that all parties can access. Next, start screen with the -S option and give it a meaningful name.

me@jjtest:~# sudo su – theOtherGuy
theOtherGuy@jjtest:~# screen -S blogTest

If screen is not installed, it is available as a package on most popular linux distributions. You are now in a screen session as theOtherGuy. Now just tell theOtherGuy to log in and that your screen is called blogTest and he will simply need to run screen with the -x option and give the screen name.

theOtherGuy@jjtest:~# screen -x blogTest

At this point you are both in the same screen session and can both interact with it. This is great for showing another user what kind of problems you may be having or how to do a certain process. It can also be used to accomplish remote pair programing on the remote system. And this doesn’t have to be limited to just two users, as long as someone can get to that system as that user they can join the screen as well.

Typing exit at this point will kill the screen and kick both of you out. If one user wants to leave but not kick the other person off the user just presses Ctrl+a and then d, this will detach you from the screen. Ctrl+a is the default hot key for activating screen commands. You can read more about the variety of command available on screen at this site or read abut it fully on the man page.

Screen can be used to do so much more than I’ve talked about here but it’s ability to facilitate the sharing of a terminal between users can be a life saver when you need it. It is not normally installed by default to may linux systems so you may have to install it or ask that it be installed.

!$
The command that this post is named for is used to reference the last argument in the previous command. For example, let’s say you set up a new directory for your client you would need to set the ownership and permissions on it.

mkdir /dir/made/for/client1
chown client1:client1 /dir/made/for/client1
chmod 750 /dir/made/for/client1

This can be shortened by using the !$ shortcut like this.

mkdir /dir/made/for/client1
chown client1:client1 !$
chmod 750 !$

The time savings here are minor but over time can help, by referencing the first command we also reduce the number of typos.

!!
This command references the complete previous command. “Doesn’t the up arrow do that?” you ask? The up arrow pulls the previous command up, allowing you to run it again or modify it. !! allows you to insert the previous command into what you are typing. Take the previous example, if you didn’t have permission to make the directory the first time you tried you could try it again using sudo, like this.

mkdir /dir/made/for/client1 (returns an error about permission denied)
sudo mkdir /dir/made/for/client1 (works)

This can be shortened using the !! shortcut like this.

mkdir /dir/made/for/client1 (returns an error about permission denied)
sudo !! (works)

!number or !-number
BASH, in most configurations, keeps a history of your commands. You can see a list of those commands using the command history.

server01:~ j2$ history
26 command1
27 command2
28 command3
29 command4
30 command5
31 command6
32 command7
33 command8
34 command9
35 history

You can reference any of these commands using the number next to them and the ! shortcut. If I wanted to run command5 again I would do this.

!30

You can also reference commands in reverse by negating them. If I remembered that I ran command5 six commands ago I could do this.

!-6

!:number
By putting a : and a number after the ! you can reference individual arguments in a previous command. Each “word” in a command gets assigned a number, starting with 0. For example, in the command chmod 750 /dir/made/for/client1 chmod = 0, 750 = 1, and /dir/made/for/client1 = 2. So if I had to change the permissions on a few directories it would look like this.

chmod 750 /dir/made/for/client1
chmod 750 /dir/made/for/client2
chmod 750 /dir/made/for/client3

Using the shortcut it would look like this.

chmod 750 /dir/made/for/client1
!:0 !:1 /dir/made/for/client2
!:0 !:1 /dir/made/for/client3

Finally, the last shortcut and this one can be combined to reference specific parts of a command in your history list. Say I just did all those chmod commands above but forgot to chown client1′s dir. Using the combination I could do this.

chown client1:client1 !-3:2

You can now see why ! is one of my favorite tools while working in BASH. If you have any questions or know of another way to use !, please leave a comment.

     Every domain has a list of authorized name servers, SOA servers. Without going into a lot of detail about DNS architecture, these are the servers that are the final knowledge keepers for your domain. When other DNS servers don’t know about your domain they ask the SOA and remember the answer for a short while. For this reason you need to make sure that your SOA servers continue to stay in your control by regularly checking the SOA record of your domain in your monitoring. Be aware that if you do not run your own DNS servers these values may occasionally change, contact your provider to better understand their policy.

     Now that your sure your DNS records are still under your control it’s time to move on to the records themselves. The more records you have the more complicated it becomes to monitor all of them. I suggest to my clients that they evaluate all of the names that they are using and ask themselves, “Will this effect my end user if it’s gone?” If the answer is yes, monitor it. For checking entries I suggest checking the response on both your SOA servers and a public DNS server ( I frequently use one of 4.2.2.[1-6] ) to make sure the public sees what your seeing.

     My last suggestion is an extension of the previous one but is often over looked, MX records. MX records are how DNS servers tell mail servers where to send your domain’s email. If these were altered or removed a key piece of email could go missing or get sent to someone you would rather not read it. The solution for this is the same as the host records, check your servers and public servers for the correct response.

     This has been a very basic overview of what you should be thinking about when you are configuring your DNS monitoring. More complex setups that include geo-load-balancing as well as distributed and redundant services will require a more complex configuration but should still be monitored along these lines.

     Since linux distributions are varied in their installs I won’t go through each but most of the “friendly” distributions start, by default, a variety of services that may not be required but could potentially have exploits. While most of these don’t have a network component, combined with other exploits they could help open the server to attack. For example, Red Hat starts processes to monitor the software raid and logical volume manager even if you aren’t using them. It also starts processes for handling bluetooth devices, HP printers, and command line mouse support, even if you don’t have them. None of these should cause you any concern but if you don’t need them they don’t need to run at all.

     Most Apache HTTP server installs suffer from the same desire for usability, many modules are made available to the server by default. For example, you probably aren’t using LDAP authentication or WebDAV as part of your server but the modules for them are preloaded on most default installs. Identifying the modules that are required for your web site or application to run and then disabling the ones that are not will reduce your apache httpd footprint and therefore reduce your exposure.

     MySQL server doesn’t have the modular nature of our prior two examples but there are some steps that you can take to reduce your exposure. First off, after doing the install and setting the root password, remove the test user and database. These have no known exploits but aren’t needed. Second, ensure that your users are bound to a host instead of a wild card address, this makes sure that connections are only authorized from known hosts. Finally, if you are running mysql on the same host as your webserver and this is the only server that needs to access it, configure it to only listen on localhost ( There is no place like 127.0.0.1 ), this ensures that remote hosts cannot connect to your database even if your firewall fails.

     While I did focus on some of the more simple things that can be done to a LAMP server, this should give you an idea of what kind of changes can be made that won’t effect your service but will reduce your exposure footprint. Remember that before you make any changes you should do a backup and make copies of the files you are editing. We will see what this month brings as far as unpublished exploits and should also take this time to remember that not all exploits are published or patched, or even discovered yet.

Why do we monitor?

The simple answer is that we want to ensure that the vital service we provide is available to our clients. A developer approaches this from the application. A systems administrator approaches this from the server. A network administrator approaches this from the network. They are all correct but it only takes a failure to understand one of the components and you have a huge blind spot in what you think is a solid first line of defense. As I continue this blog I will explore various pieces of the monitoring puzzle and, hopefully, help you understand where your monitoring may be lacking.

As a systems administrator in many mixed Windows and linux environments I have seen and made many of my own attempts at integrating the linux servers into the Windows Active Directory structure with mixed results. Linux registration and authentication inside of a Windows domain is akin to the bigfoot: some have claimed to see one, many have worked long hours to find them, no one can produce consistent results. Until now.

At the suggestion by a coworker I decided to give Likewise a try on my most recent attempt at bringing our linux servers into the windows domain. Skeptically I built a new CentOS 5.3 server and proceeded to follow the instructions for installing Likewise. The seemed too simple and with each step I waited for the fatal issue that would bring the test to it’s demise. Software installed, no issues. As I reviewed the instructions for adding the system to the domain and found that it only required one command and no further configuration of files I thought, “This is so going to fall on it’s face.” Command executed and I’m waiting, then it comes up on the screen: Success. Success? Really? I don’t believe it, I log onto the domain controller and there it is, right where new computers are supposed to go in the domain. I quickly flip over to the manual and look up how to authenticate for ssh, simple DOMAIN\\username@host, and give it the final test. Success. In the span of 15 minutes I was able to install the app, add my computer to the domain and authenticate against the domain. 15 more minutes and I was able to limit who could log in and give them sudo access. This is a huge win for any admin who deals with linux servers in a domain.

I’m still testing the limits of Likewise and I will say that it hasn’t been without it’s speed bumps but I plan on paying for a little support and getting the answers I need. Like may open source products, Likewise is making it’s money on support and by selling upgraded functionality. I applaud this model, allowing the flexibility of open source while still finding a way to pay for all that hard work. I will be continuing to test the limits of Likewise but as of now I am thoroughly impressed and will continue to use it.

This past week saw two high profile security breaches on twitter.com and macrumorslive.com, each one a classic example of poor login management by websites. For full disclosure, I do not know anyone at either site but have read about each here and here as well as other websites. The short version on twitter is that an admin at twitter had a common word password and twitter has no limit on the times you can try your login. The short on Mac Rumors Live is that they either had no password on their admin interface or that the login page was sniffed, either way it was a bummer for them.

As a system administrator it is a constant challenge to keep our systems and the web applications that run on them secure while still keeping the usability that our great developers have created. So when I see things like the twitter and mac rumors issues I get very frustrated because the prevention of this is so simple. If you develop a website with user accounts, and most of them do, please consider the following.

Complex password requirements. Google returns about 232,000 results for javascript to check for strong passwords so you don’t even need to invent the wheel to add this to your change / create password page. If you don’t want to bother your users with this at least take some time to require it on admin accounts.

HTTPS, the S is for secure. Developers, please use ssl to encrypt your login page and use redirects to ensure that users don’t accidentally end up there without ssl. As more of us move out of the office and into the coffee shop we are also moving off of a private network and onto a public one. We log into twitter, brightkite, our blog, our email and most people give no thought to the idea that everyone else in that coffeeshop could now have your account and password, unless we use something secure like https. Webmail sites learned this lesson long ago and it’s time for the rest of the web to learn it too. The worst part of this is that most sites already have an https version of their site and it would be a very simple fix for them to redirect to it. For those sites that don’t ( I’m looking at you brightkite ), it’s time to step up, spend the money and get https going. For you wordpress users, setup a self-signed certificate (or purchase one) and wordpress has a plugin to do all the wp-admin encryption redirects for you.

More services are being provided via websites and as users of these we need to demand that they are secure. Give them feedback asking that logins be secure by default so that what happened with twitter and mac rumors doesn’t happen to you.