If you have scripts that connect to S3, and you run out of buckets (Amazon only allows 100 buckets per account), you might get a nasty surprise.

See, you may have been using create_bucket(name-of-bucket) to get your bucket object. It’s undocumented as far as I can see, but apparently if you use create_bucket() on an bucket that actually exists, it’ll return the Bucket object. That’s handy! Except it breaks if you’re unable to create more buckets (even though you aren’t really trying to create more). Sigh, so I refactored as such:

# old and busted: bucket = s3_conn.create_bucket(bucket_name)

# new hotness:
# iterate over Bucket objects and return the one matching string:
def find_s3_bucket(s3_conn, string):
    for i in s3_conn.get_all_buckets():
        if string in i.name:
            return i

Used as: bucket = find_s3_bucket(s3_conn, bucket_name)

There is likely a more elegant way, but hey this works.

Feel free to share!

Hide Sites

Related posts:

1. Finding instances by name with boto in python
2. LDAP: Understand the Protocol and Work With Entries

Random tidbits from today:
I got annoyed with EC2 failures, and having to manually terminate and redeploy instances today, so I finally worked on a script I’ve been meaning to write. One thing I had to figure out (which isn’t all that complex), is how to discover an instance by name.

If you tag an instance with the hostname you’re using in your deployment script, you don’t need to fumble in the AWS console to find an instance ID. Ever. I don’t find it acceptable to manually click around or run scripts to discover information that’s available from an API

So, to “find” the instance using python and boto (assume AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are defined in your shell environment):

import boto
ec2conn = boto.connect_ec2(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
reservations = ec2conn.get_all_instances()
instances = [i for r in reservations for i in r.instances]
my_fqdn = "example.com" # trailing part of my domain

Now, ‘instances’ can be iterated over to find instances with the name you desire. I wrote a little wrapper function to do this, and it returns an instance object (which I call instance.terminate() on, for this purpose). Code:

def find_instance_by_nametag(instances, name):
    # support short or full hostname usage
    if not my_fqdn in name:
        name = name + my_fqdn
    for i in instances:
        if "Name" in i.tags and name in i.tags['Name']:
            return i
    sys.exit("sorry, I couldn't find an instance with that name!")

Easy as that!

Feel free to share!

Hide Sites

Related posts:

1. Connecting to existing buckets in S3 with boto, the right way

They even have an API from which you can fetch an XML document with all your current tracks! The unfortunate part, and the reason for this post, is that they only keep 30 days of GPS coordinates. It’s not a problem if you export your data to Spot Adventures and create an “adventure” — that will live forever. But if you wish to present “my current location” on your personal web page, for example, you’re out of luck. It will only work as long as you’ve used the track progress functionality within the last 30 days.

I guess you need to cache the last used location yourself. OK, this shouldn’t be hard. This is how to do it with a little perl script (download here, see in action in CharlieTracker here):

#!/usr/bin/perl
# Author: Charlie Schluting <charlie@schluting.com> (c) 2011
#
use XML::Simple;
#use strict.. heh, no, this barely works.
$CACHEFILE = "/home/charlie/lastspotlocation.txt";
$XML = "/home/charlie/spot.xml";
$JS = "/stash/www/charlierides.com/files/map.js";

The CACHEFILE is where I store the last known latitude and longitude. It’s mostly for reference, so that other things (aside from this script) can use it.

XML is the location I store the fetched XML from SPOT’s API.

JS is the file I write the javascript out to. OK, you probably don’t need to care about this part of the script.. but here’s the details: if you use joomla or wordpress, you might not want to enable PHP code execution or other evil things like that. So, in order to include a snippet of generated HTML, the only choice (as far as I know, aside from editing the DB where a “display html” module stores its data), is to write javascript to include another javascript file. So that’s what I do.. the script actually generates javascript that when run, will spit out html. If you don’t care about that and just want the map part, read on.

# hahaha, oh man..
`wget -q -O $XML http://share.findmespot.com/messageService/guestlinkservlet?glId=0Vn4kA4MiPgNSYD52NgPjuVJDpUCSUlGW`;

This part fetches the XML file from spot, using my shared page identifier.

# create object
$xml = new XML::Simple;
# read XML file
$data = $xml->XMLin("$XML");

# this is how we overcome spot's API not keeping >30 days. If they've aged out, do nothing (i.e. keep using the old data).
die("No messages found, totalCount is 0, ABORTING LIKE AN UGLY KID") unless $data->{totalCount} > 0;

# the first object is always the most recent:
$lat = $data->{message}->[0]->{latitude};
$long = $data->{message}->[0]->{longitude};

# just because (hey, what if something else wants to use this?)
open(FILE, ">$CACHEFILE");
print FILE $lat . "," . $long . "\n";
close(FILE);

And the rest (above) is pretty self explanatory. Using just this part of the script, you’ve overcome the annoying limitation that is SPOT Track Progress (losing your last known location). Your last known coordinates will be in CACHEFILE. For completeness, I will include the rest of this horrible hack I used:

# ugly shit that writes out javascript to write out html, to include a linked static google maps image
$googleoptions = "&zoom=8&size=140x152&sensor=false&maptype=hybrid";
$googlelink = "<a target=\"blank\" href=\"http://maps.google.com/maps?q=" . $lat . "," . $long . "+(charlie)&z=8&t=h\"\>";
$header = "<p>Current location (since the last GPS update):</p>";
$js = 'document.write(\''. $header .'\');' . '
document.write(\''. $googlelink .'\');' . '
document.write(\'<img src="http://maps.googleapis.com/maps/api/staticmap?center=' . $lat .
',' . $long . $googleoptions . '" />\');
document.write(\'</a>\');
document.write(\'<p>Or <a href="http://share.findmespot.com/shared/faces/viewspots.jsp?glId=0Vn4kA4MiPgNSYD52NgPjuVJDpUCSUlGW" 
target=blank>view all recent tracks.</a></p> \');
';
open(FILE, ">$JS");
print FILE $js;
close(FILE);

So, what this writes out (in HTML, finally), is an IMG tag embedding a static google map (because I’m using a small thumbnail), which links to the google maps page when clicked:

Map: http://maps.googleapis.com/maps/api/staticmap?center=45.42553,-122.52251&zoom=8&size=140×152&sensor=false&maptype=hybrid
Link target: http://maps.google.com/maps?q=45.42553,-122.52251+(charlie)&z=8&t=h

The parameters you can pass google maps are well documented, so I don’t need to rehash them here. You may even prefer to use the real maps API, rather than the static image one. That too, is well documented.

Run this from cron every 15 minutes, and you’ll always have your last checked-in GPS coordinates!

Feel free to share!

Hide Sites

No related posts found.

Flex Links are a pair of layer 2 interfaces configured to act as a backup of each other. Configuring Flex Links is very simple, but it’s a manual process. Spanning tree can configure itself if you just enable it, albeit likely a sub-optimal configuration, but a working one nonetheless. Flex Links, on the other hand, require manual setup and layout of your layer 2 network. If you don’t want to leave anything to chance, then Flex Links are preferred over STP.

The benefits of FlexLinks include:

• simplicity, which equals stability.
• instant failover.
• rudimentary load balancing capabilities, so one link isn’t wastefully idle.
• load balancing works across switches in a stack, including port channels.

Flex Links’ primary operating mode is just like spanning tree: one on, one off. With per-VLAN spanning tree, a trunk port can have some VLANs enabled and some blocked at the same time, so on the surface it seems that STP is superior. In reality, you can configure Flex Links to load balance VLANs, and we’ll show you how shortly.

Configuration

Conceptually, you configure Flex Links by telling one link it’s the active link, and another that it’s the backup of that

Flex Links Design Map

primary (active) one. Without configuring VLAN load balancing, it will completely disable the backup, and if the active link goes down the backup will take over.

For example, to configure port gi1/0/1 as a active link, and gi1/0/2 as the backup, you’d run:

Switch# configure terminal
Switch(conf)# interface gigabitethernet1/0/1
Switch(conf-if)# switchport backup interface gigabitethernet1/0/2

That’s all there is to configuring the basic mode, which gets you failover but no load balancing. Before talking about load balancing, let’s take a look at preemption and “mac address-table move update.”

Preemption

Preemption, that is, the preferred port for forwarding traffic, is also configurable. This is most often used in combination with multiple links that have differing bandwidth capacities. If you wish to ensure that port 1, a primary port that has more bandwidth, will return to the active link when it comes back up, you would set:  interface preemption mode bandwidth andswitchport backup interface preemption delay. The delay is used to set the amount of time (in seconds) to wait before allowing port 1 to preempt port 2 and begin taking over traffic again.

MAC Address-Table Move Update

Enabling the MAC address-table move update feature allows for rapid convergence when a primary link goes down and the backup takes over traffic forwarding duties. Without this feature enabled, neighboring switches may continue to forward traffic for a short time to a dead port, since they have learned MAC addresses associated with that link.

When move update is enabled, the switch containing Flex Links will broadcast an update packet to let other switches know what happened, and they will in turn un-learn that false MAC address mapping.

On the switch with Flex Links, simply configure:

Switch(conf)# mac address-table move update transmit

All switches, including ones with Flex Links, need to receive these updates. This is not enabled by default, so you’ll need to run the following command on all of your devices:

Switch(conf)# mac address-table move update receive

To see the status and verify that “move update” is enabled, run: show mac address-table move update. Checking the status of your Flex Links is much the same: show interfaces [interface-id] switchport backup.

Load Balancing

Flex Links should be configured such that both ports are forwarding traffic at the same time. This way, you get load balancing in addition to redundancy. The limitation is that only one port can be forwarding a single VLAN at a time. If we have VLANs 1-200, we need to choose which VLANs are forwarded primarily through which port. The most simple configuration, ignoring traffic requirements, would be that VLANs 1-100 use port 1, and VLANs 101-200 use port 2.

Before we get into configuring preferred VLANs, let’s talk about multicast. Multicast, of course, becomes an issue with this type of setup. If a port passed an IGMP join, and the switch is part of a multicast group, when the port goes down the switch will no longer be able to receive multicast traffic for that group. The quick fix is to make both Flex Links always be part of learned groups, with the command: switchport backup interface gigabitEthernet 1/0/12 multicast fast-convergence.

Now, on to VLAN load balancing. It is quite easy; just specify which VLANs you prefer on which links:

Switch(config-if)#switchport backup interface gigabitEthernet1/0/2 prefer vlan 101-200.

If you have VLANs 1-200 on the switch, show interfaces switchport backup will show you:

Vlans Preferred on Active Interface: 1-100
Vlans Preferred on Backup Interface: 101-200

If a link goes down, VLANs that are preferred on that interface will be moved to the other link in the pair. Likewise, when a link returns to service, its preferred VLANs are blocked on the backup and returned to the preferred link.

Be sure to run show interfaces switchport backup detail to see the full status, including link speeds, preemption modes, the MAC address-table move update status.

In summary, the simplicity of Flex Links make it a better choice for carrier and core enterprise networks over the ubiquitous spanning tree protocol. Link-level redundancy is had via STP, but with Flex Links you have more control and better load balancing capabilities. This certainly means that it takes longer to configure since you are planning the layer 2 network manually, but when you need a stable no-surprises link-layer network, Flex Links are definitely the way to go.

Feel free to share!

Hide Sites

Related posts:

1. Built-in Security with Cisco IPS
2. Cisco AutoQoS: VoIP QoS for Mere Mortals
3. Manage Devices and Configurations with Cisco SDM

The first whack at AutoQoS was Cisco recognizing the need to simplify VoIP traffic prioritization. VoIP is especially sensitive to any latency, jitter, or loss, and users will notice problems. To ensure the best possible VoIP call, the network must ensure that lower priority traffic does not interfere with time-sensitive VoIP. AutoQoS can be enabled on both WAN links and Ethernet switches to automatically provide a nice best-practices based template for VoIP prioritization. If you’re lucky enough to have metro Ethernet service, like AT&T ethernet service for example, you should contact your provider to find out if QoS settings on your switches can be duplicated through theirs.

How it Works

QoS allows a router to classify which types of traffic are most important, and ensure that that traffic passed as quickly as possible. If necessary, other traffic will be queued until the higher priority traffic has had a chance to pass. Before a router can know when to queue versus when to attempt to pass all traffic, it must be configured with bandwidth settings for each link.

Configuring QoS on a Cisco router normally involves a complex series of interactions, which require understanding not only the protocols, but a router’s strange way of associating policies. The basic steps are:

• Use an ACL to define which traffic gets matched
• A class-map classifies matched traffic into classes
• A policy-map assigns priorities to the classes
• The policy-map is applied to the interface, which enables the processing of all packets through the ACL, class-map, and policy-map

Each of these “maps” are quite complicated and prone to error. Most sites are going to be duplicating effort because of common problems, like VoIP, needing QoS help.

Why AutoQoS

QoS configuration is not simple. It requires understanding the protocols your network interfaces are using, as well as the type of data you’re passing. To configure QoS for VoIP, for example, you must understand how VoIP works. In short, it requires a guru. If you’re like me, you literally giggled out loud the first time you encountered the word, “AutoQoS.”

AutoQoS enables any network administrator to just “turn on” a solid solution for ensuring VoIP is happy. VoIP is the pain point for most organizations, so that’s what Cisco focused on first, and that’s what we’re focusing on here. Given the limited scope of AutoQoS, it’s believable that it works well enough. In reality, QoS configurations generally classify many types of traffic, and then place a priority on each one.

The main benefit of AutoQoS is that administrator training is much quicker. It also means that VoIP deployments often go much smoother, and upgrading WAN links isn’t usually required. Finally, AutoQoS creates templates that can be modified as needed and copied elsewhere for deployment.

Limitations

Before talking about how to enable AutoQoS, which is literally three commands, let’s talk about where this works best, and what’s required to use AutoQoS.

First and foremost, you can only configure AutoQoS on a few types of router interfaces. These interfaces include:

• PPP or HDLC serial interfaces
• ATM PVCs
• Frame Relay (point-to-point links only)

Cisco catalyst switches also support an AutoQoS command to prioritize Cisco VoIP phones, but you cannot prioritize (using AutoQoS) generic VoIP protocols.

Next, there are some limitations with ATM sub-interfaces. If you have a low-speed ATM link (less than 768Kbps), then AutoQoS will only work on point-to-point sub-interfaces. Higher speed ATM PVCs are fully supported though. For standard serial links, AutoQoS is not supported at all on sub-interfaces. A quick litmus test to see if AutoQoS will work on your desired interfaces or not is to verify that the service-policy configuration is supported. If not, you’ll probably have to reconfigure some links.

AutoQoS will not work if an existing QoS configuration exists on an interface. Likewise, when you disable the AutoQoS configuration, any changes you may have made to the template after the initial configuration will be lost.

Bandwidth statements are used by AutoQoS to determine what settings it should use, so remember that after updating bandwidth statements in the future, you have to re-run the AutoQoS commands.

Making it Work

In the most standard situation, where VoIP isn’t performing as it was promised, the network admin can quickly save the day by running the following on the WAN interface:

interface Serial0
bandwidth 256
autoqos voip

If it’s the local network that needs tuning, the following can be run on Catalyst switches (if running Enhanced Images):

auto qos voip cisco-phone
auto qos voip trust

It really couldn’t be easier than that.  For the WAN example, we told the router that interface Serial0 has 256 Kbps, and to enable VoIP QoS. The switch example is similar, for Cisco phones.

The neat part about this is that AutoQoS is actually doing more than just generating a configuration for you and forgetting about it. If you run the command show autoqos interface s0, you will see much more than just your standard old interface configuration. It will show that a Virtual Template “interface” has been created, and that a class is applied to the interface. The same output will also show you the configuration of the template and class-map, with an asterisk next to each entry that was generated by AutoQoS. It’s actually keeping track of what was done automatically so that you can learn what AutoQoS is doing. As mentioned previously, however, don’t forget that removing the AutoQoS configuration will destroy all QoS settings on an interface, not just the ones that AutoQoS configured.

Finally, remember to enable QoS on both sides of a WAN link to truly prioritize VoIP packets. Don’t forget to read through the Cisco documentation before deploying it, even though AutoQoS is simple, in comparison. It is simple, but the more prepared you are the easier it is to deploy.

Cisco will hopefully continue this trend of providing Auto features for complicated, but common tasks. AutoQoS for VoIP sure does enable a much larger audience to correctly deploy VoIP over a wide variety of networks.

Feel free to share!

Hide Sites

Related posts:

1. Built-in Security with Cisco IPS
2. Are Cisco Flex Links the End of STP?
3. Manage Devices and Configurations with Cisco SDM

Layer 2, the Data Link layer, is where Ethernet lives. We’ll be talking about bridges, switching, and VLANs with the goal of discovering how they interact in this part of Networking 101. You don’t really need to study the internals of Ethernet to make a production network operate, so if you’re inclined, do that on your own time.

Ethernet switches, as they’re called now, began life as a “bridge.” Traditional bridges would read all Ethernet frames, and then forward them out every port, except the one it came in on. They had the ability to allow redundancy via STP, and they also began learning which MAC addresses were on which port. At this point, a bridge then became a learning device, which means they would store a table of all MAC addresses seen on a port. When a frame needed to be sent, the bridge could look up the destination MAC address in the bridge table, and know which port is should be sent out. The ability to send data to only the correct host was a huge advancement in switching; collisions were much less likely. If the destination MAC address wasn’t found in the bridge table, the switch would simply flood it out all ports. That’s the only way to find where a host actually lives for the first time, so as you can see, flooding is an important concept in switching. It turns out to be quite necessary in routing too.

Important terminology in this layer includes:

Unicast segmentation : Bridges can limit which hosts hear unicast frames (frames sent to only one MAC address). Hubs would simply forward everything to everyone, so this alone is a huge bandwidth-saver.

Collision Domain : The segment over which collisions can occur. Collisions don’t happen any more, since switches use cut-through forwarding and NICs are full-duplex. If you see collisions on a port, that means someone negotiated half-duplex accidentally, or something else is very wrong.

Broadcast Domain : The segment over which broadcast frames are sent and can be heard.

A few years later, the old store-and-forward method of bridge operation was modified. New switches started only looking at the destination MAC address of the frame, and then sending it instantly. Dubbed cut-through forwarding, presumably because frames cut through the switch much quicker and with less processing. This implies a few important things: a switch can’t check the CRC to see if the packet was damaged, and that implies collisions needed to be made impossible.

Now, to address broadcast segmentation, VLANs were introduced. If you can’t send a broadcast frame to another machine, they’re not on your local network, and you will instead send the entire packet to a router for forwarding. That’s what a Virtual LAN (VLAN) does, in essence–it makes more networks. On a switch, you can configure VLANs, and then assign a port to a VLAN. If host A is in VLAN 1, it can’t talk to anyone in VLAN 2, just as if they lived on totally disconnected devices. Well, almost; if the bridge table is flooded and the switch is having trouble keeping up, all data will be flooded out every port. This has to happen in order for communication to continue in these situations. This needs to be pointed out because many people believe VLANs are a security mechanism. They are not even close. Anyone with half a clue about networks (or with the right cracking tool in their arsenal) can quickly overcome the VLAN broadcast segmentation. In fact, a switch will basically turn into a hub when it floods frames, spewing everyone’s data to everyone else.

If you can’t ARP for a machine, you have to use a router, as we already know. But does that mean you have to physically connect wires from a router into each VLAN? Not anymore, we have layer 3 switches now! Imagine for an instance, if you will, a switch that contains 48 ports. It also has VLAN 1 and VLAN 2, and ports 1-24 are in VLAN 1, while ports 25-48 are part of VLAN 2. To route between the two VLANs, you have basically three options. First, you can connect a port in each VLAN to a router, and assign the hosts the correct default route. In the new-fangled world of today, you can also simply bring up two virtual interfaces in each VLAN. In Cisco land, the router interfaces would be called vlan1 and vlan2. They get IP addresses, and the hosts use the router interface as their router.

The third way brings us to the final topic of the layer 2 overview. If you have multiple switches that need to contain the same VLANs, you can connect them together so that VLAN 1 on switch A is the same as VLAN 1 on switch B. This is accomplished with 802.1q, which will tag the packets as they leave the first switch with a VLAN identifier. Cisco calls these links “trunk ports,” and you can have as many VLANs on them as the switch allows (currently 4096 on most hardware). So, the third and final way to route between VLANs is to connect a trunk to a router, and bring up the appropriate interfaces for each VLAN. The hosts on VLAN 1, on both switch A and B will have access to the router interface (which happens to be on another device) since they are all “trunked” together and share a broadcast domain.

We’ve saved you from the standard “this is layer 2, memorize the Ethernet header” teaching method. To become truly guru you must know it, but to be a useful operator, (something the cert classes don’t teach you) simply understand how it all works. Join us next time for an exploration of most interesting protocol in the world, Spanning Tree.

Feel free to share!

Hide Sites

Related posts:

1. Networking 101: Understanding Layers
2. Networking 101: More Subnets, and IPv6
3. Networking 101: IP addresses

FC wouldn’t be much use without something on top of it, namely SCSI. FC is the low-level transport that ships data, but hosts are normally talking SCSI as far as they’re concerned. The hubs, switches, and HBAs in a SAN all speak FC, while the applications that use SAN storage continue to use familiar protocols, like SCSI.

The idea behind FC was to create a high throughput, low latency, reliable and scalable protocol. Ethernet wouldn’t quite cut it for highly-available storage needs. FC can currently operate of speeds up to 10Gb/s (10GFC) for uplinks, and 4Gb for standard host connections. FC also provides small connectors. As silly as it sounds, SCSI cables become unruly after time, and small strands of fibre are certainly easier to manage. The equipment required to connect to a FC SAN (multiple HBAs for each host, fibre, and switches) is extremely expensive, and was the main reason SAN technologies took so long to become widely adopted.

Topologies
In reality, two different protocols, or topologies, make up the FC protocol. FC supports all topologies, but the behavior of the protocol changes depending on the topology. The following three types of topologies are supported:

• PTP (point to point): normally used for DAS configurations.
• FC-AL (FC Arbitrated Loop): Fabric Loop ports, or FL ports on a switch, and NL_Ports (node loop) on an HBA, support loop operations.
• FC-SW (FC Switched): the mode when operating on a switched SAN.

FC-AL operation is quite scary, but sometimes a device doesn’t support FC-SW operations, and there’s no choice. A hub has no choice but to operate in FC-AL mode, and therefore attached hosts must as well. When a device joins an FC-AL, or when there’s any type of error or reset, the loop must reinitialize. All communication is temporarily halted during this process, so it can cause problems for some applications. FC-AL is limited to 127 nodes due to the addressing mechanism, in theory, but in reality closer to 20. FC-AL is mostly relegated to niche uses now, including but not limited to internal disk array communications and internal storage for high-end servers.

FC switches can be connected any way you please, since the FC protocol avoids the possibility of a loop by nature. Ethernet isn’t so lucky. The addressing scheme used does impose a limit of 239 switches though. FC switches use FSPF, a link-state protocol like OSPF in the IP world, to ensure loop-free and efficient connectivity.

FC networks are generally designed in one of two ways: either one big star, or one big star with edge switches hanging off it. These are commonly known as “core-only” and “core-edge” configurations. Normally a SAN will contain two of these networks, and each host’s HBA or storage device’s controller will attach to each. Keeping these networks separate isn’t as necessary as it is with FC-AL topologies, but even with FC-SW setups it still provides complete isolation and assurance that a problem in one fabric won’t impact the other. An FSPF recalculation, for example, could cause a brief interruption in service.

Ports
As previously mentioned, there are different port types in a SAN, and it can get confusing. Let’s try to clear up some of that terminology:

• N_Port: Node Port; the node connection point; end points for FC traffic
• F_Port: Fabric Port; a switch-connected port, that is a “middle point” connection for two N_Ports
• NL_Port: Node Loop Port; connects to others via their NL_Ports, or to a switched fabric via a single FL_Port; or NL_port to F_Port to F_Port to N_Port (through a switch)
• FL_Port: Fabric Loop Port; a shared point of entry into a fabric for AL devices; example  NL_Port to FL_Port to F_Port to N_Port
• E_Port: Expansion Port; used to connect multiple switches together via ISL (inter-switch links)
• G_Port: Generic Port; can switch between F_Port and E_Port operation depending on how it’s connected
• TE_Port: Trunked Expansion Port; link aggregation of multiple E_Ports for higher throughput

You’ll generally only see F_Ports and FL_Ports when looking at a single SAN switch, and knowing the difference helps. FL means that you’re talking FC-AL, and there’s a device attached that is either a hub, something that can’t do anything but FC-AL, or something strange. Ports will automatically configure themselves as an FL_Port if the attached device is Loop-only, otherwise it will be an F_Port. It’s also worth noting that some brands of FC switches don’t allow you to have an E_Port unless you pay a higher licensing fee. It’s something to think about if you ever plan to connect multiple switches together.

FC Layers
FC has its own layers, so in fact, calling it “like Ethernet” isn’t quite accurate, even if it helps for understanding. They are:

• FC-0: The interface to the physical media; cables, etc
• FC-1: Transmission protocol or data-link layer, encodes and decodes signals
• FC-2: Network Layer; the core of FC
• FC-3: Common services, like hunt groups
• FC-4: Everything! Protocol mapping for SCSI, iSCSI, FCP, IP, and others

The bulk of FC is really in FC-2. FC-PH refers to FC-0 through FC-2, which are strangely dubbed the physical layers.

FC also supports its own naming and addressing mechanism, which sheds light on the previously mentioned limitations in FC-AL and FC-SW topologies. Next time, we’ll discuss the header format for FC-2 as well as FC address assignment and name resolution.

In a Nutshell:

• FC is the transport mechanism, and SCSI or even IP rid atop FC
• FC-AL is a loop, where all connected devices see each other, and a re-initialization takes out the entire SAN
• Port types reveal what is actually happening, and knowing what they stand for can aid in topology visualization when looking at a switch’s configuration

Feel free to share!

Hide Sites

Related posts:

1. Networking 101: Understanding Layers
2. Understanding Linux Virtual Memory
3. SAN 101: Intro to SANs and Storage

There are basically two groups of people who’d want to restrict login users heavily. First, the collaborators, possibly two separate organizations that have been forced to work together. Second, people who wish to allow some shady characters access to a shell, but believe they may attempt to compromise security. If at all possible, the best policy is to simply not give access out, and if you do, make sure patches are applied daily.

To say that you simply shouldn’t give out shells to untrustworthy users may work in a few instances. Say, for example, there is a need for remote users at another site to login and run the same series of commands every day. Say, for the sake of argument, their task can be easily scripted. If this is their only purpose on the server, a shell certainly isn’t necessary. OpenSSH allows a set of restrictions to be applied to an SSH key.

At the end of an SSH key entry, you can tack on these options:
no-port-forwarding,no-X11-forwarding,no-agent-forwarding,command=”~/bin/script.sh”
This effectively restricts any SSH connections using this key to only being allowed to run the mentioned script. This can even be a setuid script that restarts a web server, for example. It’s quite safe, because OpenSSH will reject any variation of the command= text. Users possessing this key will only be able to execute the command that is explicitly allowed.

Aside from that, and possibly some fancy web-based tools or cron jobs, there aren’t may options left. At times users just need to be able to login and work.

It should go without saying that you need to stay up-to-date on patches. We won’t focus too much on that, aside from saying: automate! Securing a machine is an entirely different topic all together, but here are a few points to consider.

Enabling SELinux (Security-Enhanced Linux) is your first line of defense against unknown attacks. SELinux can prevent buffer overflows, as opposed to simply taking the “updates” path, which requires that a publicly known hole be fixed before some tries to exploit it. SELinux provides a significantly improved access system to limit programs from accessing things they don’t require to be operational. That, combined with overflow prevention makes it quite difficult to compromise a Linux system.

Further, on the issue of securing a multi-user machine, there is a much-debated precept: that users shouldn’t be able to see what processes are running, unless they own them. This restriction is simple to enable in Linux and the BSD’s, but does it really buy you anything? The answer is “maybe,” and at the same time, “not really.” To satisfy the maybe camp, consider a process’s arguments. When you run a command with a given set of arguments, the command as well as the arguments will show up in a ‘ps’ listing. If you have provided a password on the command-line for some reason, it will be visible to anyone running a ‘ps’ while your process is still running. Many people think that allowing users to see running daemon processes on a server will allow them to know what to try attacking. This information is trivial to obtain via other means anyway, so “not really.”

Every time this discussion starts, someone quickly suggests a chroot jail. The chroot command stands for “change root,” which does just that. If you run the command: ‘chroot /home/charlie /bin/bash’ then chroot will look for the shell in /home/charlie/bin/bash, and then proceed to lock you into that directory. The new root of the file system, for the lifetime of the bash shell, is /home/charlie. You now have zero access to any other part of the actual file system. Any available command, and its required libraries, needs to be copied into the chroot jail. Providing a usable environment is a ton of work. It’s actually easier to give each user their own Linux Xen or Solaris Zone instance. Really.

Finally we come to the restricted shells. The most popular, rbash, is a restricted bash shell. Setting a user’s shell to rbash will provide absolutely zero security. In theory, rbash will prevent users from running anything by specifying a full path, including ‘./’ (the current directory). This implies that it’s difficult for users to run commands, including scripts they write or downloaded exploits. Since $PATH is controlled globally, users can only run things in those locations. Unfortunately, /bin/ is going to need to be in their path, so all a user needs to do is run a new shell, and rbash is no longer in the picture: ‘exec bash’

One method of alleviating this is to give users only one item in their path, a directory the administrator created. Within the directory, simply place symlinks to all the authorized commands. This is nearly as cumbersome as setting up chroot, but much more tolerable.

Security isn’t convenient, and if it is, you’re doing something wrong.

There are certainly ways to prevent users from running downloaded programs, but in the end, the multi-user security of a system will depend on security of every piece of software installed. Preventing the exploits from being successful, a la SELinux, adds the most viable method of protection. Coupled with a frequently updated system, additional restrictions such as rbash aren’t generally necessary.

Feel free to share!

Hide Sites

Related posts:

1. The Perils of Sudo With User Passwords
2. Built-in Security with Cisco IPS
3. Understanding Linux Virtual Memory

The International Standards Organization (ISO) developed the OSI (Open Systems Interconnection) model. It divides network communication into seven layers. Layers 1-4 are considered the lower layers, and mostly concern themselves with moving data around. Layers 5-7, the upper layers, contain application-level data. Networks operate on one basic principle: “pass it on.” Each layer takes care of a very specific job, and then passes the data onto the next layer.

The physical layer, layer 1, is too often ignored in a classroom setting. It may seem simple, but there are aspects of the first layer that oftentimes demand significant attention. Layer one is simply wiring, fiber, network cards, and anything else that is used to make two network devices communicate. Even a carrier pigeon would be considered layer one gear (see RFC 1149). Network troubleshooting will often lead to a layer one issue. We can’t forget the legendary story of CAT5 strung across the floor, and an office chair periodically rolling over it leading to spotty network connectivity. Sadly, this type of problem exists quite frequently, and takes the longest to troubleshoot.

Layer two is Ethernet, among other protocols; we’re keeping this simple, remember. The most important take-away from layer 2 land is that you should understand what a bridge is. Switches, as they’re called nowadays, are bridges. They all operate at layer 2, paying attention only to MAC addresses on Ethernet networks. The common fledgling network admin always seem to mix up layers two and three. If you’re talking about MAC address, switches, or network cards and drivers, you’re in the land of layer 2. Hubs live in layer 1 land, since they are simply electronic devices with zero layer 2 knowledge. Layer two will have it’s own section in Networking 101, so don’t worry about the details for now, just know that layer 2 translates data frames into bits for layer 1 processing.

On the other hand, if you’re talking about an IP address, you’re dealing with layer 3 and “packets” instead of layer 2′s “frames.” IP is part of layer 3, along with some routing protocols, and ARP (Address Resolution Protocol). Everything about routing is handled in layer 3. Addressing and routing is the main goal of this layer.

Layer four, the transport layer, handles messaging. Layer 4 data units are also called packets, but when you’re talking about specific protocols, like TCP, they’re “segments” or “datagrams” in UDP. This layer is responsible for getting the entire message, so it must keep track of fragmentation, out-of-order packets, and other perils. Another way to think of layer 4 is that it provides end-to-end management of communication. Some protocols, like TCP, do a very good job of making sure the communication is reliable. Some don’t really care if a few packets are lost–UDP is the prime example.

And arriving at layer seven, we wonder what happened to layer 5 and 6. They’re useless. A few applications and protocols live there, but for the understanding of networking issues, talking about these provides zero benefit. Layer 7, our friend, is “everything.” Dubbed the “Application Layer,” layer 7 is simply application-specific. If your program needs a specific format for data, you will invent some format that you expect the data to arrive in, and you’ve just created a layer 7 protocol. SMTP, DNS, FTP, etc, etc are all layer 7 protocols.

The most important thing to learn about the OSI model is what it really represents. Pretend you’re an operating system on a network. Your network card, operating at layers 1 and 2, will notify you when there’s data available. The driver handles the shedding of the layer 2 frame, which reveals a bright, shiny layer 3 packet inside (hopefully). You, as the operating system, will then call your routines for handling layer 3 data. If the data has been passed to you from below, you know that it’s a packet destined for yourself, or it’s a broadcast packet (unless you’re also a router, but never mind that for now). If you decide to keep the packet, you will unwrap it, and reveal a layer 4 packet. If it’s TCP, the TCP subsystem will be called to unwrap and pass the layer 7 data onto the application that’s listening on the port it’s destined for. That’s all!

When it’s time to respond to the other computer on the network, everything happens in reverse. The layer 7 application will ship its data onto the TCP people, who will stick additional headers onto the chunk of data. In this direction, the data gets larger with each progressive step. TCP hands a valid TCP segment onto IP, who give its packet to the Ethernet people, who will hand it off to the driver as a valid Ethernet frame. And then off it goes, across the network. Routers along the way will partially disassemble the packet to get at the layer 3 headers in order to determine where the packet should be shipped. If the destination is on the local Ethernet subnet, the OS will simply ARP for the computer instead of the router, and send it directly to the host.

Grossly simplified, sure; but if you can follow this progression and understand what’s happening to every packet at each stage, you’re just conquered a huge part of understanding networking. Everything gets horribly complex when you start talking about what each protocol actually does. If you are just beginning, please ignore all that stuff until you understand what the complex stuff is trying to accomplish. It makes for a much better learning endeavor! In future Networking 101 articles we will begin the journey up the stack, examining each layer in detail by discussing the common protocols and how they work.

Feel free to share!

Hide Sites

Related posts:

1. Networking 101: More Subnets, and IPv6
2. Networking 101: Subnetting – Slice Up 32-bits
3. Networking 101: Layer 2, Link and Spanning Tree

What is a storage network?
A storage network is any network that’s designed to transport block-level storage protocols. Hosts (servers), disk arrays, tape libraries, and just about anything else can connect to a SAN. Generally, one would use a SAN switch to connect all devices, and then configure the switch to allow friendly devices to pair up. The entire concept is about flexibility: in a SAN environment you can move storage between hosts, virtualize your storage at the SAN level, and obtain a new level of redundancy than was ever possible with direct-attached storage.

A FC-SAN, or Fiber Channel SAN, is a SAN comprised of the Fiber Channel protocol. Think of Fiber Channel (FC) as an Ethernet replacement. In fact, Fiber Channel can transport other protocols, like IP, but it’s mostly used for transporting SCSI traffic. Don’t worry about the FC protocol itself for now; we’ll cover that in another article later on.

A fairly new type of SAN is the IP-SAN: an IP network that’s been designated as a storage network. Instead of using FC, an IP-SAN uses Ethernet with IP and TCP to transport iSCSI data. There’s nothing to stop you from shipping iSCSI data over your existing network, but an IP-SAN typically means that you’re using plumbing dedicated for the storage packets. Operating system support for the iSCSI protocol has been less than stellar, but the state of iSCSI is slowly improving.

Another term you’ll frequently see thrown around is NAS. Network Attached Storage doesn’t really have anything to do with SANs—they’re just file servers. A NAS device runs something like Linux, and serves files using NFS or CIFS over your existing IP network. Nothing fancy to see here; move along.

There is one important take-away from the NAS world, however. That is the difference between block-level storage protocols and file-level protocols. A block-level protocol is SCSI or ATA, where as file protocols can be anything from NFS or CIFS to HTTP. Block protocols ship an entire disk block at once, and it gets written to disk as a whole block. File-level protocols could ship one byte at a time, and depend on the lower-level block protocol to assemble the bytes into disk blocks.

Block-level protocols
A protocol always defines a method by which two devices communicate. Block storage protocols are no different: they define how storage interacts with storage controllers. There are two main block protocols used today: SCSI and ATA.

ATA operates in a bus topology, and allows for two devices on each bus. Your IDE disk drive and CD ROM are, you guessed it, using the ATA protocol. There are many different ATA standards, but we’ll cover just the important ones here. ATA-2 was also known as EIDE, or enhanced IDE. It was the first of the ATA protocol we know today. ATA-4 introduced ATAPI, or the ATA Packet Interface, which allows for CD ROM devices to speak SCSI-like on the same bus as a regular ATA device.

The neat thing about ATA is that the controllers are integrated. The only “traffic” sent over the ATA bus is plain electrical signals. The host operating system is actually responsible for implementing the ATA protocol, in software. This means that ATA devices will never, ever be as fast as SCSI, because the CPU has to do so much work to just talk to these devices. As far as SANs are concerned, ATA isn’t that important. There are some ATA-based devices that allow you to connect cheap disks, but they translate operations into SCSI before sending them out to the SAN.

SCSI, on the other hand, is very confusing. SCSI-1 and SCSI-2 devices were connected via a parallel interface to a bus that could support 8 or 16 devices, depending on the bus width. Don’t worry about the details unless you’re unfortunate enough to have some older SCSI gear lying around.

SCSI-3 separated the device-specific commands into a different category. The primary SCSI-3 command set includes the standard commands that every SCSI-3 device speaks, but the device-specific commands can be anything. This opened up a whole new world for SCSI, and it has been used to support many strange and wonderful new devices.

SCSI controllers normally contain a storage processor, and the commands are processed on-board so that the host operating system doesn’t become burdened to do so, as with ATA. Such a SCSI controller is called a Host Bus Adapter. In the SAN world, the FC card is always called an HBA.

The main thing to know about SCSI is that it operates in a producer/consumer manner. One SCSI device (the initiator) will initiate the communication with another device, which is known as the target. The roles can be reversed! Most people call this a command/response protocol, because the initiator sends a command to a target, and awaits a response, but not always. In asynchronous mode, the host (initiator) can simply blast the target with data until it’s done. The SCSI bus, parallel in nature, can only support a single communication at a time, so subsequent sessions must wait their turn. SAS, or Serial Attached SCSI, does away with this limitation by automatically switching back and forth.

SCSI is tremendously more complex, but that’s the gist of it.

We need to understand SCSI to know how our storage network is going to ship data. The SCSI protocol plays an enormous role in storage networking, so you may even want to look at it more in-depth.

Next up, we’ll begin talking about Fiber Channel itself, which, as chance would have it, is much more complex than Ethernet. This is certainly going to be a fun journey.

In A Nutshell:

- A FC-SAN is a network that uses Fiber Channel at Layer 2, instead of Ethernet, and is dedicated to moving around SCSI commands.
- SCSI initiator, generally a host’s storage controller, is called an HBA; the SCSI target is most often the storage device you’re talking to.
- iSCSI can transport SCSI over your existing network, but a network dedicated to iSCSI is called an IP-SAN.

Feel free to share!

Hide Sites

Related posts:

1. Understanding Fibre Channel
2. Networking 101: Understanding Layers
3. How Much Server do you Need?

We’re going to focus on standalone (non-blade) servers for the moment, but many aspects are also applicable to blade servers. Blade servers are wonderful for centralized management of the hardware, but the specs of the individual server blades can vary tremendously.

Hardware Management

Want to avoid trudging down to the datacenter late at night, or even worse, across the world if something breaks? Then don’t skimp on the management controller, lights out manager, or whatever the vendor is calling it. Many vendors ship a simple version by default: it may allow serial console access only, for example. Make sure to get the full-featured controller, because even if the hardware is only a few doors down, getting up from your desk should never be necessary.

If you aren’t thinking of switching vendors any time soon, you might think that the management interface will always work the same as it has on all your other servers. Unfortunately, that’s not the case. Sun x86 hardware, for example, has many different hardware management controllers to choose from. The more expensive and feature-rich servers have the better controllers, but don’t make the mistake of thinking the interface never changes. The unfortunate part is that you never know how well it works until you get a server on-site.

Hardware management comes in two forms: IPMI (most support), and the user interface. The user interface is more often than not, a Web-based java application that provides remote console access. Some are extremely buggy, and others work quite well from all Web browsers. We can’t make a recommendation, though, because these things change often.

Memory

Shucks, this one is a no-brainer: as much as you can afford. Within reason, that is. If you aren’t going to run virtual machines, and this server’s only job is to serve up some simple Web pages, then 16GB of RAM is likely overkill. Likewise, make sure you know what your application can support. Many java applications are limited to a heap size of 2 or 4GB.

It’s also overkill to purchase more than 4GB of RAM if you need to run a 32-bit operating system. Yes, Windows Server does some tricks and it can use more than 4GB, but it’s a huge performance it.

If virtualization is in your future, load up as much as possible. You also want to pay attention to how many DIMM slots the server has. The 8GB DIMMs are horribly expensive now, so you’ll probably want to stick with 4GB sticks. Just remember, if you fill all the slots in the server, the only memory upgrade path is to buy higher capacity DIMMs.

CPU

Do you want to run many threads at an even pace or just a few threads as fast as possible? Sun’s T2 processors aren’t fast by any measure, but they can run many threads at the same speeds consistently. These are ideal for database servers, but not for Web servers.

Will this server be executing a wide variety of processes over and over again, as opposed to just running the same big application server constantly? If so, make sure you pay attention to the amount of cache each core of the CPU has.

For virtualization, you want the fastest multi-core processors available, with the largest amount of L2 cache. Cache is very important as it minimizes the number of times the CPU needs to fetch data from slower RAM. It makes a very noticeable difference on heavily used servers.

Disks, Controllers, and RAID

If you need local storage, do pay attention to the type of disks you’re ordering. A SATA disk is likely to disappoint if you have an IO-heavy workload. SAS, and FC disks should perform equally well, since they are both SCSI disks underneath.

Even if you don’t need much local storage, you should always buy a server with a RAID controller that can mirror the operating system disks, unless you’re SAN booting of course. You don’t want the OS to crash just because of a failed disk. Likewise, if you’re keeping tons of local storage for some reason, make sure to get a RAID card that does RAID-5, so that you can at least lose one disk at a time without losing data. If performance is a concern you should really be using iSCSI or SAN storage, but you may also think about a RAID 0+1 configuration to avoid the slower RAID-5 parity calculations.

If you’re attaching to a SAN, make sure to include the correct HBA as well.

Networking

When servers started showing up with two or four gigabit NICs I must admit, I was confused. Why would someone need that many? Aside from large servers that do a lot of network IO, you might also want to separate out your iSCSI traffic from normal Ethernet. It’s also important these days to make sure that the network cards support TOE, or a TCP Offload Engine. This will task the network card with computing TCP checksums, freeing your CPUs for more important things.

In summary, most of these things may seem common sense, but you need to remember to ask all the right questions every time you spec a server. Here’s a good checklist:

• Adequate hardware management controller
• Enough (but not too much) RAM, that’s fast enough, but not faster than the CPU’s front-side bus
• Enough memory slots for expansion, if that seems likely
• Correct CPU for this server’s needs
• RAID-1 for the OS, and (optionally) other RAID levels for other local storage
• FC HBAs?
• Multiple gigabit NICs with TOE capabilities

Feel free to share!

Hide Sites

Related posts:

1. Managing Virtual Machine and Cloud Sprawl
2. Understanding Linux Virtual Memory
3. Squeeze Your Gigabit NIC for Top Performance

An oft-asked question in networking classes is “why can’t we just put everyone on the same subnet and stop worrying about routing?” The reason is very simple. Every time someone needs to talk, be it to a router or another host, they have to send an ARP request. Also, there’s broadcast packets that aren’t necessarily limited to ARP, which everyone hears. When there are only 255 devices on a /24 subnet, the amount of broadcast packets are fairly limited. It is important to keep this number low, because every time a packet destined for a specific host or a broadcast address is seen, the host must handle the packet. A hardware interrupt is created, and the kernel of the operating system must read enough of the packet to determine whether or not it cares about it.

Broadcast storms happen at times, mainly because of layer 2 topology loops. We’ll explain layer 2 topology issues in excruciating (actually, enlightening) detail in a future issue. When thousands of packets hit a computer at a time, slow and fast computers alike can become very slow. The kernel spends so much time handling interrupts that it doesn’t have much left for dealing with “trivial” things like making sure your web browser process gets a chance to run. So that, my friends, is why subnets are very important. This is also known as a broadcast domain, because it limits the amount of broadcasts that you will hear.

The natural follow-up question normally involves a host’s notion of a broadcast address and netmask. We hopefully understand that a host needs to understand what computers are on the same subnet. Those IP addresses can be spoken to directly, making a router unnecessary. When the netmask or broadcast address is incorrectly configured, you’ll quickly find that some hosts are unreachable.

The most common erroneous configuration happens when someone configure an IP address without specifying the netmask and broadcast address. For some reason, most operating systems don’t take the liberty of updating these things, even though one can be determined from the other. If you run ‘ifconfig eth0 130.211.0.1 netmask 255.255.255.0′ you might expect that everything is ready to go. Unfortunately, it’s very likely that your broadcast address was set to 255.255.0.0. It largely depends on the router’s configuration, but normally this results in all broadcast packets being dropped. Conversely, if the netmask is configured incorrectly, the computer wouldn’t know where the subnet starts and begins. If a computer thinks a host is on the same subnet when it actually isn’t, it will attempt to ARP for it instead of the router. Routers can be configured to handle this and pretend they are the host (called Proxy Arp), but normally the result is unreachable hosts.

Understand how the netmask is configured, to avoid this problem. Figuring out the network and broadcast address isn’t very difficult when you remember that the netmask simply means “cover some bits,” but deciphering netmask representation can induce a double-take. The netmask for a /24 network is 255.255.255.0, that’s easy. But what does 255.255.240.0 mean? The best way to decipher it is to begin with the masked off part. Comparing it to the /24, which had three octets masked, we see that 255.255.240.0 has two octets masked, and part of another. We know it’s between a /16 and a /24. We have to understand binary, and realize how many bits are masked. The last 16 bits are clearly part of the network portion. The third octet, 240, allows 16 IP addresses beyond the mask, so it must mean that four bits are left (2^4=16). The four remaining bits, plus the 16 bits used for the first two octets means that we’re dealing with a /20!

What about 1.0.0.0/255.255.255.248? We’re definitely in a land smaller than the /24 subnet. If we look at the remaining bits in the last octet, we can see that there are eight IP addresses available. Remember that only 2^3 can make eight, so we’re using all but three bits in the network portion. This is a /29 network. Of course, the easy ones are pretty clear: 255.255.255.128 allows half as many host addresses in the last octet compared to the /24 network, so it’s a /25.

On the topic of confusing netmasks, IPv6 addresses certainly have a place. The netmask isn’t really an issue–the same concept applies, just with larger numbers to remember. The real problem lies within the address representation itself; the IETF seemed to take pride in creating confusion. Typically an IPv6 address is represented in hex, or base-16. Our old friend IPv4 could represent an IP address in hex too, which would look like B.B.B.B for the address 11.11.11.11. Unfortunately, IPv6 isn’t quite that nice looking. To represent 128 bits, IPv6 normally breaks up the address into eight 16-bit segments.

An IPv6 address looks like: 2013:4567:0000:CDEF:0000:0000:00AD:0000. It does get a bit easier. For example, leading zeros are not written, and contiguous quads of zeros get collapsed to ::. Trailing zeros ,however, must be shown. This is a bit confusing, but the rules always allow for a non-ambiguous IP address. Leading zeros in each quad can always be removed, but the collapsing of contiguous blocks of zeros can only happen once per address. The above address with collapsed zeros will look like: 2013:4567:0000:CDEF::AD:0000. IPv6 provides 2^128 addresses, more than enough to allocate roughly 1000+ IP addresses per square meter of the earth.

If you remember the rules of binary, the address representation rules with IPv6, and a few simple subnets for reference, you’ll be Master of Subnets – the one who everyone asks for help.

Feel free to share!

Hide Sites

Related posts:

1. Networking 101: Subnetting – Slice Up 32-bits
2. Networking 101: IP addresses
3. Networking 101: Understanding Layers

The (wrong) idea is that you shouldn’t use the root account, because apparently it’s too “dangerous.” This argument usually comes from new Linux users and people that call themselves “network administrators,” but has no basis in reality. We’ll come back to that in a moment.

The concept behind sudo is to give non-root users access to perform specific tasks without giving away the root password. It can also be used to log activity, if desired. Role-based access control isn’t available in Linux, so sudo is a great alternative, if used properly. Solaris 10 has greatly improved RBAC capabilities; so you can easily allow a junior admin access to web server restart scripts with the appropriate access levels, for example. Sudo is supposed to be configured to allow a certain set of people to run a very limited set of commands, as a different user.

Unfortunately, sysadmins and home users alike have begun using sudo for everything. Instead of running ‘su’ and becoming root, they believe that ‘sudo’ plus ‘command’ is a better alternative. Most of the time, sysadmins with full sudo access just end up running ‘sudo bash’ and doing all their work from that root shell. This is a problem.

Using a user account password to get a root shell is a bad idea.

Why is there a separate root account anyway? It isn’t to simply protect you from your own mistakes. If all sysadmins just become root using their user password (running: sudo bash), then why not just give them uid 0 (aka root) and be done with it? For a group of sysadmins, the only reason they should want to use sudo is for logging of commands. Unfortunately, this provides zero additional security or auditing, because an attacker would just run a shell. If sysadmins are un-trusted such that they need to be audited, they shouldn’t have root access in the first place.

Surprisingly, the home-user rational makes its way into the workplace as well. The recurring argument is that running a root shell is dangerous. Partially to blame for this grave misunderstanding is X login managers, for allowing the root user to login. New users are always scolded and explained to that running X as root is wrong. The same goes for many other applications, too. As time progressed, people started remembering that “running as root” is wrong, passing this idology down to their children, but without any details. A genetic mutation may have occurred, but insufficient research has been done on that topic thus far. Now that Ubuntu Linux doesn’t enable a root account by default, but instead allows full root access to the user via sudo, the world will never be the same.

People praise sudo, while demeaning Windows at the same time for not having any separation of privileges by default. The answer to security clearly is a multi-user system with privilege separation, but sudo blurs these lines in its most common usage. The Ubuntu usage of sudo simply provides a hoop to jump through, requiring users to type their password more often than they’d like. Of course this will prevent a user’s web browser from running something as root, but it isn’t security.

We’d really like to focus on the Enterprise, where sudo has very little place.

The sudo purists, or sudoists, we’ll call them, would have you run sudo before every command that requires root. Apparently running ‘sudo vi /etc/resolv.conf’ is supposed to make you remember that you’re root, and prevent mistakes. Sudoists will also say that it protects against “accidentally left open root shells” as well. If there are accidental shells left on computers with public access, well that’s an HR action item.

Sudo atheists will quickly point out that using sudo without specifically defined commands in the configuration file is a security risk. Sudoists user account passwords have root access, so in essence, sudo has un-done all security mechanisms in place. SSH doesn’t allow root to login, but with sudo, a compromised user password removes that restriction.

In a true multi-user environment, every so often a root compromise will happen. If users can login, they can eventually become root, and that’s just a fact of life. The first thing any old-school cracker installs is a hacked SSH program, to log user passwords. Ideally, this single hacked machine doesn’t have any sort of trust relationship with other computers, because users are allowed access. The next time an administrator logs into the hacked machine, his user account is compromised. Generally this isn’t a big deal, but with sudo, this means a complete root compromise, probably for all machines. Of course SSH keys can help, as will requiring separate passwords for administrators on the more important (non user accessible) servers; but if they’re willing to allow their user account access to unrestricted root-level commands, then it’s unlikely that there’s any other security in place elsewhere.

As we mentioned, sudo has its place. Allowing a single command to be run with elevated privileges in an operating system that doesn’t support such things is quite useful. Still, be very careful about who gets this access, even for one item. As with all software, sudo isn’t without bugs.

For the love of security, please, we beg of you, do not use sudo for full root access. Administrators keep separate, non-UID 0 accounts for a reason, and it’s not for “limiting the mistakes.” Everything should be done from a root shell, and you should have to know an uber-secret root password to access anything as root.

Feel free to share!

Hide Sites

Related posts:

1. Multi-user Security in Linux
2. Back to Basics: Unix File Permissions
3. Zenoss: We Can Ditch Nagios Now

Unfortunately, the same tools don’t exist universally across all Unix variants. A few commonly underused ones do, however, and that is what we’ll focus on first.

Disk Activity
A common source of “slowness” is disk I/O, or rather the lack of available I/O. On Linux especially, it may be a difficult diagnosis. Often the load average will climb quickly, but without any corresponding processes in top eating much CPU. Linux counts “iowait” as CPU time when calculating load average. I’ve seen load numbers in the tens of thousands, on more than one occasion.

The easiest way to see what’s happening to your disks is to run the ‘iostat’ program. Via iostat, you can see how many read and write operations are happening per device, how much CPU is being utilized, and how long each transaction takes. Many arguments are available for iostat, so do spend some time with the man page on your specific system. By default, running ‘iostat’ with no arguments produces a report about disk IO since boot. To get a snapshot of “now” add a numerical argument last, which will prompt iostat to gather statistics for that number of seconds.

Linux will show number of blocks read or written per second, along with some useful CPU statistics. This is one particularly busy server:

 avg-cpu:  %user   %nice %system %iowait  %steal   %idle
 1.36    0.07    5.21   23.80    0.00   69.57

Device:   tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda       18.22     15723.35       643.25 65474958946 2678596632

Notice that iowait is at 23%. This means that 23% of the time this server is waiting on disk I/O. Some Solaris iostat output shows a similar thing, just represented differently(iostat -xnz):

    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
295.3   79.7 5657.8  211.0  0.0 10.3    0.0   27.4   0 100 d101
134.8   16.4 4069.8  116.0  0.0  3.5    0.0   23.3   0  90 d105

The %b (block) column shows that I/O to device d101 is 100% blocked waiting for the device to complete transaction. The average service time isn’t good either: disk reads shouldn’t take 27.4ms. Arguably, Solaris’s output is more friendly to parse, since it gives the reads per second in kilobytes rather than blocks. We can quickly calculate that this server is reading about 19KB per read by dividing the number of KB read per second by the number of reads that happened. In short: this disk array is being taxed by large amounts of read requests.

Vmstat
The ‘vmstat’ program is also universally available, and extremely useful. It, too, provides vastly different information among operating systems. The vmstat utility will show you statistics about the virtual memory subsystem, or to put it simply: swap space. It is much more complex than just swap, as nearly every IO operation involves the VM system when pages of memory are allocated.A disk write, network packet send, and the obvious “program allocates RAM” all impact what you see in vmstat.

Running vmstat with the -p argument will print out statistics about disk IO. In Solaris you get some disk information anyway, as seen below:

 kthr      memory            page            disk
 faults      cpu  r b w   swap
free  re  mf pi po fr de sr m0 m1 m2 m7
in   sy   cs us sy id  0 0 0 7856104 526824 386 2401 0 0 0  0  0  3  0  0  0
16586 22969 12576 8 9 83  1 0 0 7851344 522016 18 678 32 0  0  0  0  2
0  0  0 13048 11737 10197 7 6 86  0 0 0 7843584 514128 76 3330 197 0
0 0  0  2  0  0  0 4762 131492 4441 16 8 76

A subtle, but important differences between Solaris and Linux is that Solaris will start scanning for pages of memory that can be freed before it will actually start swapping RAM to disk. The ‘sr’ column, scan rate, will start increasing right before swapping takes place, and continue until some RAM is available. The normal things are available in all operating systems; these include: swap space, free memory, pages in and out (careful, this doesn’t mean swapping is happening), page faults, context switches, and some CPU idle/system/user statistics. Once you know how to interpret these items you quickly learn to infer what they indicate about the usage of your system.

The two main programs for finding “slowness” are therefore iostat and vmstat. Before the obligatory tangent into “what Dtrace can do for you,” here’s a few other tools that no Unix junkie should leave home without:

lsof

Lists open files (including network ports) for all processes

netstat

Lists all sockets in use by the system

mpstat

Shows CPU statistics (including IO), per-processor

Dtrace
We cannot talk about system visibility without mentioning Dtrace. Invented by Sun, Dtrace provides dynamic tracing of everything about a system. Dtrace gives you the ability to ask any arbitrary question about the state of a system, which works by calling “probes” within the kernel. That sounds intimidating, doesn’t it?

Let’s say that we wanted to know what files were being read or written on our Linux server that has a high iowait percentage. There’s simply no way to know. Let’s ask the same question of Solaris, and instead of learning Dtrace, we’ll find something useful in the Dtrace ToolKit. In the kit, you’ll find a few neat programs like iosnoop and iotop, which will tell you which processes are doing all the disk IO operations. Neat, but we really want to know what files are being accessed so much. In the FS directory, the rfileio.d script will provide this information. Run it, and you’ll see every file that’s read or written, and cache hit statistics. There’s no way to get this information in other Unixes, and this is just one simple example of how Dtrace is invaluable.

The script itself is about 90 lines, inclusive of comments, but the bulk of it is dealing with cache statistics. An excellent way to start learning Dtrace is to simply read the Dtrace ToolKit scripts.

Don’t worry if you’re not a Solaris admin: Dtrace is coming soon to a FreeBSD near you. SystemTap, a replica of Dtrace, will be available for Linux soon as well. Until then, and even afterward, the above mentioned tools will still be invaluable. If you can quickly get disk IO statistics and see if you’re swapping the majority of system performance problems are solved. Dtrace also provides amazing application tracing functionality, and if you’re looking at the application itself, you already know the slowness isn’t likely being caused by a system problem.

Soon, I’ll publish a few Dtrace tutorials.

Some things have surely been left out – discuss below!

Feel free to share!

Hide Sites

Related posts:

1. Back To Basics: Unix Differences in Performing Tasks
2. Back to Basics: Unix File Permissions
3. Working With Unix Variant Differences

In this final Back to Basics With Unix piece, we’d like to spend some time talking about some common, routine sysadmin tasks and how they differ between Unix variants.

Sure, you can clunk around and change configuration files to mostly make something work on a foreign system. But will those changes remain after security patches get applied and stomp all over your work? Did you just change a file that was meant to never change, because there’s a separate file for local modifications? If you’re not familiar with “how it’s done” in that particular OS, it’s as likely as not.

GUIs
Yes, I make fun of GUI configuration utilities. People that don’t understand systems often use them and “get by,” but they cannot fix things when they break, unless the GUI tool can do it for them. That said, they do have their place. When learning a new system, it often makes sense to use the provided configuration utilities, as you know without a doubt they will adjust the necessary setting they way the OS wants it done. Here’s a list of some handy general administration GUIs:

- AIX: smitty (does pretty much everything)
- FreeBSD: sysinstall (not recommended for use after the initial install, but it works)
- HP-UX: sam (like AIX’s smitty)
- Linux: system-config, webmin and many others (distro-dependant)
- Solaris: admintool, wbem (use with caution)

Often, these tools still don’t do what you need. They certainly don’t help you learn a system unless you take the time to examine what the tool actually changed. Let’s start off with the basics: gathering system information and managing hardware. It can be a nightmare to add a disk to a foreign system, so hopefully this list will get you steered in the proper direction.

Show hardware configuration:
- AIX: lsdev, lscfg, prtconf
- FreeBSD: /var/run/dmesg.boot, pciconf
- HP-UX: ioscan, model, getconf, print_manifest
- Linux: dmesg, lspci, lshw, dmidecode
- Solaris: prtconf, prtdiag, psrinfo, cfgadm

Note that ‘dmesg’ is a circular kernel buffer on most systems, and after the machine has been up for a while the boot information listing devices gets overwritten. FreeBSD thoughtfully saves it in dmesg.boot for you, but in other systems you’re left relying on the above-mentioned exploratory tools.

Add a new device (have the OS discover it without a reboot):
- AIX: cfgmgr
- FreeBSD: atacontrol, camcontrol
- HP-UX: ioscan, insf
- Linux: udev, hotplug (automatic)
- Solaris: devfsadm, disks, devlinks (all a hardlink to the same binary now)

If you connect a new internal disk and need it recognized, you should not need to reboot in the Unix world. The above commands will discover new devices and make them available. If you’re talking about SAN disks, the utilities are mostly the same, but there are other programs that make the process much easier and also allow for multipathing configurations.

Label and partition a disk:
- AIX: mkvg then mklv
- FreeBSD: fdisk or sysinstall
- HP-UX: pvcreate then lvcreate, or sam
- Linux: fdisk or others
- Solaris: format or fmthard

Of course, you’ll also want to create a file system on your new disk. This is newfs or mkfs everywhere, with the exception of AIX which forces you to use crfs. The filesystem tab file, which describes file systems and mount options, vary a bit as well. In Linux, FreeBSD, and HP-UX it is /etc/fstab, Solaris uses /etc/vfstab, and AIX references /etc/filesystems. We spent so much time on filesystems and hardware because that’s the generally the biggest hurdle when learning a new system, and when you’re needing to do it, often you’re in a hurry.

Other tasks may or may not be covered by GUI utilities in the various flavors of Unix, so here’s a few more that we deem crucial to understand.

Display IP information and change IP address permanently:
- AIX: ifconfig/lsattr; smitty or chdev
- FreeBSD: ifconfig; /etc/rc.conf
- HP-UX: ifconfig/lanadmin; set_params
- Linux: ‘ip addr’; /etc/sysconfig/network or /etc/network/interfaces
- Solaris: ifconfig; edit /etc/hosts, /etc/hostname.*

Linux will of course vary, but those two files cover the most popular distros.

When taking over a foreign system, we frequently want to two two things: install missing software (like GNU utilities), and verify that the system is up-to-date on security patches. Where to get packages and where to gete latest security patches varies too much to cover here—you’ll likely need to search to the OS in question—but the way you install packages and show installed patches is extremely useful to know.

List installed patches:
- AIX: instfix, oslevel
- FreeBSD: uname
- HP-UX: swlist
- Linux: rpm, dpkg
- Solaris: showrev

Install packages:
- AIX: smitty, rpm, installp
- FreeBSD: pkg_add, portinstall, sysinstall
- HP-UX: swinstall
- Linux: rpm, yum, apt, yast, etc.
- Solaris: pkgadd

As you can see, things vary immensely between the Unix variants. Even within all of Linux you can easily find yourself lost. Google is a friend to all sysadmins, but too often the conceptual questions go unanswered. Here’s a general rule of thumb, and something I’ve seen done incorrectly too many times: if you see a configuration file in /etc/, say syslog.conf, and there is an accompanying syslog.d directory, you are not supposed to edit the syslog.conf file directly. The same goes for pam.conf and pam.d. Each service will have their own file within the .d directory, and that is where they are configured.

The .d directory example is mostly applicable to Linux, but be sure to pay attention when you see similar multi-config layouts anywhere else. Future sysadmins using the system will thank you if the OS’s conventions are followed and it’s easy to identify customizations. It also means that your changes aren’t likely to be stomped over by updates.

Feel free to share!

Hide Sites

Related posts:

1. Back to Basics: Unix System Stats Utilities
2. Back to Basics: Unix File Permissions
3. Working With Unix Variant Differences

The LDAP protocol supports just a few fairly easy to understand operations. Knowing what’s available provides administrators with the ability to surmise how various applications are using LDAP, troubleshoot issues, and construct their own search queries and filters more effectively.

A client, be it a PHP script, command-line program like ldapsearch, or LDAP libraries for user authentication in Unix, will connect to a server on port 389 (or 636 with SSL), and send one of roughly a dozen operation requests. The following operations define how the LDAP protocol works:

Bind
Binding is the pivotal concept to understand. It is optional, depending on access control restrictions defined in the server. The act of binding is authentication: it sends a user’s DN and password. Binding anonymously may not allow access to all directory entries, or it may not be allowed at all, again depending on how the server is configured.

Search or Compare
Search is used to both list entries and search for them. Searching supports a number of parameters, which define how the search is carried out.

• Base: object to start at
• Scope: how much to search; one entry only, a single level below, or the entire subtree below
• Filter: limit (optimize) search based on attribute/value or object filters
• derefAliases: whether or not to follow alias entries
• attributes: which attributes to return (none specified means return all)
• sizeLimit, timeLimit: number of entries to return, and a time limit
• typesOnly: just return the attribute types, not the actual values

Add, Delete, Modify (Update types)
Updating an LDAP entry can take the form of three operations: add, delete, or modify. Actually four, because modify can modify either an entry or a DN. As was explained last week, modifying the DN simply means moving an entry. Add and Delete do the obvious.

Extended Operations
Extended operations can be added at will. For example, many servers support the STARTTLS command, tells the server to start a secure connection.

Abandon
An Abandon operation will abandon any operation, hopefully. There is no guarantee the server will honor an abandon request.

Unbind
Unbind abandons any outstanding operations and disconnects a client.

As mentioned before, LDAP is pretty simple. You can connect, search or update entries, and then disconnect. Nearly every LDAP communication follows those three steps.

So how does one connect? The majority of connections to an LDAP server are made by LDAP client programs on a Unix machine, in environments that use LDAP for server directory services. Web applications often gather and display directory information, or use LDAP to authenticate people. Aside from those, LDAP connections can also be made by Perl or even shell scripts to manage the information within. When you want to manually search or update information, you will generally use some common tools such as ldapsearch, ldapvi, or ldapmodify.

Searching an LDAP directory can be challenging if you’ve never done it before. The command-line utilities have a few arguments that aren’t optional. Let’s take a look at an ldapsearch example:
ldapsearch –h ldapserver.example.com –b ou=People,dc=example,dc=com uid=charlie

The ldapsearch program, in most Unix/Linux environments, take the same arguments. You must specify a server (-h) and a base (-b) to begin searching at. The base can be as broad or as specific as you’d like. We’ve chose to start searching at the ou (organizational unit) called people, withing the domain components used to designate our portion of the tree. I could have left out the ou=People portion, but if there is anything else at the level below dc=example, then it would search through those too. It faster to specify the subtree as close to the entry as possible, if you know it. Finally, the last argument was a search filter. I stated that I was interested in all entries where the value of the attribute uid was “charlie.”

The previous example used an anonymous bind, since a DN wasn’t specified. If you need to search information that is restricted to certain people, then specifying –D followed by a user DN will cause ldapsearch to bind as that user, and prompt for a password.

Search filters can be quite complex. When you’re searching manually with ldapsearch, you probably won’t get very complex. When writing a script that could potentially be run very often, you want as optimal a search as possible. Search filters can specify many thing, including what object classes to look for. It’s all about providing as many hints to the server as possible, so that it may make best use of its search indexes.

A search filter has a few basic operators, including “and” and “or” operators. The general syntax is similar to RPN (for math geeks) or functional languages (for programmers). If we want to search for a person whose given name is Bob, and mail attribute is also bob, we could use a search filter of:
(&(givenName=bob)(mail=bob))

If we wanted to return all entries where either bob is the givenName or the mail attribute, we could simply specify: (|(givenName=bob)(mail=bob))
Notice the | symbol, followed by two or more attribute/value pairs. In reality, we would really want to specify what object class we’re looking for, if this was used in a script: (&(objectClass=person)(|(givenName=bob)(mail=bob)))
The filter ensures that the objectClass is person, and the other nested statement is true. Again, we’re just trying to give as many hints to the server as possible.

An LDAP URL is similar, but it contains all the information necessary to both identify a server and perform a search. URLs similar to this one, or portions of it, may be required to configure some LDAP clients: ldap://ldap.example.com/ou=People,dc=example,dc=com?one?(pod=evil)

The general format is: ldap://host:port/BaseDN?attributes?scope?filter

LDAP is extremely powerful, and is certainly the best place for server-based directory information and people information. If you already live in an LDAP environment, hopefully you have a better understanding now. If you’re pondering an LDAP deployment, go and unleash the power now.

Feel free to share!

Hide Sites

Related posts:

1. An Introduction to LDAP
2. Managing Virtual Machine and Cloud Sprawl
3. Zenoss: We Can Ditch Nagios Now

Systems

First, recall that Unix started off in research labs, and two main flavors came about: System V (SysV), and BSD. SysV (five, not “vee”) spawned from AT&T Unix, in their fourth version, SVR4. BSD, from Berkeley, is the competing Unix variant. They both derived from the same Unix from Bell labs, but quickly diverged. Despite POSIX efforts, there are still BSD and SysV systems today, and their functionality still diverges.

Most operating systems are pretty clearly associated with one or the other, and generalizations about BSD vs. SysV prove correct. FreeBSD is the main branch from the traditional BSD, soon followed by NetBSD and OpenBSD. Then OS X came about, which was loosely based on FreeBSD (but is very BSD-like). On the SysV side of the house, AIX, IRIX, and HP-UX were the main variants. In short: commercial entities focused on SysV, academics focused on BSD.

Linux, however, is an oddball. Linux certainly adopted many SysV methodologies, but these days it is also very BSD-like. Sun Solaris, too, is confusing. SunOS started off as BSD, but SunOS 4 was the last BSD version; SunOS 5.x (aka Solaris) is now SysV. The details are much crazier than I’ve alluded to here, and we probably don’t want yet another Unix history lesson. A fun place to start for further reading is the Wikipedia page on Unix_wars.

Fundamental Differences
It has been said that one can tell which system they are using based on two indicators: whether or not the system boots with inittab, and the format of their accounting file. Process accounting isn’t really used any longer, and most people don’t even know what it’s for, so that’s mostly moot. The boot system, however, is still critical to understand.

SysV booting means you use inittab. The init program, when run by the kernel, will check /etc/inittab for the initdefault entry, and then boots to the runlevel defined there. Entering a runlevel means that each startup script in the directory will be run in order. Sequentially, and slowly. Sun was so annoyed with this they implemented a mechanism to fire up services in parallel, among other things, with the Service Management Facility (SMF). Ubuntu Linux implemented Upstart, which basically works around the sequential nature of init scripts too.

BSD booting means that init simply runs /etc/rc, and that’s all. Well, it used to. Soon BSD systems implemented rc.local, so that software and sysadmins alike could implement changes without fear of harming the critical system startup routines. Then /etc/rc.d/ was implemented, so that each script could live separately, just like SysV init scripts. Traditionally, BSD-style scripts didn’t take arguments, because there are no runlevels, and they only run once: on startup. There are still no runlevels in BSD, but the startup scripts generally take “start” and “stop” arguments, to allow sysadmins and package management tools to restart services easily.

Command Arguments
The most frustrating, and quickest to surface differences between SysV and BSD, are in the traditional utilities. Some common commands take very different arguments, and even have some very different functionality. This isn’t so important if you’re in Linux now, as it generally supports both, but once you find yourself in BSD-land, you’re up for some confusion.

The first command people usually run into is ‘ps.’ The arguments differ:

• SysV: ps –elf
• BSD: ps aux

Linux supports both, BSD does not. Often we may want to list all processes owned by a particular user. In BSD, you must run, “ps aux |grep username” but in SysV you can run, “ps –u username.” Just plain ‘ps’ will list your own processes in both flavors.

Another commonly noticed difference is with the ‘du’ command. Not because some older systems don’t support the –h argument to provide human-readable output, but because they display different things.

• SysV: shows the amount available in 512-byte blocks
• BSD: nice output showing size in bytes and percentage used

Printing in BSD is always confusing for SysV users, and vice-versa. Again this isn’t as common, since newer OSes support both, but it’s noteworthy nonetheless. BSD systems traditionally used lpr, lpq, and lprm to administer print jobs, whereas SysV had lp, lpstat, and cancel. Most systems adopted the BSD style, since lpr-ng (next generation) provided these commands, and CUPS subsequently adopted the BSD variants.

Other programs, such as du, who, ln, tr and more will have slight differences between SysV and BSD. Heck, the differences between the various Unix standards are confusing enough that a single Unix variant may have multiple directories of utilities. Take a look at Solaris’s /usr/ucb, /usr/xpg4, and /usr/xpg6 directories. Each standard they support, which has differences from POSIX, is documented and implemented in a separate location. Too bad Linux doesn’t comply with any standards.

In the end, the differences outlined here are probably the only ones anyone would ever notice. The nuances between du, for example, may be applicable for people writing shell scripts for systems administration procedures. The differences do turn up often enough to be mentionable, but in reality this level of work requires reading manual pages so often that they’d figure it out quickly. User-level utilities are “similar enough” with the exception of ps.

There are so many other differences in system maintenance procedures that those are more frequently focused on. Once the ‘ps’ hurdle is out of the way, and you understand how the system boots, the main problems are more conceptual, as in “how do I add a user.” These vary by OS, and also by distribution of Linux.

Come back next week to learn about the different ways Unix-like operating systems facilitate systems administration tasks.

Feel free to share!

Hide Sites

Related posts:

1. Back To Basics: Unix Differences in Performing Tasks
2. Back to Basics: Unix System Stats Utilities
3. Back to Basics: Unix File Permissions

But first, let’s get one thing straight: there is no Class in subnetting. In the olden days, there was Class A, B, and C networks. These could only be divided up into equal parts so VLSM, or Variable Length Subnet Masks, were introduced. The old Class C was a /24, B was a /16, and A was a /8. That’s all you need to know about Classes. They don’t exist anymore.

An IP address consists of a host and a network portion. Coupled with a subnet mask, you can determine which part is the subnet, how large the network is, and where the network begins. Operating systems need to know this information in order to determine what IP addresses are on the local subnet and which addresses belong to the outside world and require a router to reach. Neighboring routers also need to know how large the subnet is, so they can send only applicable traffic that direction. Divisions between host and network portions of an address are completely determined by the subnet mask.

Classless Internet Domain Routing (CIDR), pronounced “cider,” represents addresses using the network/mask style. What this really means is that an IP address/mask combo tells you a lot of information:

network part / host part
0000000000000000/0000000000000000

The above string of 32-bits represents a /16 network, since 16 bits are masked.

Throughout these examples (and in the real world), certain subnet masks are referred to repeatedly. They are not special in any way; subnetting is a simple string of 32 bits, masked by any number of bits. It is, however, helpful for memorizing and visualizing things to start with a commonly used netmask, like the /24, and work from there.

Let’s take a look at a standard subnetting table, with a little bit different information:

Because of the wonders of binary, it works out that a /31 has two IP addresses available. Imagine the subnet: 2.2.2.0/31. If we picture that in binary, it looks like:

00000010.00000010.00000010.00000000 (2.2.2.0)
11111111.11111111.11111111.11111110 (31)

The mask is “masking” the used bits, meaning that the bits are used up for network identification. The number of host bits available for tweaking is equal to one. It can be a 0 or a 1. This results in two available IP addresses, just like the table shows. Also, for each additional bit used in the netmask (stolen from the network portion), you can see that the number of available addresses gets cut in half.

Let’s figure out the broadcast address, network address, and netmask for 192.168.0.200/26. The netmask is simple: that’s 255.255.255.192 (26 bits of mask means 6 bits for hosts, 2^6 is 64, and 255-64 is 192). You can find subnetting tables online that will list all of this information for you, but we’re more interested in teaching people how to understand what’s happening. The netmask tells you immediately that the only part of the address we need to worry about is the last byte: the broadcast address and network address will both start with 192.168.0.

Figuring out the last byte is a lot like subnetting a /24 network, but you don’t even need to think about that, if it doesn’t help you. Each /26 network has 64 hosts. The networks run from .0 to .64, .65 to .128, .128 to .192, and from .192 to .256. Our address, 192.168.0.200/26, falls into the .192 to .256 netblock. So the network address is 192.168.0.192/26. And the broadcast address is even simpler: 192 is 11000000 in binary. Take the last six bits (the bits turned “off” by the netmask), turn them “on”, and what do you get? 192.168.0.255. To see if you got this right, now compute the network address and broadcast address for 192.168.0.44/26. (Network address: 192.168.0.0/26; broadcast 192.168.0.63).

It can be hard to visualize these things at first, and it helps to start with making a table. If you calculated that you wanted subnets with six hosts in each of them, (eight, including the network and broadcast address that can’t be used) then you can start making the table. The following is 2.2.2.0/29, 2.2.2.8/29, 2.2.2.16/29 and the final subnet of 2.2.2.249/29.

In reality, you’re much more likely to stumble upon a network where there’s three /26′s and the final /26 is divided up into two /27′s. Being able to create the above table mentally will make things much easier.

That’s really all you need to know. It gets a little trickier with larger subnets in the /16 to /24 range, but the principal is the same. It’s 32 bits and a mask. Do, however, realize that there are certain restrictions governing the use of subnets. We cannot allocate a /26 starting with 10.1.0.32. If we utter the IP/mask of 10.1.0.32/26 to most operating systems, they will just assume we meant 10.1.0.0/26. This is because the /26 space requires 64 addresses, and they must start at a natural bit boundary for the given mask. In the above table, what would 2.2.2.3/29 mean? It means you meant to say 2.2.2.0/29.

Those tricky ones do demand a quick example. Remember how the number of IP addresses in a subnet gets halved when you take another bit from the network side to create a larger mask? The same concept works in reverse. If we have a /25 that holds 128 hosts, and steal a bit from the host (netmask) portion, we now have a /24 that holds 256. Google for a “subnet table” to see the relationship between netmasks and network sizes all at once. If a /16 holds 65536 addresses, a /17 holds half as many, and a /15 holds twice as many. It’s tremendously exciting! Practice, practice, practice. That’s what it takes to understand how this works. Don’t forget, you can always fall back to counting bits.

The next step, should you want to understand more about subnets, is to read up on some routing protocols. We’ll cover some of them soon, but in the next installment of Networking 101, we’re starting our trip up the OSI model.

Feel free to share!

Hide Sites

Related posts:

1. Networking 101: IP addresses
2. Networking 101: More Subnets, and IPv6
3. Networking 101: Understanding Layers

LDAP is actually quite simple, even though it does make use of the ITU X.500 standard—a notoriously complex specification. X.500 directories were accessed via DAP, or Directory Access Protocol. It was large, complex, and unruly, so Lightweight DAP was created. That’s almost accurate; in fact, LDBP (Lightweight Directory Browsing Protocol) came first, because all you could do was search. When the functionality to modify entries was implemented, LDAP was born.

LDAP Structure
A directory can be defined as a set of objects with similar attributes, organized in a hierarchical manner. Sorry, but I must use the old phone book analogy now. In a phone book, an object is a person, and each person has a set of similar attributes: a phone number and perhaps an address. LDAP is the same, but you may make use of many other types of attributes.

LDAP directories are organized in a tree manner, and the design often will reflect organizational or geographic boundries. X.500 tells us:

• A directory is a tree of directory entries
• An entry contains a set of attributes
• An attribute has a name, and one or more values.

Attributes are defined in a schema, which specifies what types of things can be attributes and whether or not you can multiple values per attribute.

Every entry in a directory has a unique identifier, called the Distinguished Name (DN). The Relative DN (RDN) is part that specifies the current attribute you’re dealing with, sort of like a relative path in Unix (./file). The DN, then, would be a full path (/var/lib/file). A sample directory entry’s DN, therefore, would look like: cn=”john doe”,dc=mytree. The RDN is cn=”john doe”, and the DN is the full path, starting at the top of the tree. A “cn” simply means the “common name” that the entry is referred to as, and “dc” is the “domain component.”

You will often see examples of LDAP structures that use DNS names for the domain component, such as: dc=example,dc=com. This is not necessary, but since DNS itself often implies organizational boundaries, it usually makes sense to just use your existing naming structure. One final note about a DN; it changes over time. If you change a DN, you’re effectively moving an entry in the tree. Some LDAP servers support unique identifiers that will track the movement of entries, but you often don’t need to care. Just know that even though a DN is unique, it changes over time.

LDIF Example
A sample directory entry (of a person) looks like this:
dn: cn=John Doe,dc=myplace
cn: John Doe
givenName: John
sn: Doe
telephoneNumber: +1 555 555 1234
telephoneNumber: +1 555 555 5555
mail: john@example.com
manager: cn=Bob Smith,dc=example,dc=com
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
objectClass: top

All of the attributes (objects) listed above are associated with the DN; it is a single directory entry. Objects (givenName, sn, etc) are defined by schemas. Every entry must list the objectClass that every attribute is using. For example, organizationalPerson defines what values can live in the attribute called “manager.” If the objectClass wasn’t listed, the LDAP server wouldn’t know what values were allowed, so it wouldn’t allow you to define an attribute called manager.

The example above is an LDIF, LDAP Data Interchange Format, entry. That is the entire LDAP entry in text form. You could insert the data into a directory, and in fact, this is exactly what a backup of your directory looks like. It’s just text, and that’s all there is to an LDAP entry. Well almost: most servers also support aliases and references. An LDAP alias can point to another local entry in the same directory, to avoid duplicating information. A reference will provide a new DN to an LDAP client and tell it to go ask another server. Some LDAP servers even support chained references, where the server will go get the answer and return it to the client; the client never knows a referral has taken place. Regardless, LDAP entries are quite simple.

LDAP Schemas
A schema defines the attribute types that entries can contain, as well as the format of their values. It will specify that: Mail contains a well-formed e-mail address, Photo contains a JPEG image, and uidNumber contains an integer, for example.

Here is an example schema we recently created:
attributeTypes: ( 1.1.1.2.1
NAME 'pod'
DESC 'A pod for people to belong in'
EQUALITY caseIgnoreMatch
SYNTAX 1.3.6.1.4.1.1466.115.121.1.15
)
objectClasses: ( 1.1.1.1.1
NAME 'podPerson'
DESC 'A person who belongs in some pods'
SUP top
MUST cn
MAY pod
)

The objectClass is defined, as well as the allowed attributeTypes. Each schema must have a unique OID (object identifier), which is part of the way X.500 works (SNMP is the same way). We created an objectClass called podPerson, gave it a description, an said the entry must contain a ‘cn,’ and may contain a ‘pod.’ The pod attribute can contain any value, because the only restriction specified is that case doesn’t matter. After loading that scheme into our LDAP server, we could then add a ‘pod’ attribute to each person entry.

Since LDAP is so lightweight and simple, it is not suitable for a few things. It’s very tempting to store tons of data in LDAP, since so many applications can reference LDAP. Unix machines can use LDAP for passwd, shadow, group, netgroup, protocols, and just about everything in nsswitch.conf. LDAP is a database, so print accounting programs, configuration management systems, and just about everything that stores data in a DB will support LDAP. It’s fine for most of these things, but LDAP is not ideal for replicating a relational database. The data in LDAP is not ordered, which means you could get results in any order. If your application is querying for only one result at a time, this is fine, but if multiple results are common and order is important, LDAP just won’t work.

Check back next week (i.e. follow me on Twitter and subscribe via RSS, links at top-right of this page) for a look at the protocol, and some practical examples of querying and using LDAP data.

Feel free to share!

Hide Sites

Related posts:

1. LDAP: Understand the Protocol and Work With Entries
2. Back to Basics: Unix File Permissions
3. Multi-user Security in Linux

The Concept
At the most basic level, there are three types of access:

• Read – the ability to open a file and read it
• Write – the ability to write the file
• Execute – the ability to execute (run) the file

Directories, though similar, are subject to special rules. Write permissions on a directory imply that you can create new files and directories within. Execute permissions are required to ‘cd’ into the directory, and read permissions are required to list the contents (‘ls’).

You will generally see permissions represented as r, w, or x; for read, write, and execute. Running ‘ls –al’ on the command line will show three sets of these strung together.

For example: -rwxr-xr-x

The dash means that the permission is not set. The first place is always reserved for special identifiers, like ‘d’ for directories or ‘c’ for character devices. The next place begins the actual permissions, for the user, group, and other categories.

Every access control in Unix is based on “who you are.” The user is identified by the uid (user ID), as defined by a person’s user account. The third field in the /etc/password file, for example, specifies what a user’s uid is. Similarly, every user belongs to a default group, as identified by the fourth field in the passwd file. Users can belong to many groups, but they’re always a member of their default group.

The above example of -rwxr-xr-x means that the owner of the file may read, write and execute it, the group members may read or execute it, and everyone else on the system may also read or execute the file.

A full example, from the output of ‘ls -l’ is:
-rw-r–r–  1 charlie root        164 2006-12-10 23:51 test.js

The file named test.js is owned by me with read and write permissions, is set to the root group who can only read it, and also allows everyone else to read it.

How it Really Works
That’s basically enough to get by, but being able to understand the more advanced modes of file permissions, your umask, and the numeric representation demands a full understanding. In reality, there are 8-bits available for each type of attribute. Take a look at Figure 1 and note that wherever you see a 1 in the binary column, a corresponding permission will exist.

As you can see, if a “bit” in a certain position of the binary representation is set, the permissions in that space are activated. The number column is the octal representation, and the “Binary” column is how it really works, from the operation system’s perspective.

Example time. Let’s say we wish to give ourselves read/write/execute permissions, the group read/execute, and everyone else read/execute permissions. The following commands both do the same thing:

• chmod u+rwx .; chmod go+rx .
• chmod 755 .

Since we know that setting ‘5’ means rx, we can simply say ‘5’ instead of ‘rx.’ The real advntage to knowing the octal representation is that we can set any arbitrary permissions with a single command. Running the chmod command using the mnemonic requires that we run it each time for each set of permissions.

Likewise, to set our umask, we must know how the permissions are numerically represented. The umask is the default mode with which files and directories will get created. It’s a mask, so if we want to create all files with permissions like 755, we must take the mask. Simply subtract 7 from each item, and 022 reveals itself as the magic setting. See the umask man page for further details.

Advanced Modes
There are, in fact, three other modes you can set on a file or directory. All Unixes support the following:

• 4000 set user id (suid) on execution
• 2000 set group id on execution
• 1000 the sticky bit

If suid is enabled, the permissions look like: -rws——
This means that when the file is executed, it will run with the permissions of the owner of the file. It’s dangerous, but some times necessary and quite useful. For example, a file suid and owned by root will always run as root.

When sgid is enabled, the permissions look like: -rwxrws—
When set on a directory, sgid means that all files created within the directory will have the gid set to the current directory’d gid. This is handy when sharing files with other people, who will often forget to give other members read or write permissions.

The sticky bit looks like: -rwx——T
When the sticky bit is enabled, only the owner of the file can change its permissions or delete it. Without the sticky bit, anyone with write permissions can change the modes (including ownership) or delete a file. This one is also handy when sharing files with a group of people.

There are other tidbits of information, once you get into the nuts and bolts of Unix file permissions too. For example, you can also set ACL attributes, which get horribly complex. Yes, you can give individual users access to your files, but it’s better not to. Creating a new group and sticking to general permissions can accomplish most things. Often the extended attributes aren’t necessary, and ACLs likely won’t work over NFS if you’re using Linux.

Spend some time with the chmod manual page to master tricky parts, if they still aren’t clear. It will also mention some implementation-specific limitations you may need to be aware of.

Feel free to share!

Hide Sites

Related posts:

1. Back to Basics: Unix System Stats Utilities
2. Back To Basics: Unix Differences in Performing Tasks
3. Working With Unix Variant Differences

First, CAM stands for Content Addressable Memory. A CAM is a special type of memory; some would say the opposite of RAM. With normal computer memory (RAM) the operating system provides an address, and receives the data stored at the supplied address. With a CAM, the operating system supplies the data, and the CAM returns a list of addresses where the data is stored, if it finds any. Furthermore, a CAM searches the entire memory in one operation, so it is considerably faster than RAM.

CAMs are very expensive, so they aren’t normally found in PCs. Even router vendors will sometimes skimp, opting to instead implement advanced software-based searching algorithms to plod through RAM. Most commonly, CAMs and TCAMs are found in network processing devices, including Intel IXP cards and various routers or switches. The most commonly implemented CAMs are called binary CAMs. They search only for ones and zeros; a simple operation. MAC address tables in switches commonly get stored inside binary CAMs. You can bet that any

A Renesas TCAM

switch capable of forwarding Ethernet frames at line-speed gigabit is using CAMs for lookups. If they were using RAM, the operating system would have to remember the address where everything is stored. With CAMs, the operating system can find what it needs in a single operation. In this case desired data is the switchport that data should be sent out, based on the given MAC address, i.e. the essence of a MAC table. Some older Cisco switches running CatOS even opted to call this table the cam table, thereby causing great confusion across the land. Bridge table, forwarding table, mac-address table, cam table; it’s all the same.

Finally, a TCAM is a Ternary CAM. This allows the operating system to match a third state, “X.” The X state is a mask, which means you don’t care what it is. This naturally lends itself to networking, since netmasks operate this way. To calculate a subnet address we mask the bits we don’t care about, and then apply the logical AND operation to the rest. Being able to do this in hardware is a great benefit for routers. Additionally, routers can store their entire routing table in these TCAMs, allowing for very quick lookups. A router with routing tables in TCAMs can find the next-hop destination in a single operation every time instead of trying to search through a tree (or other data structure) in RAM.

Hardware can sometimes seem magic, but it isn’t always transparent. When configuring routers most people will run into a situation where enabling a new feature will require that the Cisco SDM (Switching Database Manager) template be changed. This template is actually a method Cisco uses to assign specific applications to specific TCAM resources.

Some routers will allow you to manually specify how much TCAM space you want to allocate to a specific feature. Others aren’t so nice. They make you choose from a few restrictive templates, which allocate the resources automatically based on a few predetermined settings. For example, on the Cisco 3750, we recently wanted to enable policy-based routing (PBR) to implement a layer 3 jail. The basic idea with template-only routers is that you have to choose where you want most of the optimizations, and compromise on the rest.

For this platform, there are four templates: default, routing, PBR, and VLAN. Each of these tries to allow for a bit more resources allocated to the specified task. For policy routing, we’d have to choose “routing” or “PBR,” which in turn limits the amount of unicast MAC addresses that can be held in TCAMs. Likewise, selecting a VLAN template will make PBR impossible, but allow for more VLAN database information to be held in TCAMs. There are always compromises when we need to use more advanced features. Keeping true with the spirit of router operating systems, there’s also some mysterious side-effects when a new template is chosen. On our specific router, if the PBR template is chosen, the router will become unable to support VPN routing/forwarding tables (VRF). The next unsightly gotcha is that with the IOS version that supports IPv6, you cannot even enable PBR. There is no template to allow both policy routing and IPv6.

Perhaps the main idea of TCAM allocation still isn’t clear. Just because, for example, 8K is allocated to routing tables, this doesn’t mean that you can only have a routing table of that size. There’s always the fallback of process switching. Process switching means that everything will be done by the processor instead of in hardware (TCAMs). Processor intervention is not desirable, mostly because it is much slower than hardware lookups. Also, the processor is supposed to be used for things like sending logs to a syslog server and controlling SSH sessions. If a router doing process switching gets really busy, it may be unable to service your console access attempts. Generally speaking, the more expensive the router, the less it will use the processor.

Hardware is finite, and we always need more. More expensive routers don’t always suffer from the constant struggle for TCAMs because they have enough to support most features that currently exist. Unfortunately, most companies won’t want to purchase the latest and greatest router with seemingly endless hardware resources unless they can justify the added cost by showing a current need for them. So, most of us are stuck having to adjust TCAM allocations.

Further reading: an interesting blog from Plixer.

Feel free to share!

Hide Sites

Related posts:

1. Networking 101: Subnetting – Slice Up 32-bits
2. Are Cisco Flex Links the End of STP?
3. Networking 101: Layer 2, Link and Spanning Tree

Each VM is another server that an administrator must manage. Security updates must be applied and global configuration changes now need to be propagated to all these new machines. While it’s easy to create 3-4 (or more) servers on one physical piece of hardware, you’ll certainly struggle if you aren’t already set up to scale.

Unfettered Growth
The number of physical machines in a small company may drop dramatically; maybe 40%, when virtualization is implemented. Unfortunately, the number of OS instances will generally increase by two-fold or more at the same time. The power and cooling savings are realized, as was promised by virtualization, but taking 20 servers to 12 servers, for example, will means you may soon have 40 OS instances to manage.

Puppet, from Reductive Labs

The reasons for VM proliferation depend on your culture, but the most common reason is that delegating control of an entire OS is easier than managing an application for customers. IT customer, be they engineers, application developers, or smaller IT units within an organization, frequently need more access then cenral IT is willing to give. The easy solution: give them a server of their own. Test environments, too, are best served by virtual machines.

To keep hardware (and power and cooling) costs down, many companies implement policies about the implementation of new services. New applications and servers need to be run on VMs first, unless it’s really requires its own server. Policies such as these are good, in that they limit wastefulness, but they do tend to exacerbate VM sprawl.

Sprawl aside; it’s worth noting that higher utilization levels on your servers does not mean that they’ll use an appreciably larger amount of power. In fact, the power savings claims are really true, and can be even greater if your utilization is low and you use VirtualCenter’s power management features. VMWare can migrate VMs to fewer servers if utilization isn’t high enough, and actually power off unnecessary servers. This works best with Dell hardware, but other large vendors are supported as well. Imagine: all your VMs migrating to a few blades in a blade server during the nighttime, and then as utilization increases during the day, blades quickly boot up and take the load as needed. Granted, I don’t personally know any enterprise environments that are brave enough to try it yet, but in theory the concept is wonderful.

Dealing
Something magical happens when a company grows to around 50 operating systems. It’s too many to manage by simply logging in and running commands, so people start to write scripts. In Windows land, if it hasn’t already happened, you must implement Active Directory. For the Unix/Linux servers, configuration management becomes even more important. Writing a script that SSH’s to each server and runs a command doesn’t scale, no matter how hard people want it to. You need a real configuration management system (such as puppet or cfengine) to ensure that servers are configured exactly how you want, and that they will remain that way.

If you already operate in a large environment with good automated installations and configuration management systems, chances are scaling 100-fold won’t be a problem. Barring scaling issues with the management software itseld, that is. A good network-booting deployment system is only half the battle, because every server isn’t going to be configured identically. If you’re “doing it right,” you should be able to arbitrarily reinstall any server, walk away, and know that it’ll come back up patched and running all the services it’s supposed to. Servers, or rather the OS that runs on them, should be truly disposable.

Management of a “golden image” is promised by VMWare, probably because ITIL mentions it, but it doesn’t really help in practice. You have to create your images (somehow). There’s no mechanism to update a golden image with security patches and apply them to existing systems; you’ll generally have to reinstall the OS instances. And that’s what you should do periodically, but without some kind of configuration management system, you’ll also be manually installing and configuring the services that the VMs used to provide in order to restore service functionality.

VM growth, therefore, is no different from server growth. It may be easier and cheaper, but from the OS management viewpoint, you’re doing the same thing. Likewise, the availability of your services is also in danger. Running five VMs on a single piece of hardware means that a hardware failure takes out five servers instead of one. VMWare and Xen can both be clustered and run from shared storage, such that a hardware failure will result in the VMs immediately (instantly, even) being migrated to other servers. The problem is that VMotion requires the most expensive VMWare license, and a VirtualCenter server. Shared storage isn’t as big of an issues these days with iSCSI, but its still another aspect that must be configured. We’ll cover this issue in-depth in a future article, focusing on Xen and RHEL Clustering Services.

The point is: dealing with VM sprawl is no different than dealing with scaling up to support more physical servers. Use whatever mechanisms are available on your given platforms, and “do it right.” A VM is, and always will be, just another server.

Feel free to share!

Hide Sites

Related posts:

1. Understanding Linux Virtual Memory
2. How Much Server do you Need?
3. Zenoss: We Can Ditch Nagios Now

Network admins can use SDM to generate Cisco TAC approved configurations with the click of a few buttons. It’s not just limited to simple configurations either. Some tricky configuration tasks such as QoS and VPNs also become easier with the SDM because it ensure that configuration errors don’t exist. In short, you can deploy new devices and services much quicker by using the SDM.

As the name implies, SDM also intently focuses on security. A feature called “one-click lockdown” will set your router up as Cisco recommends—a good starting point for new routers. Also, the security audit function of the SDM will check your configuration and offer up a surprisingly large set of recommendations for hardening security. Many are things that most administrators don’t worry about, but with the SDM you can easily click “fix it” for each item after reading a description. There’s no reason to leave any possible vulnerability open when you have a quick, easy GUI manager pointing out what should change.

Cisco SDM user interface

The SDM is also a management console that gives you a real-time look at your device. It provides a nice interface for viewing system logs, firewall logs, and even real-time performance statistics. You probably already gather performance data via SNMP for historical charting, but being able to see the real-time information while you’re logged into the device manager, where you can also make changes to the configuration, is quite convenient.

SDM is available for most IOS-based routers running 12.2 and above. It is install by downloading a zip file from Cisco and copying it to the router’s flash memory. It’s then accessed from your Web browser (Firefox or IE required, as well as certain Java versions).

Making it Work

First, we must point out that using the SDM requires that you enable the HTTP server on your device. Yes, most Cisco security holes involve the Web server, and yes, a Web spider can easily DoS your router if it starts crawling Web pages and runs it out of RAM. Fortunately, both of these are negligible if you don’t allow access to the Web server from external networks. So first things first, enable: ip http secure-server, then configure ACLs to limit access properly.

After unzipping the file downloaded from Cisco, you can browse to: https://$server/flash/sdm.shtml

Then, login with a highly privileged account (level 15 is required). Up comes the Java applet, and you’re in! It couldn’t be easier than that.

Features

At the top, you’ll see things like Wizard, Advanced, and Monitor. The left had side lists things you can do in Wizard mode, and includes things such as VPN, Firewall, and LAN configuration options.

At the top you’ll also see a “deliver” button, which is another way of saying “commit.” All changes made within the SDM are committed to flash and merged into the running configuration when deliver is clicked.

Various configuration menus exist, most of which make the task at hand slightly easier. For the advanced administrator, it means you can just select options quickly without remembering the specific syntax. More junior admins can make previously confusing concepts work with little effort as well, and then look at the configuration that was generated.

The neatest feature is the security audit. When run, it will gather information about your device and then provide a list of problems. A nice “fix it” check box next to each item can be clicked, or you can elect to choose “fix all.” Beware that Cisco’s idea of security is basically very locked down. Selecting “fix all,” for example, will disable SNMP. It’s true that exposing SNMP to the external world is unwise, but you really do need it enabled for internal access.

You can also configure ACLs and interface parameters from within the GUI. Interfaces can be configured completely via the SDM, and the really nice part is that it lists all available setting for the particular interface. You’ll see check boxes for every option, along with a nice description of each option. ACLs can also be configured, and the GUI presents a nice view of which services will be allowed, and in which direction, on each interface.

In advanced mode, you can easily change many things, including OSPF and BGP settings. It’s just a matter of a few clicks to add another OSPF process ID or add another network to an existing one. Being able to see networks each OSPF process advertises and configure passive interfaces in a single well laid out window is very exciting.

In Monitor mode, you can see which interfaces are down, how much CPU is being utilized, and how much RAM is being taken up by which processes. Very useful information, sure to put a smile on your face the first time you see it.

The SDM does not support everything you’d want to do on a router, but the majority of common tasks are covered. It’s definitely a time-saver, learning tool, and convenience crutch all in one. Don’t feel bad using the SDM; convenience always outweighs prestige, assuming you can do it via the command line too. Enable the “show changes before delivering config” option to see what commands the SDM is about to run, and you’ll avoid surprises and possibly learn something at the same time.

Feel free to share!

Hide Sites

Related posts:

1. Built-in Security with Cisco IPS
2. Cisco AutoQoS: VoIP QoS for Mere Mortals
3. Are Cisco Flex Links the End of STP?

The concept of virtual memory is generally taught as though it’s only used for extending the amount of physical RAM in a system. Indeed, paging to disk is important, but virtual memory is used by nearly  every aspect of an operating system.

In addition swapping, virtual memory is used to manage all pages of memory, which incidentally are required for file caching, process isolation, and even network communication. Anything that queues data, you can be assured, traverses the virtual memory system. Depending on a server’s role, virtual memory functionality may not be optimal. An administrator can dramatically improve overall system performance by adjusting certain virtual memory manager settings.

To optimally configure your Virtual Memory Manager (VMM), it’s necessary to understand how it does its job. We’re using Linux for example’s sake, but the concepts apply across the board, though some slight architectural differences will exist between the Unixes.

Nearly every VMM interaction involves the MMU, or Memory Management Unit, excluding the disk subsystem. The MMU allows the operating system to access memory through virtual addresses by using data structures to track these translations. Its main job is to translate these virtual addresses into physical addresses, so that the right section of RAM is accessed.

The Zoned Buddy Allocator interacts directly with the MMU, providing valid pages when the kernel asks for them. It also manages lists of pages and keeps track of different categories of memory addresses.

The Slab Allocator is another layer in front of the Buddy Allocator, and provides the ability to create cache of memory objects in memory. On x86 hardware, pages of memory must be allocated in 4KB blocks, but the Slab Allocator allows the kernel to store objects that are differently sized, and will manage and allocate real pages appropriately.

Finally, a few kernel tasks run to manage specific aspects of the VMM. The bdflush manages block device pages (disk IO), and kswapd handles swapping pages to disk. Pages of memory are either Free (available to allocate), Active (in use), or Inactive. Inactive pages of memory are either dirty or clean, depending on if it has been selected for removal yet or not. An inactive dirty page is no longer in use, but is not yet available for re-use. The operating system must scan for dirty pages, and decide to deallocate them. After they have been guaranteed sync’d to disk, an inactive page my be “clean,” or ready for re-use.

Tuning
Tunable parameters may be adjusted in real-time via the proc fils system, but to persist across a reboot, /etc/sysctl.conf is the preferred method. Parameters can be entered in real-time via the sysctl command, and then recorded  in the configuration file for reboot persistence.

You can adjust everything from the interval pages are scanned to the amount of memory to reserve for pagecache use. Let’s see a few examples.

Often we’ll want to optimize a system for IO performance. A busy database server, for example, is generally only going to run the database, and it doesn’t matter if the user experience is good or not. If the system doesn’t require much memory for user applications decreasing the available bdflush tunables is beneficial. The specific parameters being adjusted are just too lengthy to explain here, but definitely look into them if you wish to adjust the values further. They are fully explained in vm.txt, usually located at: /usr/src/linux/Documenation/sysctl/vm.txt.

In general, a IO-heavy server will benefit from the following setting these values in sysctl.conf:
vm.bdflush=”100 5000 640 2560 150 30000 5000 1884 2”


The pagecache values control how much memory is used for pagecache. The amount of pagecache allowed translates directly to how many programs and open files can be held in memory.

The three tunable parameters with pagecache are:

• Min: the minimum amount of memory reserved for pagecache
• Borrow: the percentage of pages used in the process of reclaiming pages
• Max: percentage at which kswapd will only page pagecache pages; once it falls below, it can swap out process pages again

On a file server, we’d want to increase the amount of pagecache available, so that data isn’t moved to disk as often. Using vm.pagecache=”10 50 100″ provides more caching, allowing larger and less frequent disk writes for file IO intensive work loads.

On a single-user machine, say your workstation, large number will keep pages in memory, allowing programs to execute faster. Once the upper limit is reached, however, you will start swapping constantly.

Conversely, a server with many users that frequently executes many different programs will not want high amounts of pagecache. The pagecache can easily eat up available memory if it’s too large, so something like vm.pagecache=”10 20 30” is a good compromise.

Finally, the swappiness and vm.overcommit parameters are also very powerful. The overcommit number can be used to allow more memory allocation than RAM exists, which allows you to overcommit the amount of pages. Programs that have a habit of trying to allocate many gigabytes of memory are a hassle, and frequently they don’t use nearly that much memory. Upping the overcommit factor will allow these allocations to happen, but if the application really does use all the RAM, you’ll be swapping like crazy in no time (or worse: running out of swap).

The swappiness concept is heavily debated. If you want to decrease the amount of swapping done by the system, just echo a small number of the range 0-100 into: /proc/sys/vm/swappiness. You don’t generally want to play with this, as it its more mysterious and non-deterministic than the advanced parameters described above. In general, you want applications to swap to avoid them using memory for no reason. Task-specific servers, where you know the amount of RAM and the application requirements, are best suited for swappiness tuning (using a low number to decrease swapping).

These parameters all require a bit of testing, but in the end, you can dramatically increase the performance of many types of servers. The common case of disappointing disk performance stands to gain the most: give the settings a try before going out and buying a faster disk array.

Feel free to share!

Hide Sites

Related posts:

1. Multi-user Security in Linux
2. Managing Virtual Machine and Cloud Sprawl
3. Understanding Fibre Channel

Actively preventing the attack makes it an IPS. The standard old IDS solution means that it can detect and alert, but blocking attacks is not normally part of an IDS’s feature set. Thus, if you want to prevent attacks rather than just receive alerts, you need an IPS. Cisco’s IPS works like any other: you get a signature file, called the Signature Definition File (SDF) by Cisco, and if the IPS finds that a packet matches a signature, it’s blocked.

There are appliances, Catalyst switch modules, and router modules, but IPS is also built-in to certain IOS images now. Since Cisco claims IPS features won’t impact router performance (since the latest release), it may be possible to skip the purchase of a dedicated module for IPS.

The catch, of course, is that an IPS is not robust without constant signature updates. Attacks are constantly evolving, and without an update you aren’t protected against the latest and greatest attacks. Something completely new could sneak in, but the idea is that after the first few attacks Cisco will update the SDF and you’ll be notified that it’s time to download a new version. That’s right, you have to manually download and install a new signature file. This requires a subscription service above and beyond what you pay for SMARTnet. Services for IPS, as it’s called, provides SDF updates and the other features (support, warranty) that SMARTnet does as well. Accordingly, your SMARTnet contract is discounted when you purchase a Cisco Services for IPS contract, according to Cisco’s Q&A documentation.

Configuring IPS for Sensor Modules

There are many different cases for configuring IPS depending on your device. First, we’ll show you how to enable it on any IPS sensor module that uses the IPS 5.1 or later, then we’ll show you how to take advantage of the IOS built-in default IPS features.

The IDS Device Manager (IDM) is a graphical interface for configuring all IDS (and IPS) functionality. If you prefer that, then refer to the Cisco documentation after reading about how it’s done via the CLI here.

The general idea we’re working with here is called the VLAN pair method. This means that we’ll configure two VLANs in a pair group, and all traffic received by a sensor will be inspected and either forwarded on to the other VLAN, or dropped. Up to 255 VLAN pairs can be configured on most sensors.

First we enter configuration mode, then the service interface, and finally select the physical interface that we wish to configure:

sensor#configure terminal

sensor(config)#service interface

sensor(config-int)#physical-interfaces GigabitEthernet0/1

Next, we must configure the VLAN pair (and give it a meaningful description):

sensor(config-int-phy)#subinterface-type inline-vlan-pair

sensor(config-int-phy-inl)#subinterface 1

sensor(config-int-phy-inl-sub)#vlan1 10

sensor(config-int-phy-inl-sub)#description vlans 10 and 11

Conceptually, the interface will now be added to a virtual sensor, and once it’s enabled it will monitor traffic. We now need to enable a virtual sensor:

sensor(config)#service analysis-engine

sensor(config-ana)#virtual-sensor vs0

Once that’s completed, we simply add the previously-defined subinterface to the sensor, and we’re done:

sensor(config-ana-vir)#physical-interface GigabitEthernet0/2 subinterface-number 1

Configuring IPS for IOS

You can enable IPS features in IOS using the default SDF. Signatures may be added manually to the SDF, or you can pay Cisco for the latest signatures.

First we need to enable what’s called Security Device Event Exchange notifications:

router(config)#ip ips notify sdee

Then we must configure an IPS rule name that will be used for associating with interfaces.

router(config)# ip ips name MYIPSRULES

The next step is to specify where the SDF file will come from. The following command specifies that the file 256MB.sdf can be found in flash memory. You can also specify tftp or any other protocol your Cisco knows how to handle, but it’s best to use flash memory to ensure no dependencies on other servers.

router(config)# ip ips sdf location flash:256MB.sdf

Finally, we simply enable IPS on the interface (in both directions). It is also a good idea to enable IP reassembly on the interface, so that the IPS rule can evaluate entire IP packets at once.

router(config)#interface fastEthernet 0

router(config-if)#ip ips MYIPSRULES in

router(config-if)#ip ips MYIPSRULES out

router(config-if)#ip virtual-reassembly

Now you have a working IPS, based on the file in your flash called 256MB.sdf. That file must be downloaded from Cisco using your CCO login linked to a valid support contract.

The Power of Community

If you don’t feel like paying Cisco for signature updates, you can update the SDF yourself. When a new attack surfaces, you’ll often find Cisco IPS XML signatures posted to various online forums. You can and should use them.

To view your current SDF version, you can run: sh ip ips signatures

To merge the IPS SDF configuration with new information, you can copy in an XML file. Just like copying in any configuration snippet, the updates will be merged, not replaced. Say we got sigs.xml from a helpful network operator. To enable these signatures, we simply run:

router#copy tftp://serer.fqdn/sigs.xml ips-sdf

That’s it! You’ll see that 256MB.sdf on the flash memory is now a bit larger. It’s a good idea (and is recommended by Cisco) to rename 256MB.sdf to avoid confusion, now that you are no longer running a Cisco-sanctioned version.

Enabling IPS on supported routers is quite easy, but can lead to some interesting troubleshooting sessions. Be sure you have a syslog server that your routers all log to: it will save hours of work. Also, search around; you may find a source for XML updates that you wish to trust, and then it’s pretty easy to automate daily merges into your local SDF.

Feel free to share!

Hide Sites

Related posts:

1. Manage Devices and Configurations with Cisco SDM
2. Are Cisco Flex Links the End of STP?
3. Cisco AutoQoS: VoIP QoS for Mere Mortals