A Blog for Those With a Big Appetite for IT Knowledge...

Setting Up SSH Access with Radius Authentication on 3Com Switches (5500 and 4500 Family) using Microsoft Network Policy Services (NPS)

2012-01-23T11:13:00.000-07:00

I have been dealing with this for many days and I finally got it working ! I believe there are a lot of people out there that are also willing to implement something like this , but there is no much 3Com documentation out there to help themselves. Follow this tutorial step by step in order to start authenticating ssh access to 3Com switches (5500 Family and 4500 Family) using your active directory and the windows server 2008 Network Policy Server. I also think this procedure can be improved and I will keep working on it (check for updates later). I would also like to use the same authentication mechanism for the 3Com web interface. A very similar configuration can also be used to set up 802.1x port authentication on this switches (post coming soon ;) ). All comments are welcome at the bottom of the post ! Enjoy !

Windows Server Side Configuration

1- Install NPS Role on Windows Server 2008

2- Define the RADIUS clients

3- Modify Connection Request Policies using the Network Policy Server configuration Snap-in

5- Modify Standard Radius Attributes Dictionary on Windows Server 2008 by editing C:\Windows\System32\ias\dnary.xml to add SSH, FTP and Terminal under the Login-Service Attribute. The Standard RADIUS only allow for Telnet login by default, therefore the dictionary modification is needed in order to add, SSH, FTP and Terminal as login options under the standard RADIUS implementation.

6- Restart the Windows Server (Sometimes a Service Restart is not enough and a Server Restart might be required for the SSH option to show up among the standard RADIUS options)

7- Modify the Standard Radius Attribute inside the corresponding Network Policy to allow clients to log in using SSH.

8- Modify the Vendor Specific Attribute to provide the corresponding User Access Level. Once the client connecting to the switch through SSH have been authenticate, the RADIUS server needs to tell the switch what access level is this user allowed. 3Com switches support the following access levels:

- Level 0 (Attribute Number 1, Attribute Value 0) - 3Com-Visit
- Level 1 (Attribute Number 1, Attribute Value 1) - 3Com-Monitor
- Level 2 (Attribute Number 1, Attribute Value 2) - 3Com-Manager
- Level 3 (Attribute Number 1, Attribute Value 3) - 3Com-Administrator

In this case we are going to provide the users of the WEM-Local-NetAdmin domain group with User Access Level 3 (Administrator), so the user will be able to run any commands on the switches.

3Com Switch Side Configuration

1- Set up a Radius Scheme on 3Com

radius scheme domain_name
server-type extended
primary authentication IP_Address_Of_NPS_Server
primary accounting IP_Address_Of_NPS_Server
accounting optional
key authentication systems2006
key accounting systems2006
timer realtime-accounting 15
timer response-timeout 5
retry 5
user-name-format with-domain
nas-ip IP_Address_Of_Switch
calling-station-id mode mode2 uppercase

2- Set up a Domain and link it to the corresponding Radius Scheme

domain domain.local
scheme radius-scheme domain_name local
scheme lan-access radius-scheme domain_name
scheme login local
accounting lan-access radius-scheme domain_name
authentication login radius-scheme domain_name local
accounting login radius-scheme domain_name local
access-limit enable 60
idle-cut enable 20 2000

3- Set up the VTY interfaces to allow for ssh connections and to use the Radius Authentication Scheme

user-interface vty 0 4
authentication-mode scheme

4- Set up the SSH Server parameters

ssh-server source-ip ip_address
ssh server authentication-retries 5

5- Set up the SSH user that are allowed to log in using SSH. Ideally, I would like to specify a group, instead of a domain user (userid@wem.local), so that all users on that group are valid users for the 3Com switch, but I have not found the way to do that yet.

ssh user admin authentication-type password
ssh user admin service-type all
ssh user userid@domain.local authentication-type password
ssh user userid@domain.local service-type all

Troubleshooting

1- Check the Windows Server Event Viewer for Security , under Windows Logs. Verify the events are being logged as Audit Success.

2- Use Wireshark on the NPS Server and try to connect to the switch using SSH. Verify Radius parameters passed back from the NPS server to the Switch.

3- Use NTRadPing Test Utility freeware tool to test connectivity to the RADIUS server and check your parameters.

Recommendations

This procedure should be improved so that there is no need to define each user using the ssh user statement. Ideally only the AD user group should be used.

Implement the same authentication mecanism for the switch management web interface.

Comments are welcome !

Software RAID on Linux with mdadm

2011-06-24T16:36:00.000-06:00

RAID Controllers

We’ve spent a great deal of time examining various RAID levels such as RAID-0, 1, 5, and 6, and Nested RAID levels such as RAID-10, 50, 51, 61, and 60 or even the more complicated RAID-100 or RAID-160. In all of these discussions we have assumed there was a RAID “controller” that performed the various RAID operations. This includes sending chunks of data to the appropriate disks, computing parity, hot-swapping, disk fail-over, checking read transactions to determine if the read was successfully and if not, declaring that disk as “down”, plus other important tasks related to RAID. All of these tasks require some sort of computation and have to be performed by a RAID controller.

You really have two options for RAID controllers: (1) a dedicated RAID controller such as those on add-in RAID cards, (2) software RAID that uses the CPU for RAID chores. In the first case you have a dedicated RAID controller, typically on an add-in RAID card but it can be on a motherboard, that performs the necessary RAID computations. This controller typically uses a dedicated lower power processor, often a real-time processor such as a PowerPC, to perform the computations. Typically these controllers are put on an add-in card and you plug the drives you want in the RAID array into the card.

With software RAID, all the RAID functions run on the CPU. Pretty much all Linux distributions comes with software RAID in the form of md (Multiple Device) which is a driver within Linux that provides virtual devices created from one or more underlying devices (e.g. storage block devices). Plus the great thing is that it comes with almost all Linux distributions so you have access to the source (plus the price isn’t too bad either and it has a great deal of functionality). In addition, you can create Nested RAID configurations as needed since all RAID functions are in software (I smell a Triple Lindy coming).

One thing you have to be very careful about is what is commonly called a fakeRAID card or fakeRAID controller. FakeRAID is not hardware RAID because there is no dedicated RAID controller. Rather, they use a standard disk controller chip on an add-in card or motherboard, with some specialized firmware and drivers. At boot-time they run a special application that allows users to configure disks attached to the fakeRAID as a RAID group. But the RAID processing is really handled by the drivers which are run on the CPUs (so the CPU provides the computational power). Consequently, it’s really a software RAID solution and not hardware RAID (hence the name “fakeRAID”).

There is a great deal of discussion and grinding of teeth within the Linux community about fakeRAID. One point of the discussions is that the vendors of fakeRAID don’t tell customers that what they are actually buying is not a RAID card with a dedicated RAID controller, but rather a simple card with a disk controller coupled with drivers that use the CPU for RAID processing (false advertising). Plus there is the additional problem of developing and supporting drivers for Linux to allow these fakeRAID cards to be used. Moreover, there is a strong argument that it is probably better to use software RAID that comes with Linux (md) since it is already part of Linux and can arguably give you better performance.

However, if you want to use software RAID that comes with Linux in the kernel (md), you still need some tools to control/manage/monitor the software RAID arrays. That’s where mdadm comes in. This article will do a brief examination of mdadm and some of its options.

Introduction to mdadm

Mdadm is a software tool primarily written by Neil Brown that allows you to create, assemble, report on, grow, and monitor RAID arrays that use software RAID in Linux. Actually according to the documentation there are seven modes of operation:

   1. Create
   2. Assemble
   3. Follow or Monitor
   4. Build
   5. Grow
   6. Manage
   7. Misc

We’ll walk through these various modes of operation but the focus of this article is an introduction and not an in-depth HOWTO. You can find those types of articles on the web.

The first step in using mdadm, or any RAID for that matter, is to PLAN your RAID configuration carefully. Personally I like to work backwards starting with the purpose of the storage. Will it be used for a database? Will it be used for /home? Will it be used for high-speed scratch? Will it be used for data that requires a very high degree of reliability? Understanding the intent of the storage is really the key to creating the RAID configuration you need/want. Once you determine the storage use case, you need to develop an idea of how much I/O performance you need (throughput and IOPS) and the general ratio of read and write performance. You should also develop an idea of how much data redundancy you need for the storage.

Once you have an idea of the performance and redundancy of the array you can select the RAID configuration you think you might need. I would suggest you select a few candidate RAID configurations and then do some more reading/research on each one and select the RAID configuration that seems to be best. During this research be sure to examine the redundancy as well as the performance of the various RAID configurations and compare them to your estimations. But also be sure to examine the capacity and storage efficiency of each level. You may love the performance and storage efficiency of RAID-10 but the data redundancy may not be enough for you. Or you may love the data redundancy of RAID-61 but you may not willing to give up the performance or, perhaps more importantly, you may not be willing to have such low storage efficiency (especially if this is for your home system).

But just choosing the RAID configuration you want is not the end of your planning. You need to also consider a number of other things. Perhaps the most important thing you need to consider is if you will need to grow/shrink the storage. The reason this is important is because you are likely to have to use LVM (Logical Volume Manager) either on top of Linux software RAID or underneath it. This forces you to carefully consider how to build both LVM and software RAID and how you expand either one or both. I would recommend walking through the expansion steps to make sure you understand how to do it (you could even pass along your ideas to someone else to have another pair of eyes examine them).

One other thing you should consider before implementing your well formulated and thought-out RAID plan is the file system that sits on top of the storage. Based on your usage model for the storage, select one or two candidate file systems. Then do some research on each one to find out what problems or limitations exist, and also how you can optimize each file system for better performance (we’re all performance junkies at heart). There are a number of articles on the web that discuss tuning file systems with Linux software RAID.

Assuming that you have done your careful planning (including a backup solution), let’s move on to the first “mode” of mdadm, Creating a RAID array.

Creating a RAID array

Mdadm allows you to create a RAID array using Linux block devices. During the creation of the array, per-device superblocks are created for the RAID array (allows for the array to be assembled correctly). Using the “create” mode is the most common method for building the array and is recommended if you are just starting to use mdadm.

The basic mdadm command for creating a RAID array is fairly simple with the following generic command and typical options.

mdadm --create [md-device] --chunk=X --level=Y --raid-devices=Z [devices]

where the options are as follows:

      -c, –chunk= Specify chunk size in kibibytes. The default is 64.

      -l, –level= Set raid level, options are: linear, raid0, 0, stripe, raid1, 1, mirror, raid4, 4, raid5, 5, raid6, 6, raid10, 10, multipath, mp, faulty

      -n, –raid-devices= Specify the number of active devices in the array.

Notice that the mdadm command begins with the “–create” option that tells mdadm that it will operate in “create” mode. I like to then define the specific md-device as well but be sure you aren’t using an existing md-device name. Specifying the “chunk” option is up to you. Then you define the RAID level you want by the “–level” option with the options listed above. You can also tell mdadm how many block devices you are using with the “–raid-devices=Z” option (Z is the number of devices). Then finally you give mdadm the list of block devices you are using.

An example of using the create option with mdadm is,

% mdadm --create --verbose /dev/md0 --level=0 --raid-devices=3 /dev/sda1 /dev/sdb1 /dev/sdc1

which creates a RAID-0 configuration that is labeled as /dev/md0 and uses three block devices that are /dev/sda1, /dev/sdb1, and /dev/sdc1.

In the example, I have used the first partition of each of the three drives that are valid block devices as the block devices for mdadm. They could have easily been the entire disk such as /dev/sda or /dev/sdb. The point is that they need to be valid Linux block devices (they could even be network based devices but that’s a different discussion).

Mdadm is smart enough to build the RAID-0 configuration using the smallest common size of each of the three devices. So it is recommended that you check on the size of each block device using fdisk as shown below.

root@amadys-laptop:~/# /sbin/fdisk /dev/sdb

The number of cylinders for this disk is set to 19457.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sdb: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000bca3e

   Device Boot      Start         End      Blocks   Id System
/dev/sdb1   *           1       18704   150239848+ 83 Linux
/dev/sdb2           18705       19457     6048472+   5 Extended
/dev/sdb5           18705       19457     6048441   82 Linux swap / Solaris

Be sure to look at the column labeled “Blocks” for the particular partition you are going to use. Do this for all the devices and make sure the number of blocks is the same or very close (otherwise you are wasting space).

You might also notice that I used the “–verbose” option in the mdadm create command. I like to use this option to get more information about what mdadm is doing (“better informed than sorry” is a good motto). This is always a good habit to develop.

At this point, your RAID array should be created and running. An easy way to check this is to look at /proc/mdstat.

% cat /proc/mdstat

Fortunately, the output should be fairly easy to read at first glance. For much more detailed information you can read this article.

Immediately after the RAID array is created it may have to go through a synchronization process. This process performs the necessary RAID functions for configuration you created. For example, for RAID-1, the blocks on the first drive are copied to the second drive even if there isn’t any information on the blocks.

Once the array has finished synchronizing and is ready, then you can move to the next step which can be using the resulting RAID array device in LVM or creating a file system using the block device.

There are many options that you can use in “create” mode. You can read the man pages to get a list of them but below are some of the more important ones that haven’t been presented yet in this article.

      -x, –spare-devices= This option allows you to specify spare devices in the initial array. These are devices (disks) that are used in the event that a disk in the RAID configuration fails. Mdadm then uses the spare drive immediately and starts restoring the array to the desired configuration. Mdadm also allows spare drives to be added and removed later (they don’t have to be added when the array is created). If you use spare devices be sure that the “–raid-devices” option lists the number of devices to be the actual RAID drives plus the spares.

      -p, –layout=, –parity= Mdadm gives you remarkable control over your RAID configuration. This option lets you control the fine details of the data layout for RAID-5 and RAID-10 arrays and also controls the failure mode for a faulty or failed disk. Please read the manpages for more detail.

      -z, –size= This option is the amount of space to be used from each drive in a RAID-1, RAID-4, RAID-5, or RAID-6 configuration. The size is given in kibibytes and must be a multiple of the chunk size. In addition, you must leave about 128KB (128 kibibytes) of space at the end of the drive for the RAID superblock. After the array is created you can use the “grow” mode of mdadm (–grow) to increase the size of the RAID configuration.

Assembling an mdadm RAID array

One of the other “modes” in mdadm is “assemble”. After your RAID array has been created using mdadm, you can stop the array using the following command:

% mdadm --stop /dev/md0

which stops the RAID array /dev/md0 (be sure to unmount the file system that uses the RAID array first). However, there are problems in restarting the RAID array. When you restart the array you have to use mdadm to reassemble the array. For example,

% mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1

This command assembles the parts of a previously created array into an active array and “restarts” the array (i.e. make it function). To automate this, you could put this command as part of the system startup (for example, /etc/rc.d/rc.local) and you could create a simple script for stopping and starting the array. But mdadm can do some of the leg work for you with the following command:

% mdadm --assemble --scan

These options allow mdadm to scan the drives and reassemble the RAID array (it looks for the RAID superblocks on the drives). Typically this is done during the init phase of the system starting. For example, on my CentOS 5.5 system, there is a line in /etc/rc.d/rc.sysinit that looks like the following.

/sbin/mdadm -A -s

which will scan (“-s”) the drives and assemble (“-A”) the mdadm arrays.

However, you can get into trouble with scanning and assembling mdadm RAID arrays when you have more than one array. What is recommended is that if you want to restart an array by hand you specify the uuid for the array (a unique “name” of the RAID array). For example,

% mdadm --scan --assemble --uuid=7121b438:7d36f9f6:8aa9c8b3:b5b0d211

Since the uuid’s are unique to each array, this will ensure that mdadm can reassemble the array properly. However, mdadm‘s scanning and assembling capabilities are quite good and I routinely run two mdadm RAID configurations on my desktop and I’ve never had any confusion between which disks belong to which array (thanks to the superblocks on the devices).

Monitoring/Following an mdadm RAID array

The third mode of operation of mdadm is monitoring or following an mdadm RAID array. This mode monitors one or more arrays and allows action to be taken if the state of the array changes. According to the manpage for mdadm, this mode of operation is really only useful for RAID-1, 4, 5, 6, and 10, or multipath arrays since they have interesting states. On the other hand, RAID-0, and linear RAID are not useful because missing, spare, or failed drives cause these RAID modes to fail (i.e. non-operational).

The basic option for following or monitoring mdadm controlled arrays is the following:

mdadm --monitor options... devices...

(Note: you can use “-F” or “–follow” in place of “–monitor”). There are several options that can be used for monitoring and following mdadm arrays as listed below:

      -m, –mail This options allows you to define an email address where mdadm alerts are sent.

      -p, –program, –alert This option allows mdadm to run a “program” whenever an event is detected (it is recommended to use the full path to the program).

      -y, –syslog This option causes all events to be reported through “syslog”.

      -d, –delay This options creates a delay (in seconds) from when mdadm polls the arrays to when it next polls the arrays (i.e. the interval between polling).

      -f –daemonize This options tells mdadm to run as a background daemon if it is monitoring arrays. This causes mdadm to fork and run in the child process and disconnect from the terminal.

      -i, –pid-file This option tells mdadm to write the pid of the daemon process when mdadm is run as a daemon (see previous option). The pid is written to a specified file.

      -1, –oneshot This option checks the arrays only once and generates “NewArray” events as well as “DegradeArray” and “SparesMissing” events (this will show up in the logs). According to the manpages, if you run the command “mdadm –monitor –scan -1″ from a cron job, it will ensure regular notification of any degraded arrays (always a good thing).

      -t, –test This option generates a “TestMessage” alert for every array found at startup. This alert gets mailed and passed to the alert program (if you have defined one). This is very useful for testing that alert messages get through successfully (i.e. they work).

As you can see, mdadm gives you several pretty good options for monitoring your arrays. The exact practice of how you monitor really depends upon how you want to function and any tools or processes you have developed. There are several articles on the web that show how to monitor your mdadm array.

In the interest of helping interested people get started I will present a few tips that you can use to get started with monitoring mdadm arrays. I first recommend configuring mdadm to email you in the event of a change in state of the array. There are several articles that discuss how to do this. For example, this blog shows you how to use mdadm to send email to you in the event of a problem.

If you want to be more involved in the monitoring of your arrays there are two primary ways to get more detail: (1) cat /proc/mdstat, and (2) mdadm --detail [device]. The first option gives you a quick overview of the status of the array(s). You can parse this output (perhaps with perl or python), and create a special log or send the output to “syslog” to allow processing by syslog tools. Alternatively, you could create a simple monitoring metric that can be used in conjunction with ganglia or something similar.

The second option gives you more detail than mdstat but again, you can parse this output and then perform some action (logs, syslog, ganglia, etc.). But the details are really up to you since you likely have specific processes or techniques for monitoring.

One last option is to use Munin which is a monitoring tool (somewhat similar to Ganglia). It has plugins that allow you to monitor your mdadm created RAID arrays.

Building an mdadm RAID array

Another mode of operation in mdadm is building an array but without having per-device superblocks. This means that you cannot have mdadm scan and assemble the devices into an array. Consequently, you have to be careful that you differentiate between the initial creation of the array and the assembly of the array (or you can lose data). In addition, if you built an array using build any checks that might have been done between devices are not performed. Basically, you have to be very careful when using this mode of operation and know exactly what you are doing.

I really don’t recommend using this mode of operation so I won’t be discussing it. If you are interested in using the “build” then you can read the manpage or you can search around the Internet for references.

Growing an mdadm RAID array

One of the secret weapons you get with mdadm is the ability to grow, reshape, or even change RAID levels with md arrays. There are some limitations for growing and/or reshaping arrays however, just having that ability is a pretty major accomplishment for mdadm.

The basic option for growing or reshaping md arrays is “-G” or “–grow”. There are a number of options that can be used and things can become complex fairly quickly so I won’t go over them in this introductory article. Please read the manpages for the options or you can Google for information about growing md arrays. However, let’s take a high-level look at what the “–grow” option can do.

We can use the “–grow” option to add a third disk to a two-disk RAID-1 configuration or add another disk to our existing RAID-5 configuration. But we can also use the “grow” option to change RAID levels. For example, we can convert a two-disk RAID-1 md array to a two-disk RAID-5 md array. Direct from the mdadm author’s blog is the list of what changes (reshape) can be done:

    * A RAID-1 array can change the number of devices or change the size of individual devices. A 2 drive RAID-1 can be converted to a 2 drive RAID-5.
    * A RAID-4 can change the number of devices or the size of individual devices. It cannot be converted to RAID-5 yet (though that should be trivial to implement).
    * A RAID-5 can change the number of devices, the size of the individual devices, the chunk size and the layout. A 2 drive RAID-5 can be converted to RAID-1, and a 3 or more drive RAID-5 can be converted to RAID-6.
    * A RAID-6 can change the number of devices, the size of the individual devices, the chunk size and the layout. And RAID-6 can be converted to RAID-5 by first changing the layout to be similar to RAID-5, then changing the level.
    * A LINEAR array can have a device added to it which will simply increase its size.
    * RAID-10 and RAID-0: These arrays cannot be reshaped at all at present.

So you can see that there is a great deal of flexibility in mdadm with respect to changing shape, RAID levels, adding devices, removing devices, etc., to md arrays.

As I mentioned previously, the details of reshaping and changing RAID levels can get complicated. I suggest that before you use this option, you read all the literature you can. Then I would ask some experts who can tell you if your commands are correct or not. And before starting anything, I would definitely make sure you have a backup copy of the data and make sure the storage array isn’t in production.

Managing a md RAID array

Managing a md RAID array primarily consists of managing the devices within the arrays. This can include adding, removing, or failing disks within an array.

An easy (sort of) way to tell if you are using the manage mode, is that if you give a device before any options on the mdadm command line, or if the first option is “–add”, “–fail”, or “–remove”, then you are using the manage mode of mdadm. The general form of the mdadm command is,

mdadm device options ... devices ...

The options used in manage mode are:

      -a, –add [device] This option allows you to add the specified devices to the specified array while the array is running.

      –re-add [device] This options allows you to re-add a device to an array that was recently removed from the array.

      -r, –remove [device] This option allows you to remove the specified device. But the device must not be active so it must be either failed or a spare device.

      -f, –fail [device] This option marks the specified device as faulty (failed).

      –set-faulty This option is the same as -f.

You can combine the options in one command but all of the commands must affect the same array.

A simple example of the manage mode of mdadm is,

% /sbin/mdadm /dev/md1 --add /dev/sdc1 --fail /dev/sdb1 --remove /dev/sdb1

In this example, the array, /dev/md1 is the “target” of the mdadm command. The first option, “–add /dev/sdc1″ adds a device (/dev/sdc1) to the array. Then the option “–fail /dev/sdb1″ fails that device telling the array that the device is faulty. Finally, the third option, “–remove /dev/sdb1″ removes the device from the array at which point it can be removed from the system. Note that you need to first fail the device you intend to remove.

One cool feature of mdadm is that if you remove a device (disk) from an array you can add it back (–re-add) to the array and mdadm will only update the changed blocks from when the disk was removed. This is the default behavior if you used superblocks in the creation of the array (i.e. you didn’t use the “build” mode).

Misc

This last mode of operation is sort of a catch-all for options and commands that don’t fit into the other six modes. In general this mode supports some operations on active arrays, operations on component devices, and the gathering of information about the arrays.

The general form of the mdadm command in “misc” mode is the following.

mdadm options ... devices ...

Notice that for a “misc” command, no array was defined before the options. The options for this mode are the following:

      -Q, –query This option examines a device to see if it is an md device and if it is a component of an md array. The information discovered by mdadm is presented in the output.

      -D, –detail [md-device] This option prints out details of one or more md arrays. If you add the options “–brief” or “–scan” the amount of detail in the output is reduced but the format is amenable for /etc/mdadm.conf (an optional configuration file for mdadm).

      -E, –examine This option prints the content of the md superblock on the device (or all devices if no device is specified). As with the “–detail” option, if you use “–brief” or “–scan” the amount of output is reduced and it is more amenable to /etc/mdadm.conf.

      -X, –examine-bitmap This option reports the information about a bitmap file.

      -R, –run This option will start (activate) a partially built md array.

      -S, –stop This option stops (deactivates) an active md array.

      -o, –readonly This option marks the active array as read-only if it is not being currently used.

      -w, –readwrite This option marks the array as readwrite.

      –zero-superblock This option over-writes a valid md superblock in an array. This is useful when using disks (devices) from an old md array are used in a new one.

      -t, –test This option, when used with the –detail option, sets the exit status of mdadm to reflect the status of md device. This can be very useful when scripting monitoring tools. It is also useful if you want start the md array yourself rather than rely on the kernel to autostart it.

Of all of the “misc mode” operations the “–query”, “–detail”, and “–examine” options are the most generally used.

Summary

We’ve spent some time talking about RAID configurations, both single-level and Nested RAID configurations. In all of the discussions we’ve just mentioned that the RAID operations are handled by a “RAID controller”. There are two types of RAID controllers – hardware and software. The hardware RAID controller has a dedicated processor on an add-in card that handles all the RAID computations. In contrast, software RAID uses the system CPU for RAID computations (this includes “fakeRAID” controllers as well).

This article is an introduction to mdadm, the management/admin tool for Linux software RAID that comes with virtually every Linux distribution. The tool is very flexible allowing for the standard RAID levels and Nested RAID configurations, including some specialized RAID-10 configurations we’ve discussed previously. You can even use it to build some “Triple Lindy” Nested-RAID configurations if you want.

Mdadm has seven different “modes” of operation which we discussed. These modes allow you to create and start a RAID array, assemble a RAID array (useful when the system boots), follow or monitor a RAID array, build a RAID array (basically doing everything by hand – not recommended), grow a RAID array (one of the secret weapons of mdadm), manage a RAID array, and a “miscellaneous” category for functions that you may need that didn’t fall into the other categories.

There are some really great features in mdadm that can easily be glossed over in a mad rush to build a RAID configuration. There are two big ones that I want to highlight. The first features is the set of standard monitoring tools in mdadm that give you a great starting place for watching the status of your md array including the ability to send out email alerts. Plus you can write fairly simple scripts to parse array status information which can be used in monitoring tools such as ganglia or munin.

The second feature, which is probably the most significant, is the ability to grow and reshape md arrays. This allows you to add devices to an existing RAID array and grow the array to include the added space. However, the really cool feature is that you can use mdadm to change RAID levels without losing data. For example, you could convert a two-disk RAID-1 into a two-disk RAID-5 configuration. Then you could add disks to grow the RAID-5 configuration. Then you could convert the RAID-5 into a RAID-6 configuration. While I haven’t used this feature, this is pretty nifty if you ask me.

Mdadm is a great tool for Linux that is easy to use and gives you a great deal of control over RAID arrays. If you are thinking about RAID on Linux, be sure to take a look at mdadm.

Learning Awk by example: Part 3

2011-02-17T13:40:00.001-07:00

Formatting output

While awk's print statement does do the job most of the time, sometimes more is needed. For those times, awk offers two good old friends called printf() and sprintf(). Yes, these functions, like so many other awk parts, are identical to their C counterparts. printf() will print a formatted string to stdout, while sprintf() returns a formatted string that can be assigned to a variable. If you're not familiar with printf() and sprintf(), an introductory C text will quickly get you up to speed on these two essential printing functions. You can view the printf() man page by typing "man 3 printf" on your Linux system.

Here's some sample awk sprintf() and printf() code. As you can see, everything looks almost identical to C.

x=1
b="foo"
printf("%s got a %d on the last test\n","Jim",83)
myout=("%s-%d",b,x)
print myout

This code will print:

Jim got a 83 on the last test
foo-1

String functions

Awk has a plethora of string functions, and that's a good thing. In awk, you really need string functions, since you can't treat a string as an array of characters as you can in other languages like C, C++, and Python. For example, if you execute the following code:

mystring="How are you doing today?"
print mystring[3]

You'll receive an error that looks something like this:

awk: string.gawk:59: fatal: attempt to use scalar as array

Oh, well. While not as convenient as Python's sequence types, awk's string functions get the job done. Let's take a look at them.

First, we have the basic length() function, which returns the length of a string. Here's how to use it:

print length(mystring)

This code will print the value:

24

OK, let's keep going. The next string function is called index, and will return the position of the occurrence of a substring in another string, or it will return 0 if the string isn't found. Using mystring, we can call it this way:

print index(mystring,"you")

Awk prints:

9

We move on to two more easy functions, tolower() and toupper(). As you might guess, these functions will return the string with all characters converted to lowercase or uppercase respectively. Notice that tolower() and toupper() return the new string, and don't modify the original. This code:

print tolower(mystring)
print toupper(mystring)
print mystring

....will produce this output:

how are you doing today?
HOW ARE YOU DOING TODAY?
How are you doing today?

So far so good, but how exactly do we select a substring or even a single character from a string? That's where substr() comes in. Here's how to call substr():


mysub=substr(mystring,startpos,maxlen)

mystring should be either a string variable or a literal string from which you'd like to extract a substring. startpos should be set to the starting character position, and maxlen should contain the maximum length of the string you'd like to extract. Notice that I said maximum length; if length(mystring) is shorter than startpos+maxlen, your result will be truncated. substr() won't modify the original string, but returns the substring instead. Here's an example:

print substr(mystring,9,3)

Awk will print:

you

If you regularly program in a language that uses array indices to access parts of a string (and who doesn't), make a mental note that substr() is your awk substitute. You'll need to use it to extract single characters and substrings; because awk is a string-based language, you'll be using it often.

Now, we move on to some meatier functions, the first of which is called match(). match() is a lot like index(), except instead of searching for a substring like index() does, it searches for a regular expression. The match() function will return the starting position of the match, or zero if no match is found. In addition, match() will set two variables called RSTART and RLENGTH. RSTART contains the return value (the location of the first match), and RLENGTH specifies its span in characters (or -1 if no match was found). Using RSTART, RLENGTH, substr(), and a small loop, you can easily iterate through every match in your string. Here's an example match() call:

print match(mystring,/you/), RSTART, RLENGTH

Awk will print:

9 9 3

String substitution

Now, we're going to look at a couple of string substitution functions, sub() and gsub(). These guys differ slightly from the functions we've looked at so far in that they actually modify the original string. Here's a template that shows how to call sub():

sub(regexp,replstring,mystring)

When you call sub(), it'll find the first sequence of characters in mystring that matches regexp, and it'll replace that sequence with replstring. sub() and gsub() have identical arguments; the only way they differ is that sub() will replace the first regexp match (if any), and gsub() will perform a global replace, swapping out all matches in the string. Here's an example sub() and gsub() call:

sub(/o/,"O",mystring)
print mystring
mystring="How are you doing today?"
gsub(/o/,"O",mystring)
print mystring

We had to reset mystring to its original value because the first sub() call modified mystring directly. When executed, this code will cause awk to output:

HOw are you doing today?
HOw are yOu dOing tOday?

Of course, more complex regular expressions are possible. I'll leave it up to you to test out some complicated regexps.

We wrap up our string function coverage by introducing you to a function called split(). split()'s job is to "chop up" a string and place the various parts into an integer-indexed array. Here's an example split() call:

numelements=split("Jan,Feb,Mar,Apr,May,Jun,Jul,Aug,Sep,Oct,Nov,Dec",mymonths,",")

When calling split(), the first argument contains the literal string or string variable to be chopped. In the second argument, you should specify the name of the array that split() will stuff the chopped parts into. In the third element, specify the separator that will be used to chop the strings up. When split() returns, it'll return the number of string elements that were split. split() assigns each one to an array index starting with one, so the following code:

print mymonths[1],mymonths[numelements]

....will print:

Jan Dec

Special string forms

A quick note -- when calling length(), sub(), or gsub(), you can drop the last argument and awk will apply the function call to $0 (the entire current line). To print the length of each line in a file, use this awk script:



{
    print length()
}

Financial fun

A few weeks ago, I decided to write my own checkbook balancing program in awk. I decided that I'd like to have a simple tab-delimited text file into which I can enter my most recent deposits and withdrawals. The idea was to hand this data to an awk script that would automatically add up all the amounts and tell me my balance. Here's how I decided to record all my transactions into my "ASCII checkbook":

23 Aug 2000    food    -    -    Y    Jimmy's Buffet        30.25

Every field in this file is separated by one or more tabs. After the date (field 1, $1), there are two fields called "expense category" and "income category". When I'm entering an expense like on the above line, I put a four-letter nickname in the exp field, and a "-" (blank entry) in the inc field. This signifies that this particular item is a "food expense" :) Here's what a deposit looks like:

23 Aug 2000    -    inco    -    Y    Boss Man        2001.00

In this case, I put a "-" (blank) in the exp category, and put "inco" in the inc category. "inco" is my nickname for generic (paycheck-style) income. Using category nicknames allows me to generate a breakdown of my income and expenditures by category. As far as the rest of the records, all the other fields are fairly self-explanatory. The cleared? field ("Y" or "N") records whether the transaction has been posted to my account; beyond that, there's a transaction description, and a positive dollar amount.

The algorithm used to compute the current balance isn't too hard. Awk simply needs to read in each line, one by one. If an expense category is listed but there is no income category (it's "-"), then this item is a debit. If an income category is listed, but no expense category ("-") is there, then the dollar amount is a credit. And, if there is both an expense and income category listed, then this amount is a "category transfer"; that is, the dollar amount will be subtracted from the expense category and added to the income category. Again, all these categories are virtual, but are very useful for tracking income and expenditures, as well as for budgeting.

The code

Time to look at the code. We'll start off with the first line, the BEGIN block and a function definition:

balance, part 1


#!/usr/bin/env awk -f
BEGIN {
    FS="\t+"
    months="Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec"
}

function monthdigit(mymonth) {
    return (index(months,mymonth)+3)/4
}

Adding the first "#!..." line to any awk script will allow it to be directly executed from the shell, provided that you "chmod +x myscript" first. The remaining lines define our BEGIN block, which gets executed before awk starts processing our checkbook file. We set FS (the field separator) to "\t+", which tells awk that the fields will be separated by one or more tabs. In addition, we define a string called months that's used by our monthdigit() function, which appears next.

The last three lines show you how to define your own awk function. The format is simple -- type "function", then the function name, and then the parameters separated by commas, inside parentheses. After this, a "{ }" code block contains the code that you'd like this function to execute. All functions can access global variables (like our months variable). In addition, awk provides a "return" statement that allows the function to return a value, and operates similarly to the "return" found in C, Python, and other languages. This particular function converts a month name in a 3-letter string format into its numeric equivalent. For example, this:

print monthdigit("Mar")

....will print this:

3

Now, let's move on to some more functions.

Financial functions

Here are three more functions that perform the bookkeeping for us. Our main code block, which we'll see soon, will process each line of the checkbook file sequentially, calling one of these functions so that the appropriate transactions are recorded in an awk array. There are three basic kinds of transactions, credit (doincome), debit (doexpense) and transfer (dotransfer). You'll notice that all three functions accept one argument, called mybalance. mybalance is a placeholder for a two-dimensional array, which we'll pass in as an argument. Up until now, we haven't dealt with two-dimensional arrays; however, as you can see below, the syntax is quite simple. Just separate each dimension with a comma, and you're in business.

We'll record information into "mybalance" as follows. The first dimension of the array ranges from 0 to 12, and specifies the month, or zero for the entire year. Our second dimension is a four-letter category, like "food" or "inco"; this is the actual category we're dealing with. So, to find the entire year's balance for the food category, you'd look in mybalance[0,"food"]. To find June's income, you'd look in mybalance[6,"inco"].

balance, part 2



function doincome(mybalance) {
    mybalance[curmonth,$3] += amount
    mybalance[0,$3] += amount
}

function doexpense(mybalance) {
    mybalance[curmonth,$2] -= amount
    mybalance[0,$2] -= amount
}

function dotransfer(mybalance) {
    mybalance[0,$2] -= amount
    mybalance[curmonth,$2] -= amount
    mybalance[0,$3] += amount
    mybalance[curmonth,$3] += amount
}

When doincome() or any of the other functions are called, we record the transaction in two places -- mybalance[0,category] and mybalance[curmonth, category], the entire year's category balance and the current month's category balance, respectively. This allows us to easily generate either an annual or monthly breakdown of income/expenditures later on.

If you look at these functions, you'll notice that the array referenced by mybalance is passed in my reference. In addition, we also refer to several global variables: curmonth, which holds the numeric value of the month of the current record, $2 (the expense category), $3 (the income category), and amount ($7, the dollar amount). When doincome() and friends are called, all these variables have already been set correctly for the current record (line) being processed.

The main block

Here's the main code block that contains the code that parses each line of input data. Remember, because we have set FS correctly, we can refer to the first field as $1, the second field as $2, etc. When doincome() and friends are called, the functions can access the current values of curmonth, $2, $3 and amount from inside the function. Take a look at the code and meet me on the other side for an explanation.

balance, part 3



{
    curmonth=monthdigit(substr($1,4,3))
    amount=$7

    #record all the categories encountered
    if ( $2 != "-" )
        globcat[$2]="yes"
    if ( $3 != "-" )
        globcat[$3]="yes"

    #tally up the transaction properly
    if ( $2 == "-" ) {
        if ( $3 == "-" ) {
            print "Error: inc and exp fields are both blank!"
            exit 1
        } else {
            #this is income
            doincome(balance)
            if ( $5 == "Y" )
                doincome(balance2)
        }
    } else if ( $3 == "-" ) {
        #this is an expense
        doexpense(balance)
        if ( $5 == "Y" )
            doexpense(balance2)
    } else {
        #this is a transfer
        dotransfer(balance)
        if ( $5 == "Y" )
            dotransfer(balance2)
    }
}

In the main block, the first two lines set curmonth to an integer between 1 and 12, and set amount to field 7 (to make the code easier to understand). Then, we have four interesting lines, where we write values into an array called globcat. globcat, or the global categories array, is used to record all those categories encountered in the file -- "inco", "misc", "food", "util", etc. For example, if $2 == "inco", we set globcat["inco"] to "yes". Later on, we can iterate through our list of categories with a simple "for (x in globcat)" loop.

On the next twenty or so lines, we analyze fields $2 and $3, and record the transaction appropriately. If $2=="-" and $3!="-", we have some income, so we call doincome(). If the situation is reversed, we call doexpense(); and if both $2 and $3 contain categories, we call dotransfer(). Each time, we pass the "balance" array to these functions so that the appropriate data is recorded there.

You'll also notice several lines that say "if ( $5 == "Y" ), record that same transaction in balance2". What exactly are we doing here? You'll recall that $5 contains either a "Y" or a "N", and records whether the transaction has been posted to the account. Because we record the transaction to balance2 only if the transaction has been posted, balance2 will contain the actual account balance, while "balance" will contain all transactions, whether they have been posted or not. You can use balance2 to verify your data entry (since it should match with your current account balance according to your bank), and use "balance" to make sure that you don't overdraw your account (since it will take into account any checks you have written that have not yet been cashed).

Generating the report

After the main block repeatedly processes each input record, we now have a fairly comprehensive record of debits and credits broken down by category and by month. Now, all we need to do is define an END block that will generate a report, in this case a modest one:

END {
    bal=0
    bal2=0
    for (x in globcat) {
        bal=bal+balance[0,x]
        bal2=bal2+balance2[0,x]
        }
        printf("Your available funds: %10.2f\n", bal)
        printf("Your account balance: %10.2f\n", bal2)
}

This report prints out a summary that looks something like this:

Your available funds:    1174.22
Your account balance:    2399.33

In our END block, we used the "for (x in globcat)" construct to iterate through every category, tallying up a master balance based on all the transactions recorded. We actually tally up two balances, one for available funds, and another for the account balance. To execute the program and process your own financial goodies that you've entered into a file called "mycheckbook.txt", put all the above code into a text file called "balance", "chmod +x balance", and then type "./balance mycheckbook.txt". The balance script will then add up all your transactions and print out a two-line balance summary for you.

Upgrades

I use a more advanced version of this program to manage my personal and business finances. My version (which I couldn't include here due to space limitations) prints out a monthly breakdown of income and expenses, including annual totals, net income and a bunch of other stuff. Even better, it outputs the data in HTML format, so that I can view it in a Web browser :) If you find this program useful, I encourage you to add these features to this script. You won't need to configure it to record any additional information; all the information you need is already in balance and balance2. Just upgrade the END block, and you're in business!

I hope you've enjoyed this series. For more information on awk, check out the resources listed below.

Resources

    * Read earlier installments in the awk series: Awk by example, Part 1 and Part 2 on this blog.

    * If you'd like a good old-fashioned book, O'Reilly's sed & awk, 2nd Edition is a wonderful choice.

    * Be sure to check out the comp.lang.awk FAQ. It also contains lots of additional awk links.

    * Patrick Hartigan's awk tutorial is packed with handy awk scripts.

Learning Awk by example: Part 2

2011-02-17T11:39:00.001-07:00

Multi-line records

Awk is an excellent tool for reading in and processing structured data, such as the system's /etc/passwd file. /etc/passwd is the UNIX user database, and is a colon-delimited text file, containing a lot of important information, including all existing user accounts and user IDs, among other things. In my previous article, I showed you how awk could easily parse this file. All we had to do was to set the FS (field separator) variable to ":".

By setting the FS variable correctly, awk can be configured to parse almost any kind of structured data, as long as there is one record per line. However, just setting FS won't do us any good if we want to parse a record that exists over multiple lines. In these situations, we also need to modify the RS record separator variable. The RS variable tells awk when the current record ends and a new record begins.

As an example, let's look at how we'd handle the task of processing an address list of Federal Witness Protection Program participants:

Jimmy the Weasel
100 Pleasant Drive
Edmonton, AB T6X0R5

Big Tony
200 Incognito Ave.
Toronto, ON M3R1X6
Ideally, we'd like awk to recognize each 3-line address as an individual record, rather than as three separate records. It would make our code a lot simpler if awk would recognize the first line of the address as the first field ($1), the street address as the second field ($2), and the city, state, and zip code as field $3. The following code will do just what we want:

BEGIN {
    FS="\n"
    RS=""
}

Above, setting FS to "\n" tells awk that each field appears on its own line. By setting RS to "", we also tell awk that each address record is separated by a blank line. Once awk knows how the input is formatted, it can do all the parsing work for us, and the rest of the script is simple. Let's look at a complete script that will parse this address list and print out each address record on a single line, separating each field with a comma.

BEGIN {
    FS="\n"
    RS=""
}
{
    print $1 ", " $2 ", " $3
}

If this script is saved as address.awk, and the address data is stored in a file called address.txt, you can execute this script by typing "awk -f address.awk address.txt". This code produces the following output:

Jimmy the Weasel, 100 Pleasant Drive, Edmonton, AB T6X0R5
Big Tony, 200 Incognito Ave., Toronto, ON M3R1X6

OFS and ORS

In address.awk's print statement, you can see that awk concatenates (joins) strings that are placed next to each other on a line. We used this feature to insert a comma and a space (", ") between the three address fields that appeared on the line. While this method works, it's a bit ugly looking. Rather than inserting literal ", " strings between our fields, we can have awk do it for us by setting a special awk variable called OFS. Take a look at this code snippet.

print "Hello", "there", "Jim!"

The commas on this line are not part of the actual literal strings. Instead, they tell awk that "Hello", "there", and "Jim!" are separate fields, and that the OFS variable should be printed between each string. By default, awk produces the following output:

Hello there Jim!

This shows us that by default, OFS is set to " ", a single space. However, we can easily redefine OFS so that awk will insert our favorite field separator. Here's a revised version of our original address.awk program that uses OFS to output those intermediate ", " strings:

BEGIN {
    FS="\n"
    RS=""
    OFS=", "
}
{
    print $1, $2, $3
}

Awk also has a special variable called ORS, called the "output record separator". By setting ORS, which defaults to a newline ("\n"), we can control the character that's automatically printed at the end of a print statement. The default ORS value causes awk to output each new print statement on a new line. If we wanted to make the output double-spaced, we would set ORS to "\n\n". Or, if we wanted records to be separated by a single space (and no newline), we would set ORS to " ".

Multi-line to tabbed

Let's say that we wrote a script that converted our address list to a single-line per record, tab-delimited format for import into a spreadsheet. After using a slightly modified version of address.awk, it would become clear that our program only works for three-line addresses. If awk encountered the following address, the fourth line would be thrown away and not printed:

Cousin Vinnie
Vinnie's Auto Shop
300 City Alley
Vancouver, BC D7E1K6

To handle situations like this, it would be good if our code took the number of records per field into account, printing each one in order. Right now, the code only prints the first three fields of the address. Here's some code that does what we want:

BEGIN {
    FS="\n"
    RS=""
    ORS=""
}

{
        x=1
        while ( x
                print $x "\t"
                x++
        }
        print $NF "\n"
}

First, we set the field separator FS to "\n" and the record separator RS to "" so that awk parses the multi-line addresses correctly, as before. Then, we set the output record separator ORS to "", which will cause the print statement to not output a newline at the end of each call. This means that if we want any text to start on a new line, we need to explicitly write print "\n".

In the main code block, we create a variable called x that holds the number of current field that we're processing. Initially, it's set to 1. Then, we use a while loop (an awk looping construct identical to that found in the C language) to iterate through all but the last record, printing the record and a tab character. Finally, we print the last record and a literal newline; again, since ORS is set to "", print won't output newlines for us. Program output looks like this, which is exactly what we wanted:

Our intended output. Not pretty, but tab delimited for easy import into a spreadsheet

Jimmy the Weasel    100 Pleasant Drive    Edmonton, AB T6X0R5
Big Tony    200 Incognito Ave.    Toronto, ON M3R1X6
Cousin Vinnie    Vinnie's Auto Shop    300 City Alley    Vancouver, BC D7E1K6

Looping constructs

We've already seen awk's while loop construct, which is identical to its C counterpart. Awk also has a "do...while" loop that evaluates the condition at the end of the code block, rather than at the beginning like a standard while loop. It's similar to "repeat...until" loops that can be found in other languages. Here's an example:

do...while example

{
    count=1
    do {
        print "I get printed at least once no matter what"
    } while ( count != 1 )
}

Because the condition is evaluated after the code block, a "do...while" loop, unlike a normal while loop, will always execute at least once. On the other hand, a normal while loop will never execute if its condition is false when the loop is first encountered.

for loops
Awk allows you to create for loops, which like while loops are identical to their C counterpart:

for ( initial assignment; comparison; increment ) {
    code block
}

Here's a quick example:

for ( x = 1; x <= 4; x++ ) {
    print "iteration",x
}

This snippet will print:

iteration 1
iteration 2
iteration 3
iteration 4

Arrays

You'll be pleased to know that awk has arrays. However, under awk, it's customary to start array indices at 1, rather than 0:

myarray[1]="jim"
myarray[2]=456

When awk encounters the first assignment, myarray is created and the element myarray[1] is set to "jim". After the second assignment is evaluated, the array has two elements.

Iterating over arrays
Once defined, awk has a handy mechanism to iterate over the elements of an array, as follows:

for ( x in myarray ) {
    print myarray[x]
}

This code will print out every element in the array myarray. When you use this special "in" form of a for loop, awk will assign every existing index of myarray to x (the loop control variable) in turn, executing the loop's code block once after each assignment. While this is a very handy awk feature, it does have one drawback -- when awk cycles through the array indices, it doesn't follow any particular order. That means that there's no way for us to know whether the output of above code will be:

jim
456

or

456
jim

To loosely paraphrase Forrest Gump, iterating over the contents of an array is like a box of chocolates -- you never know what you're going to get. This has something to do with the "stringiness" of awk arrays, which we'll now take a look at.

Array index stringiness

In the previous article, I showed you that awk actually stores numeric values in a string format. While awk performs the necessary conversions to make this work, it does open the door for some odd-looking code:

a="1"
b="2"
c=a+b+3

After this code executes, c is equal to 6. Since awk is "stringy", adding strings "1" and "2" is functionally no different than adding the numbers 1 and 2. In both cases, awk will successfully perform the math. Awk's "stringy" nature is pretty intriguing -- you may wonder what happens if we use string indexes for arrays. For instance, take the following code:

myarr["1"]="Mr. Whipple"
print myarr["1"]

As you might expect, this code will print "Mr. Whipple". But how about if we drop the quotes around the second "1" index?

myarr["1"]="Mr. Whipple"
print myarr[1]

Guessing the result of this code snippet is a bit more difficult. Does awk consider myarr["1"] and myarr[1] to be two separate elements of the array, or do they refer to the same element? The answer is that they refer to the same element, and awk will print "Mr. Whipple", just as in the first code snippet. Although it may seem strange, behind the scenes awk has been using string indexes for its arrays all this time!

After learning this strange fact, some of us may be tempted to execute some wacky code that looks like this:

myarr["name"]="Mr. Whipple"
print myarr["name"]

Not only does this code not raise an error, but it's functionally identical to our previous examples, and will print "Mr. Whipple" just as before! As you can see, awk doesn't limit us to using pure integer indexes; we can use string indexes if we want to, without creating any problems. Whenever we use non-integer array indices like myarr["name"], we're using associative arrays. Technically, awk isn't doing anything different behind the scenes than when we use a string index (since even if you use an "integer" index, awk still treats it as a string). However, you should still call 'em associative arrays -- it sounds cool and will impress your boss. The stringy index thing will be our little secret. ;)

Array tools

When it comes to arrays, awk gives us a lot of flexibility. We can use string indexes, and we aren't required to have a continuous numeric sequence of indices (for example, we can define myarr[1] and myarr[1000], but leave all other elements undefined). While all this can be very helpful, in some circumstances it can create confusion. Fortunately, awk offers a couple of handy features to help make arrays more manageable.

First, we can delete array elements. If you want to delete element 1 of your array fooarray, type:

delete fooarray[1]

And, if you want to see if a particular array element exists, you can use the special "in" boolean operator as follows:

if ( 1 in fooarray ) {
    print "Ayep! It's there."
} else {
    print "Nope! Can't find it."
}
Next time

We've covered a lot of ground in this article. Next time, I'll round out your awk knowledge by showing you how to use awk's math and string functions and how to create your own functions. I'll also walk you through the creation of a checkbook balancing program. Until then, I encourage you to write some of your own awk programs, and to check out the following resources.

Resources

    * Read Awk by example, Part 1 on this blog.

    * If you'd like a good old-fashioned book, O'Reilly's sed & awk, 2nd Edition is a wonderful choice.

    * Be sure to check out the comp.lang.awk FAQ. It also contains lots of additional awk links.

    * Patrick Hartigan's awk tutorial is packed with handy awk scripts.

Learning Awk by example: Part 1

2011-02-16T15:47:00.001-07:00

In defense of awk

Sure, awk doesn't have a great name. But it is a great language. Awk is geared toward text processing and report generation, yet features many well-designed features that allow for serious programming. And, unlike some languages, awk's syntax is familiar, and borrows some of the best parts of languages like C, python, and bash (although, technically, awk was created before both python and bash). Awk is one of those languages that, once learned, will become a key part of your strategic coding arsenal.

The first awk

Let's go ahead and start playing around with awk to see how it works. At the command line, enter the following command:

$ awk '{ print }' /etc/passwd

You should see the contents of your /etc/passwd file appear before your eyes. Now, for an explanation of what awk did. When we called awk, we specified /etc/passwd as our input file. When we executed awk, it evaluated the print command for each line in /etc/passwd, in order. All output is sent to stdout, and we get a result identical to catting /etc/passwd. Now, for an explanation of the { print } code block. In awk, curly braces are used to group blocks of code together, similar to C. Inside our block of code, we have a single print command. In awk, when a print command appears by itself, the full contents of the current line are printed.

Here is another awk example that does exactly the same thing:

$ awk '{ print $0 }' /etc/passwd

In awk, the $0 variable represents the entire current line, so print and print $0 do exactly the same thing. If you'd like, you can create an awk program that will output data totally unrelated to the input data. Here's an example:

$ awk '{ print "" }' /etc/passwd

Whenever you pass the "" string to the print command, it prints a blank line. If you test this script, you'll find that awk outputs one blank line for every line in your /etc/passwd file. Again, this is because awk executes your script for every line in the input file. Here's another example:

$ awk '{ print "hiya" }' /etc/passwd

Running this script will fill your screen with hiya's. :)

Multiple fields

Awk is really good at handling text that has been broken into multiple logical fields, and allows you to effortlessly reference each individual field from inside your awk script. The following script will print out a list of all user accounts on your system:

$ awk -F":" '{ print $1 }' /etc/passwd

Above, when we called awk, we use the -F option to specify ":" as the field separator. When awk processes the print $1 command, it will print out the first field that appears on each line in the input file. Here's another example:

$ awk -F":" '{ print $1 $3 }' /etc/passwd

Here's an excerpt of the output from this script:

halt7
operator11
root0
shutdown6
sync5
bin1
....etc.

As you can see, awk prints out the first and third fields of the /etc/passwd file, which happen to be the username and uid fields respectively. Now, while the script did work, it's not perfect -- there aren't any spaces between the two output fields! If you're used to programming in bash or python, you may have expected the print $1 $3 command to insert a space between the two fields. However, when two strings appear next to each other in an awk program, awk concatenates them without adding an intermediate space. The following command will insert a space between both fields:

$ awk -F":" '{ print $1 " " $3 }' /etc/passwd

When you call print this way, it'll concatenate $1, " ", and $3, creating readable output. Of course, we can also insert some text labels if needed:

$ awk -F":" '{ print "username: " $1 "\t\tuid:" $3 }' /etc/passwd

This will cause the output to be:

username: halt     uid:7
username: operator uid:11
username: root     uid:0
username: shutdown uid:6
username: sync     uid:5
username: bin      uid:1
....etc.

External scripts

Passing your scripts to awk as a command line argument can be very handy for small one-liners, but when it comes to complex, multi-line programs, you'll definitely want to compose your script in an external file. Awk can then be told to source this script file by passing it the -f option:

$ awk -f myscript.awk myfile.in

Putting your scripts in their own text files also allows you to take advantage of additional awk features. For example, this multi-line script does the same thing as one of our earlier one-liners, printing out the first field of each line in /etc/passwd:

BEGIN {
        FS=":"
}
{ print $1 }

The difference between these two methods has to do with how we set the field separator. In this script, the field separator is specified within the code itself (by setting the FS variable), while our previous example set FS by passing the -F":" option to awk on the command line. It's generally best to set the field separator inside the script itself, simply because it means you have one less command line argument to remember to type. We'll cover the FS variable in more detail later in this article.

The BEGIN and END blocks

Normally, awk executes each block of your script's code once for each input line. However, there are many programming situations where you may need to execute initialization code before awk begins processing the text from the input file. For such situations, awk allows you to define a BEGIN block. We used a BEGIN block in the previous example. Because the BEGIN block is evaluated before awk starts processing the input file, it's an excellent place to initialize the FS (field separator) variable, print a heading, or initialize other global variables that you'll reference later in the program.

Awk also provides another special block, called the END block. Awk executes this block after all lines in the input file have been processed. Typically, the END block is used to perform final calculations or print summaries that should appear at the end of the output stream.

Regular expressions and blocks

Awk allows the use of regular expressions to selectively execute an individual block of code, depending on whether or not the regular expression matches the current line. Here's an example script that outputs only those lines that contain the character sequence foo:

/foo/ { print }

Of course, you can use more complicated regular expressions. Here's a script that will print only lines that contain a floating point number:

/[0-9]+\.[0-9]*/ { print }

Expressions and blocks

There are many other ways to selectively execute a block of code. We can place any kind of boolean expression before a code block to control when a particular block is executed. Awk will execute a code block only if the preceding boolean expression evaluates to true. The following example script will output the third field of all lines that have a first field equal to fred. If the first field of the current line is not equal to fred, awk will continue processing the file and will not execute the print statement for the current line:

$1 == "fred" { print $3 }

Awk offers a full selection of comparison operators, including the usual "==", "<", ">", "<=", ">=", and "!=". In addition, awk provides the "~" and "!~" operators, which mean "matches" and "does not match". They're used by specifying a variable on the left side of the operator, and a regular expression on the right side. Here's an example that will print only the third field on the line if the fifth field on the same line contains the character sequence root:

$5 ~ /root/ { print $3 }

Conditional statements

Awk also offers very nice C-like if statements. If you'd like, you could rewrite the previous script using an if statement:

{
if ( $5 ~ /root/ ) {
          print $3
}
}

Both scripts function identically. In the first example, the boolean expression is placed outside the block, while in the second example, the block is executed for every input line, and we selectively perform the print command by using an if statement. Both methods are available, and you can choose the one that best meshes with the other parts of your script.

Here's a more complicated example of an awk if statement. As you can see, even with complex, nested conditionals, if statements look identical to their C counterparts:

{
if ( $1 == "foo" ) {
           if ( $2 == "foo" ) {
                    print "uno"
           } else {
                    print "one"
           }
} else if ($1 == "bar" ) {
           print "two"
} else {
           print "three"
}
}

Using if statements, we can also transform this code:

! /matchme/ { print $1 $3 $4 }

to this:

{
if ( $0 !~ /matchme/ ) {
          print $1 $3 $4
}
}

Both scripts will output only those lines that don't contain a matchme character sequence. Again, you can choose the method that works best for your code. They both do the same thing.

Awk also allows the use of boolean operators "||" (for "logical or") and "&&"(for "logical and") to allow the creation of more complex boolean expressions:

( $1 == "foo" ) && ( $2 == "bar" ) { print }

This example will print only those lines where field one equals foo and field two equals bar.

Numeric variables!

So far, we've either printed strings, the entire line, or specific fields. However, awk also allows us to perform both integer and floating point math. Using mathematical expressions, it's very easy to write a script that counts the number of blank lines in a file. Here's one that does just that:

BEGIN { x=0 }
/^$/ { x=x+1 }
END   { print "I found " x " blank lines. :)" }

In the BEGIN block, we initialize our integer variable x to zero. Then, each time awk encounters a blank line, awk will execute the x=x+1 statement, incrementing x. After all the lines have been processed, the END block will execute, and awk will print out a final summary, specifying the number of blank lines it found.

Stringy variables

One of the neat things about awk variables is that they are "simple and stringy." I consider awk variables "stringy" because all awk variables are stored internally as strings. At the same time, awk variables are "simple" because you can perform mathematical operations on a variable, and as long as it contains a valid numeric string, awk automatically takes care of the string-to-number conversion steps. To see what I mean, check out this example:

x="1.01"
# We just set x to contain the *string* "1.01"
x=x+1
# We just added one to a *string*
print x
# Incidentally, these are comments :)

Awk will output:

2.01

Interesting! Although we assigned the string value 1.01 to the variable x, we were still able to add one to it. We wouldn't be able to do this in bash or python. First of all, bash doesn't support floating point arithmetic. And, while bash has "stringy" variables, they aren't "simple"; to perform any mathematical operations, bash requires that we enclose our math in an ugly $( ) construct. If we were using python, we would have to explicitly convert our 1.01 string to a floating point value before performing any arithmetic on it. While this isn't difficult, it's still an additional step. With awk, it's all automatic, and that makes our code nice and clean. If we wanted to square and add one to the first field in each input line, we would use this script:

{ print ($1^2)+1 }

If you do a little experimenting, you'll find that if a particular variable doesn't contain a valid number, awk will treat that variable as a numerical zero when it evaluates your mathematical expression.

Lots of operators

Another nice thing about awk is its full complement of mathematical operators. In addition to standard addition, subtraction, multiplication, and division, awk allows us to use the previously demonstrated exponent operator "^", the modulo (remainder) operator "%", and a bunch of other handy assignment operators borrowed from C.

These include pre- and post-increment/decrement ( i++, --foo ), add/sub/mult/div assign operators ( a+=3, b*=2, c/=2.2, d-=6.2 ). But that's not all -- we also get handy modulo/exponent assign ops as well ( a^=2, b%=4 ).

Field separators

Awk has its own complement of special variables. Some of them allow you to fine-tune how awk functions, while others can be read to glean valuable information about the input. We've already touched on one of these special variables, FS. As mentioned earlier, this variable allows you to set the character sequence that awk expects to find between fields. When we were using /etc/passwd as input, FS was set to ":". While this did the trick, FS allows us even more flexibility.

The FS value is not limited to a single character; it can also be set to a regular expression, specifying a character pattern of any length. If you're processing fields separated by one or more tabs, you'll want to set FS like so:

FS="\t+"

Above, we use the special "+" regular expression character, which means "one or more of the previous character".

If your fields are separated by whitespace (one or more spaces or tabs), you may be tempted to set FS to the following regular expression:

FS="[[:space:]+]"

While this assignment will do the trick, it's not necessary. Why? Because by default, FS is set to a single space character, which awk interprets to mean "one or more spaces or tabs." In this particular example, the default FS setting was exactly what you wanted in the first place!

Complex regular expressions are no problem. Even if your records are separated by the word "foo," followed by three digits, the following regular expression will allow your data to be parsed properly:

FS="foo[0-9][0-9][0-9]"

Number of fields

The next two variables we're going to cover are not normally intended to be written to, but are normally read and used to gain useful information about the input. The first is the NF variable, also called the "number of fields" variable. Awk will automatically set this variable to the number of fields in the current record. You can use the NF variable to display only certain input lines:

NF == 3 { print "this particular record has three fields: " $0 }

Of course, you can also use the NF variable in conditional statements, as follows:

{
if ( NF > 2 ) {
          print $1 " " $2 ":" $3
}
}

Record number

The record number (NR) is another handy variable. It will always contain the number of the current record (awk counts the first record as record number 1). Up until now, we've been dealing with input files that contain one record per line. For these situations, NR will also tell you the current line number. NR can be used like the NF variable to print only certain lines of the input:

(NR < 10 ) || (NR > 100) { print "We are on record number 1-9 or 101+" }

Another example:

{
#skip header
if ( NR > 10 ) {
          print "ok, now for the real information!"
}
}

Awk provides additional variables that can be used for a variety of purposes. We've come to the end of our initial exploration of awk. In the meantime, if you're eager to learn more, check out the resources listed below.

Resources

If you'd like a good old-fashioned book, O'Reilly's sed & awk, 2nd Edition is a wonderful choice.

Be sure to check out the comp.lang.awk FAQ. It also contains lots of additional awk links.

Patrick Hartigan's awk tutorial is packed with handy awk scripts.

Thompson's TAWK Compiler compiles awk scripts into fast binary executables. Versions are available for Windows, and DOS.

The GNU Awk User's Guide is available for online reference.

Untangling the Power of the Linux "find" command

2010-12-29T15:59:00.001-07:00

"find" is one of the most useful Linux/Unix tools in the toolbelt, but most people use only a fraction of its power. Many Linux/Unix questions seen online can be solved using the find command alone, getting familiar with its main functionality is one of the best things you can do for yourself in the long-term.

find lets you do anything from finding all your .jpg files to seeing "all of Michael's text documents that have the execute bit set and have been modified since yesterday." And when combined with exec or xargs, a properly constructed command can make quick work of some very heavy tasks.

Basics

Let's start simple by looking for things by name. Remember that the first argument you give find is where to look.

# find all files with something in their name

find . -name "*.jpg"

...
./Pictures/iPhoto Library/Data/2006/Roll 20/00697_bluewaters_1440x900.jpg
./Pictures/iPhoto Library/Data/2006/Roll 20/00705_cloudyday_1440x900.jpg
./Pictures/iPhoto Library/Data/2006/Roll 20/00710_fragile_1600x1200.jpg
./Pictures/iPhoto Library/Data/2006/Roll 20/00713_coolemoticon_1440x900.jpg
./Pictures/iPhoto Library/Data/2006/Roll 20/00714_cloudyday_1440x900.jpg
...

Note that by default when you give a location to start from (in our case "."), the find command starts there and drills all the way down during its search. So in this case I started from my home directory and it found the files all the way down in "~/Pictures/iPhoto Library/Data/2006/Roll 20" as well.

[ Placing quotes around the search criteria avoids issues with wildcard characters and is probably a good habit to get into. You can also use -iname instead of -name; it's the same but it's case insensitive ]

Find by User

# find all files that belong to a certain user

find . -user adaniel

...
./Music/iTunes/iTunes Music/Tool/Undertow/01 Intolerance.m4a
./Music/iTunes/iTunes Music/Tool/Undertow/02 Prison Sex.m4a
./Music/iTunes/iTunes Music/Tool/Undertow/03 Sober.m4a
./Music/iTunes/iTunes Music/Tool/Undertow/04 Bottom.m4a
./Music/iTunes/iTunes Music/Tool/Undertow/05 Crawl Away.m4a
./Music/iTunes/iTunes Music/Tool/Undertow/06 Swamp Song.m4a
./Music/iTunes/iTunes Music/Tool/Undertow/07 Undertow.m4a
./Music/iTunes/iTunes Music/Tool/Undertow/08 4 Degrees.m4a
./Music/iTunes/iTunes Music/Tool/Undertow/09 Flood.m4a
./Music/iTunes/iTunes Music/Tool/Undertow/69 Disgustipated.m4a
...

[ Also works for groups (-group) ]

Find by Type

# find only directories, files, links, or sockets

find . -type d

...

./Development/envelope

./Development/mhp

./Development/mservers

./Development/mservers/fortune100

./Development/mst

./Development/mst/nmap

./Development/mst/services

...

Those are all directories, and to look for the others (files, links, or sockets), just substitute f, l, s for the d in the command above.

Find by Size

# find things over a gigabyte in size

find ~/Movies/ -size +1024M

...

/Movies/Comedy/Funny.mpg

/Movies/Drama/Sad.avi

...

[ +M indicates that you're searching in megabytes, but you can also search in bytes or kilobytes if so desired. ]

Find by Modification Time

find also has a number of options that help one answer forensics-oriented questions such as when a file's contents or permissions were last changed.

# find all files in /etc owned by root that have been modified within the last day

find /etc/ -user root -mtime 1

...

/etc/passwd

/etc/shadow

...

The checks you can use here are:

* -atime: when the file was last accessed

* -ctime: when the file's permissions were last changed

* -mtime: when the file's data was last modified

These searches are done in 24 hour increments and followed by a number n. If you want to match the exact 24 hour period you use n by itself. More frequently, however, you'll want to say everything since yesterday, or everything "more than 3 days ago." This is accomplished using the -n and +n options respectively.

There are also minute versions of the atime, ctime, and mtime arguments:

* -amin: when (in minutes) the file was last accessed

* -cmin: when (in minutes) the file's permissions were last changed

* -mmin: when (in minutes) the file's data was last modified

Find by Permissions

# find all files in my directory with open permissions

find ~ -perm 777

...

~/testfile.txt

~/lab.rtf

...

Additional Forensics-oriented Options:

* -nouser: shows output that's not associated with an existing userid

* -nogroup: shows output not associated with an existing groupid

* -links n: file has n links

* -newer file: file was modified more recently than file.

* -perm mode: file has mode permissions.

Combinations

Just as with any good unix/linux command, the real power comes in combining options. You can combine find arguments using and, or, and not. By default if you use two different arguments you're and'ing them. If you want to use or you give the -o option, and if you want to get everything except something, you use the ! option.

# find .jpg images (files) owned by daniel

find . -user adaniel -type f -name *.jpg

...

./Pictures/iPhoto Library/autumn_woods.jpg

./Pictures/iPhoto Library/blue_forest.jpg

./Pictures/iPhoto Library/brothers.jpg

...

# now do the same, but exclude anything with "autumn" in the name

find . -user adaniel -type f -name *.jpg ! -name autumn*

...

./Pictures/iPhoto Library/blue_forest.jpg

./Pictures/iPhoto Library/brothers.jpg

...

# show me all ruby programs in /apps owned by root that have been accessed in the last two minutes

find /apps/ -user root -type f -amin -2 -name *.rb

...

/apps/testing.rb

/apps/runme.rb

...

Combining find with exec and xargs

What's the fun in finding a bunch of stuff if you're not going to do something with it? So while it's interesting to say, "show me stuff", it's far more useful to say, "Take every text file owned by ex-employee Jason that's hasn't been accessed in 60 days and move it to a remote backup folder."

Cookbook Examples Of find in action

Find all files on your system that are world writable. The 0002 denotes a 2 in the "other" field in the file permissions, which is the write bit:

find / -perm -0002

Collect files that are not owned by valid users and delete them:

find / -nouser -print0 | xargs -0 rm

Clean the images off of your *nix desktop:

find ~/Desktop -name "*.jpg" -o -name "*.gif" -o -name "*.png" -print0 | xargs -0 mv --target-directory ~/Pictures

** The -print0 option terminates results with a null character instead of the default newline, making it cleaner and less likely to balk in many cases

Correct the permissions on your web directory:

find /your/webdir/ -type d -print0 | xargs -0 chmod 755

find /your/webdir -type f | xargs chmod 644

Show a list of files in /etc that have been modified since last month:

find /etc -mtime -30

25 Best SSH Commands / Tricks

2010-11-25T15:47:00.000-07:00

OpenSSH is a FREE version of the SSH connectivity tools that technical users of the Internet rely on. Users of telnet, rlogin, and ftp may not realize that their password is transmitted across the Internet unencrypted, but it is. OpenSSH encrypts all traffic (including passwords) to effectively eliminate eavesdropping, connection hijacking, and other attacks. Additionally, OpenSSH provides secure tunneling capabilities and several authentication methods, and supports all SSH protocol versions.

SSH is an awesome powerful tool, there are unlimited possibility when it comes to SSH, here is the my top 25 SSH commands

1) Copy ssh keys to user@host to enable password-less ssh logins

ssh-copy-id user@host

To generate the keys use the command ssh-keygen

2) Start a tunnel from some machine’s port 80 to your local post 2001

ssh -N -L2001:localhost:80 somemachine

Now you can acces the website by going to http://localhost:2001/

3) Output your microphone to a remote computer’s speaker

dd if=/dev/dsp | ssh -c arcfour -C username@host dd of=/dev/dsp

This will output the sound from your microphone port to the ssh target computer’s speaker port. The sound quality is very bad, so you will hear a lot of hissing.

4) Compare a remote file with a local file

ssh user@host cat /path/to/remotefile | diff /path/to/localfile -

Useful for checking if there are differences between local and remote files.

5) Mount folder/filesystem through SSH

sshfs name@server:/path/to/folder /path/to/mount/point

Install SSHFS from http://fuse.sourceforge.net/sshfs.html. This will allow you to mount a folder secure over a network.

6) SSH connection through host in the middle

ssh -t reachable_host ssh unreachable_host

Unreachable_host is unavailable from local network, but it’s available from reachable_host’s network. This command creates a connection to unreachable_host through “hidden” connection to reachable_host.

7) Copy from host1 to host2, through your host

ssh root@host1 “cd /somedir/tocopy/ && tar -cf – .” | ssh root@host2 “cd /samedir/tocopyto/ && tar -xf -”

Good if you only have access to host1 and host2, but they have no access to your host (so ncat won’t work) and they have no direct access to each other.

8) Run any GUI program remotely
ssh -fX <user>@<host> <program>

The SSH server configuration requires:

X11Forwarding yes # this is default in Debian

And it’s convenient to use:

Compression delayed

9) Create a persistent connection to a machine
ssh -MNf <user>@<host>

Create a persistent SSH connection to the host in the background. Combine this with settings in your ~/.ssh/config:

Host host
ControlPath ~/.ssh/master-%r@%h:%p
ControlMaster no

All the SSH connections to the machine will then go through the persisten SSH socket. This is very useful if you are using SSH to synchronize files (using rsync/sftp/cvs/svn) on a regular basis because it won’t create a new socket each time to open an ssh connection.

10) Attach screen over ssh

ssh -t remote_host screen -r

Directly attach a remote screen session (saves a useless parent bash process)

11) Port Knocking!

knock <host> 3000 4000 5000 && ssh -p <port> user@host && knock <host> 5000 4000 3000

Knock on ports to open a port to a service (ssh for example) and knock again to close the port. You have to install knockd.

See example config file below.

[options]
logfile = /var/log/knockd.log
[openSSH]
sequence = 3000,4000,5000
seq_timeout = 5
command = /sbin/iptables -A INPUT -i eth0 -s %IP% -p tcp –dport 22 -j ACCEPT
tcpflags = syn
[closeSSH]
sequence = 5000,4000,3000
seq_timeout = 5
command = /sbin/iptables -D INPUT -i eth0 -s %IP% -p tcp –dport 22 -j ACCEPT
tcpflags = syn

12) Remove a line in a text file. Useful to fix

ssh-keygen -R <the_offending_host>

In this case it’s better do to use the dedicated tool

13) Run complex remote shell cmds over ssh, without escaping quotes

ssh host -l user $(<cmd.txt)

Much simpler method. More portable version: ssh host -l user “`cat cmd.txt`”

14) Copy a MySQL Database to a new Server via SSH with one command

mysqldump –add-drop-table –extended-insert –force –log-error=error.log -uUSER -pPASS OLD_DB_NAME | ssh -C user@newhost “mysql -uUSER -pPASS NEW_DB_NAME”

Dumps a MySQL database over a compressed SSH tunnel and uses it as input to mysql – i think that is the fastest and best way to migrate a DB to a new server!

15) Remove a line in a text file. Useful to fix “ssh host key change” warnings

sed -i 8d ~/.ssh/known_hosts

16) Copy your ssh public key to a server from a machine that doesn’t have ssh-copy-id

cat ~/.ssh/id_rsa.pub | ssh user@machine “mkdir ~/.ssh; cat >> ~/.ssh/authorized_keys”

If you use Mac OS X or some other *nix variant that doesn’t come with ssh-copy-id, this one-liner will allow you to add your public key to a remote machine so you can subsequently ssh to that machine without a password.

17) Live ssh network throughput test

yes | pv | ssh $host “cat > /dev/null”

connects to host via ssh and displays the live transfer speed, directing all transferred data to /dev/null
needs pv installed
Debian: ‘apt-get install pv’
Fedora: ‘yum install pv’ (may need the ‘extras’ repository enabled)

18) How to establish a remote Gnu screen session that you can re-connect to

ssh -t user@some.domain.com /usr/bin/screen -xRR

Long before tabbed terminals existed, people have been using Gnu screen to open many shells in a single text terminal. Combined with ssh, it gives you the ability to have many open shells with a single remote connection using the above options. If you detach with “Ctrl-a d” or if the ssh session is accidentally terminated, all processes running in your remote shells remain undisturbed, ready for you to reconnect. Other useful screen commands are “Ctrl-a c” (open new shell) and “Ctrl-a a” (alternate between shells). Read this quick reference for more screen commands: http://aperiodic.net/screen/quick_reference

19) Resume scp of a big file

rsync –partial –progress –rsh=ssh $file_source $user@$host:$destination_file

It can resume a failed secure copy ( usefull when you transfer big files like db dumps through vpn ) using rsync.
It requires rsync installed in both hosts.

rsync –partial –progress –rsh=ssh $file_source $user@$host:$destination_file local -> remote

or

rsync –partial –progress –rsh=ssh $user@$host:$remote_file $destination_file remote -> local

20) Analyze traffic remotely over ssh w/ wireshark

ssh root@server.com ‘tshark -f “port !22″ -w -’ | wireshark -k -i -

This captures traffic on a remote machine with tshark, sends the raw pcap data over the ssh link, and displays it in wireshark. Hitting ctrl+C will stop the capture and unfortunately close your wireshark window. This can be worked-around by passing -c # to tshark to only capture a certain # of packets, or redirecting the data through a named pipe rather than piping directly from ssh to wireshark. I recommend filtering as much as you can in the tshark command to conserve bandwidth. tshark can be replaced with tcpdump thusly:

ssh root@example.com tcpdump -w – ‘port !22′ | wireshark -k -i -

21) Have an ssh session open forever

autossh -M50000 -t server.example.com ‘screen -raAd mysession’

Open a ssh session opened forever, great on laptops losing Internet connectivity when switching WIFI spots.

22) Harder, Faster, Stronger SSH clients

ssh -4 -C -c blowfish-cbc

We force IPv4, compress the stream, specify the cypher stream to be Blowfish. I suppose you could use aes256-ctr as well for cypher spec. I’m of course leaving out things like master control sessions and such as that may not be available on your shell although that would speed things up as well.

23) Throttle bandwidth with cstream

tar -cj /backup | cstream -t 777k | ssh host ‘tar -xj -C /backup’

this bzips a folder and transfers it over the network to “host” at 777k bit/s.
cstream can do a lot more, have a look http://www.cons.org/cracauer/cstream.html#usage

for example:

echo w00t, i’m 733+ | cstream -b1 -t2

24) Transfer SSH public key to another machine in one step

ssh-keygen; ssh-copy-id user@host; ssh user@host

This command sequence allows simple setup of (gasp!) password-less SSH logins. Be careful, as if you already have an SSH keypair in your ~/.ssh directory on the local machine, there is a possibility ssh-keygen may overwrite them. ssh-copy-id copies the public key to the remote host and appends it to the remote account’s ~/.ssh/authorized_keys file. When trying ssh, if you used no passphrase for your key, the remote shell appears soon after invoking ssh user@host.

25) Copy stdin to your X11 buffer

ssh user@host cat /path/to/some/file | xclip

Have you ever had to scp a file to your work machine in order to copy its contents to a mail? xclip can help you with that. It copies its stdin to the X11 buffer, so all you have to do is middle-click to paste the content of that looong file :)

Have Fun

Managing Samba (Part 6 of 6): Remote GUI tools

2010-10-27T17:38:00.000-06:00

While there are many tasks that can be performed from a command line, we no longer live in the Dark Ages, back when there were no graphical user interfaces (GUIs). There are graphical tools that can be used to manage Samba across a large corporate network. This tip addresses that subject. Are you ready?

Where eagles soar

Explore remote management options, and you'll find that neither free nor commercial Windows network management tools suit all IT shops' needs. There are no silver bullets. If your goal is to find a universal network management tool that will permit you to meet all site needs, no matter what, the news is not too bright.

While your thoughts may have been to soar with the eagles, you may have to content yourself with a more mundane solution. On the other hand, if your objective is to find a tool that may save a little hassle and make it possible for you to delegate some tasks to departmental managers, there is some good news.

Basic management tools

The Samba PDC (primary domain controller) configuration that was used in the last tip is rather simple. It uses the tdbsam binary database in which to store the SambaSAMAccount information.

The tdbsam passdb backend is unsuitable for use with a backup domain controller (BDC). Unix user and group accounts must be resolvable either from the /etc/passwd and /etc/group system account files or via the name service switcher (NSS) facility.

The fact that tdbsam does not permit operation of the PDC with a BDC means that this back end is limited to those sites that either have few users or many potential users at one location. The largest site I know that uses the tdbsam back end has 4,500 users. This back end can be used where the network is multi-segmented, but then fast WAN bandwidth is essential; otherwise, the infrastructure would collapse as a result of network logon failures.

With these provisos out of the way, let's consider remote network administration needs. Where this type of network configuration is in use, the most likely need is to permit delegation of the ability to join workstations to the network as well as user and group accounts. If care and attention is given to the design of file storage layout, it may not be necessary for departmental or divisional managers to create, delete or change share configurations.

In this situation, the Windows NT4 Domain User Manager will most likely be a sufficient tool for remote network management of users and groups. The NT4 Domain Server Manager, or the MMC Computer Management Console, could be used to administer shares and computers.

The NT4 Domain User Manager

In Figure 1, the Domain User Manager could be used to make Alex Monteiro a member of an additional group. It can also be used to change the primary group he belongs to. This is an example of a user account change.

Figure 1: Changing group membership

Figure 2 demonstrates the ability, using the NT4 Domain User Manager, to change the users who are members of a particular group.

Figure 2: Manipulation of group membership

It is increasingly necessary to limit user network access times and days. Figure 3 illustrates the capability to do this using the Windows NT4 Domain User Manager. This screen is accessed by clicking on the user account that must be managed.

Figure 3: Setting network access time limits

Domain access policies can also be set using the NT4 Domain User Manager. An example of policy setting is shown in Figure 4.

Figure 4: Domain access policy setting

Using the NT4 Domain Server Manager

The NT4 Domain Server Manager can be used to manage domain computer accounts, access controls on shares (share level ACLs) and to create shares. The example shown in Figure 5 demonstrates the settings of share level ACLs. In NT4, this is referred to as the setting of Share Permissions. The Share Permissions configuration acts as a filter that controls who can connect to the share. This ACL information is stored in the Samba file /var/lib/samba/share_info.tdb; it is not set in the smb.conf file.

Figure 5: Managing Share Permissions (ACLs)

A share can also be created using this tool. For that operation to work, the smb.conf file must define the script that should be called to add, change or delete a share. Example scripts are provided in the Samba source code tarball in the examples/scripts/share directory.

Figure 6 demonstrates the use of the NT4 Server Manager tool to create a new share. The example scripts that are included with Samba-3 DO NOT automatically create the new directory; that is a responsibility the administrator must take care of either before or after creating the share.

Figure 6: Adding a new share

The Windows XP MMC Computer Management Console

The Windows 2000/XP Professional MMC Computer Management Console can be used to manage share level ACLs as well as file system permission settings. This use of this tool to create a new share is shown in Figure 7.

Figure 7: Use of the MMC Console to create a new share

In Figures 6 and 7, it should be noted that the Samba UNIX path must be entered using Windows file system semantics. The path c:\data\archive\ will be automatically converted to the Unix semantics /data/archive/.

So far, it has been demonstrated that a Samba domain can be managed using the NT4 Domain User Manager and the NT4 Domain Server Manager. At this time, it is necessary to point out that these tools have their limitations when used from Windows 2000 or Windows XP Professional. That limitation is brought about because Microsoft has changed the underlying remote procedure call architecture in these products so that these tools can no longer be used to manage user rights and privileges from the later generation Windows platforms.

The NT4 Domain Management tools can be obtained from Microsoft's Web site. Just search for SRVTOOLS.EXE, download them and execute the file in the directory in which you want them to be installed.

There exists an equivalent tool called the Nessus Toolkit that can be installed on a Windows 9x/Me client. This tool can be used to manage all settings including user rights and privileges.

The NT4 Domain User Manager and the NT4 Domain Server Manager can be used, regardless of the passdb backend that Samba is using. However, sites that are using an LDAP directory to store and manage network users and groups will almost certainly desire an LDAP directory management tools that more succinctly manages the LDAP directory.

Administrative tool preferences are particularly personal. I have met administrators who prefer to use Webmin to manage the Samba configuration, user and group management.

There do not seem to be many today who make use of Samba's SWAT (Samba Web Administration Tool). SWAT was originally developed for the Samba-2 series.

LDAP Directory Managers

Just mentioning an LDAP directory browser is often enough to evoke a reaction. That is really just too bad, because directory browsers serve a very useful purpose, particularly for those who have a profound appreciation for the power they offer. Admittedly, not all LDAP browsers are created equally!

You may wish to check out the full range of open source offerings, or you may simply prefer to purchase a commercial tool set; the choice is yours. The following are LDAP directory browsing tools I find useful:

Name    Source URL    OS Platform Language
LDAP Browser http://www-unix.mcs.anl.gov/~gawor/ldap/download.html Any      Java
LDAP Admin    http://ldapadmin.sourceforge.net/    Win32 Native EXE
phpldapadmin    http://phpldapadmin.sourceforge.net/    Web server PHP
Directory Administrator http://diradmin.open-it.org/    Linux + BSD   C
GQ    http://sourceforge.net/projects/gqclient      Linux + BSD   C

Of the above, it would appear that GQ and Directory Administrator are no longer actively maintained. GQ is current in its architecture, but Directory Administrator does not fully support recent releases of OpenLDAP. The other tools are currently active projects.

There are many other choices in LDAP directory browsers. But as useful as they are to the administrator who manages LDAP directories that span a complex array of services and purposes, the use of a more work-flow or task-oriented tool is often preferred.

There are three tools that outshine the rest where it pertains to managing a Samba-3+ LDAP environment:

Name    Source URL      OS Platform      Language
LAM (LDAP Account Manager)    http://lam.sourceforge.net/      Web server      PHP
IMC (Idealx Management Console) http://imc.sourceforge.net/      Web server      Perl

The LDAP Browser can be used to view and edit directory information, as is shown in Figure 8. The LDAP Browser lacks the facility to perform standard account operations without exposing information that a human resources manager or a departmental head would not need to know.

Figure 8: LDAP Browser showing a user account

Compared to the LDAP Browser, the LDAP Admin tool for MS Windows skillfully bridges both worlds. It looks and feels like LDAP Browser, but has added utilities that are more task-focused. Figure 9 provides a basic overview of what this tool looks like; but in Figure 10 you can see how it adds the nice touch of removing the complexity of having to deal with specific LDAP entities when adding a user account. It has this same smart touch for all management tasks.

Figure 9: LDAP admin basic view

Figure 10: LDAP admin create new user account

Although these tools look neat, they are oriented towards the more technically-competent directory administrator. There are a few tools that are designed to remove the appearance of being for the LDAP guru and that have a stronger task orientation. One example is LAM, the LDAP Account Manager. An example showing a partial list of users is given in Figure 11.

Figure 11: LDAP account manager user list

On the other hand, if you are using Samba-3 + LDAP, and have chosen to use the Idealx smbldap, Perl-based scripts, to permit Samba to interface to with the LDAP directory, you may choose to use the Idealx Management Console with the SambaConsole plug-in. Figure 12 presents a typical IMC console.

Figure 12: IMC user account console

Commercial utilities

It is good to see the number of companies that are providing tools and utilities to ease and facilitate the deployment and management of Samba-3. If your interest is in finding commercially-supported and/or commercial tools, check the Samba Website's vendors section and the GUI section.

Looking for more tools of convenience? Check out the following Web sites: QCD Interstructures; Vintela; and Centeris.

Conclusion

A detailed overview has been provided demonstrating how Samba-3 can form part of a fully integrated network management infrastructure. The series of articles now completed has shown how Samba-3 user and groups and Windows clients can be fully integrated. The documentation explained how security identifiers are handled across the disparate platforms, how user rights and privileges are implemented using Samba-3, the use of basic command-line management and configuration tools, how to create a basic smb.conf file for a PDC and how it is possible to manage the whole show using some simple as well as sophisticated GUI tools. Enjoy!

Managing Samba (Part 5 of 6): Configuration with the net utility, part two

2010-10-27T17:37:00.001-06:00

In part one of this Samba-3 Management tip, we prepared for the big act. Now, the excitement begins. We're ready to use the net utility in the final steps in configuration of the primary domain controller.

Up to this point, no user account has been granted Windows network administrative rights and privileges. Our objective is to give the account mstone full administrative rights. This is simply achieved by making mstone a member of the Linux managersgroup. The managers group is mapped to the Windows Domain Admins group. However, life is not that simple. By default, the Domain Admin group has not rights other than to assign rights and privileges. This means that specific privileges must be assigned even to the Domain Admins group.

Create an administrative user account

Let's verify that mstone is a member of the managers group within the Linux environment:

root#> id mstone
uid=1001(mstone) gid=100(users) groups=100(users),1001(managers)

Now we must demonstrate that within Samba mstone is a member of the Domain Admins group:

root#> net rpc group members "Domain Admins" -S violetsblue -Umstone%n3v3r2l8
ROSESARERED\mstone

Good, mstone is a member of the Windows Domain Admins group. This is achieved by way of the mapping we established by executing:

root#> net groupmap modify ntgroup="Domain Admins" unixgroup=managers

Assign rights and privileges to the domain admins group

In this step, the Domain Admins group is assigned (given, or granted) all administrative rights:

root#> net rpc rights grant "Domain Admins" \
SeMachineAccountPrivilege \
SeTakeOwnershipPrivilege \
SeBackupPrivilege \
SeRestorePrivilege \
SeRemoteShutdownPrivilege \
SePrintOperatorPrivilege \
SeAddUsersPrivilege \
SeDiskOperatorPrivilege -S violetsblue -Umstone%n3v3r2l8
Successfully granted rights.

Make the PDC a domain member

The next step is to make our PDC a member of its own domain. This step requires domain administrative privilege which mstone has. Execute the following:

root#> net rpc join -Umstone%n3v3r2l8
Joined domain ROSESARERED

It is a good practice to validate every step, as we have done so far. The domain trust account that was created by joining the domain can appear to proceed correctly, but it may not work. This can be checked simply by executing:

root#> net rpc testjoin
Join to 'ROSESARERED' is OK

Let's run a further check to see obtain the status of the domain environment:

root#> net rpc info -S violetsblue
Domain Name: ROSESARERED
Domain SID: S-1-5-21-3169455399-2908770435-3209857667
Sequence number: 1135058837
Num users: 2
Num domain groups: 4
Num local groups: 0

So far, so good!

Create additional users

So far, the net command has been used to:

map Linux groups to Windows groups;
check Windows group membership;
join the PDC to its own domain;
validate the domain account (join); and,
check domain information (note: not dependent on the join).

In the last step, we confirmed that there are only two Windows user accounts and four Windows group accounts.

Let's add accounts for the users misty, jable, dstornton using the remote management net tool:

root#> net rpc user add misty -S violetsblue -Umstone%n3v3r2l8
root#> net rpc user add jable -S violetsblue -Umstone%n3v3r2l8
root#> net rpc user add dstornton -S violetsblue -Umstone%n3v3r2l8

The use of the net rpc group add facility results in Samba calling the add user script to add the account to the Linux account database (/etc/passwd), followed by addition to the passdb backend (tdbsam) specified in the smb.conf file.

Unfortunately, these accounts do not yet have a password. We must rectify that at once:

root#> net rpc password misty secretpw1 -S violetsblue -Umstone%n3v3r2l8
root#> net rpc password jable secretpw2 -S violetsblue -Umstone%n3v3r2l8
root#> net rpc password dstornton secretpw3 -S violetsblue -Umstone%n3v3r2l8

If the password secretpw1 is not added to the command line, this tool will prompt for the password to be entered. It looks like this:

root#> net rpc password misty -S violetsblue -Umstone%n3v3r2l8
Enter new password for misty: XXXXXXXX

Now let's add misty to the group scientists:

root#> net rpc group addmem scientists misty -S violetsblue -Umstone%n3v3r2l8

It is possible to add the other new members. We can add a new group called warriors by executing this command:

root#> net rpc group add warriors -S violetsblue -Umstone%n3v3r2l8

Let's add misty so she will be a member of the new warriors group:

root#> net rpc group addmem warriors misty -S violetsblue -Umstone%n3v3r2l8

To remove misty from the warriors group, just use the delmem operator, as shown here:

root#> net rpc group delmem warriors misty -S violetsblue -Umstone%n3v3r2l8

Assign user rights

Often, it is necessary to give a user certain limited administrative privileges. An example is making it possible for a normal user to manage printing operations. In this case misty is assigned the printer management capabilities:

root#> net rpc rights grant "ROSESARERED\misty" SePrintOperatorPrivilege \
-S violetsblue -Umstone%n3v3r2l8

Assigned rights can be examined as shown here:

root#> net rpc rights list accounts -S violetsblue -Umstone%n3v3r2l8
BUILTIN\Print Operators
No privileges assigned
BUILTIN\Account Operators
No privileges assigned
ROSESARERED\Domain Admins
SeMachineAccountPrivilege
SeTakeOwnershipPrivilege
SeBackupPrivilege
SeRestorePrivilege
SeRemoteShutdownPrivilege
SePrintOperatorPrivilege
SeAddUsersPrivilege
SeDiskOperatorPrivilege
BUILTIN\Backup Operators
No privileges assigned
BUILTIN\Server Operators
No privileges assigned
ROSESARERED\misty
SePrintOperatorPrivilege
BUILTIN\Administrators
No privileges assigned
Everyone
No privileges assigned

Wrapping up

The net utility permits very extensive remote management of a Samba server. So far, I have demonstrated how this tool can be used to join a Samba server to its domain, add/delete/change user and group accounts, map Linux groups to Windows groups, add users to groups, and so on. The use of this tool to assign rights and privileges has also been briefly touched upon.

The use of this command is well documented in The Official Samba-3 HOWTO and Reference Guide in chapter 12. The latest version of this document is available from Samba.org. This document is also available from Amazon.com in hard copy under ISBN No: 0131882228.

The series continues

This is the fifth article in my Managing Samba series. Articles in this series have so far explained:

Windows network identity basics and their use in Samba-3;
Windows NT/200x user rights and privileges in Samba 3.0.11 and later;
Domain control parameters and operating system interface scripts in the Samba smb.conf file.
The pdbedit utility to manage domain and user account policy settings.

The next article will deal with remote GUI management tools and facilities. It will review various GUI tools that can be used to facilitate network management. Of course, some will quickly point out that if this can be made simple enough, it should be possible to delegate many day-to-day operations to senior user staff and thus reduce the cost of keeping the network operational.

Managing Samba (Part 5 of 6): Configuration with the net utility, part one

2010-10-27T17:37:00.000-06:00

The net utility plays a key role in completing the configuration of a Samba-3 domain controller or activating and managing a Samba-3 domain member server.

This tip discusses the practical use of the net command. It can be used for initial configuration as well as for on-going system maintenance of Samba-3 domain security environments. My key objective is to demonstrate the assignment and use of network administration tasks using a non-root account. This capability was first available with Samba-3.0.11 and is surely ready for prime-time use. The examples shown here provide the key to safe deployment of Samba using samba-3.0.21.

Pre-flight check

Before launching into the use of the net utility, it is best to quickly check that you have installed a valid smb.conf.master file. A modified version of the file I used previously is shown in Figure 1. This file can be converted into the operational smb.conf file by executing:

root#> testparm -s smb.conf.master > /etc/samba/smb.conf

Always check the validity of the file before starting Samba. The recommended way to do this is by executing:

root#> testparm
Load smb config files from /etc/samba/smb.conf
Processing section "[homes]"
Processing section "[printers]"
Processing section "[netlogon]"
Processing section "[profiles]"
Loaded services file OK.
Server role: ROLE_DOMAIN_PDC
Press enter to see a dump of your service definitions
...

Before starting Samba, it is a good idea to check that there are no tdb files on the system and that may be a left-over from a previous execution of the smbd and nmbd deamons.

Samba-3 will create tdb files in the following directories:

Red Hat Linux: /etc/samba, /var/cache/samba

Never remove the tdb files from a working Samba server unless you really do know what you are doing. The /etc/samba/secrets.tdb file contains essential security data that includes the domain SID. The /etc/samba/passdb.tdb file contains the SambaSAMAccount information. Our example network uses the passdb backend = tdbsam method of storing Windows network account information.

You can remove old Samba run-time files after first stopping the nmbd and smbd daemons by executing:

Red Hat Linux:

root#> service smb stop && service nmb stop
root#> rm /etc/samba/*tdb /var/cache/samba/*

Start the Samba smbd and nmbd daemons using the appropriate method for your Linux platform. On a Red Hat Linux execute:

root#> service nmb start && service smb start

Sites that use an LDAP directory to store all account information require more detailed configurations that are beyond the scope of this article series; however, the net command can be used in the same manner as is described below. This tool is independent of the account back end and simply calls the scripts that are specified in the smb.conf file for all host environment dependent operations.

There are yet a few more things that must be completed before your start can move on. The objective of this series of tips is to end up with a working Samba domain controller. This requires the creation of user and group accounts on the PDC. There are two sides to these accounts: the Linux system account; and the SambaSAMAccount.

The first article in this series described the relationship between the system account and the Windows networking accounts (or SambaSAMAccounts).

The last tip in this series, which covered pdbedit, stepped through the account creation process. If you followed each step, your system should now have a number of Linux user and group accounts, each of which has been configured to have an equivalent Windows networking account.

So that all steps are covered in this tip, I will briefly, and without explanation, create the accounts that are necessary for the use of the net command. The steps that are necessary for this are shown in Figure 2.

Note: Some Linux distributions do not permit addition of a user (machine) account with the home directory specified as /dev/null. If your platform is one of these it will be necessary to change the add machine script parameter in the smb.conf file to use a permitted value.

All right! You've laid the foundation. Now, in part two of this tip, we'll get down to the final steps in configuration of the primary domain controller.

Managing Samba (Part 4 of 6): The pdbedit utility, part 2

2010-10-27T17:36:00.000-06:00

The pdbedit tool in Samba-3 is the only one that can manage account security and policy settings. As it was said in part one of this tip, having control of account security and policy settings is important to businesses that must comply with the Sarbanes-Oxley Act of 2002. Now, I'll show you how to put pdbedit to work creating Linux system user and group accounts; assembling Windows group accounts, thus mapping Windows group accounts to Linux group accounts; adding Windows user accounts; and establishing network access policies and controls.

First, let's get on with the establishment of Linux system user and group accounts.

Two group accounts will be created, one for scientists, and one for managers:

root #> groupadd scientists
root #> groupadd managers

Two user accounts will be created here: one for Tom Bryant (scientists), and one for Melinda Stone (managers). Both users require a home directory. The following steps will create these accounts:

root #> useradd -m -c "Tom Bryant" -G scientists -g users tbryant
root #> passwd tbryant
New Password: XXXXXXX
Re-enter New Password: XXXXXXX
root #> useradd -m -c "Melinda Stone" -G managers -g users mstone
root #> passwd mstone
New Password: XXXXXXX
Re-enter New Password: XXXXXXX

Both accounts are specifically created to be primary members of the users group, and secondary members of the respective groups to which they also belong.

Creation of Windows group accounts

Linux group accounts must now be mapped to Windows group accounts. The following commands will map the key Windows domain groups to local Linux system accounts:

root #> net groupmap modify ntgroup="Domain Admins" unixgroup=managers
root #> net groupmap modify ntgroup="Domain Users" unixgroup=users
root #> net groupmap modify ntgroup="Domain Guests" unixgroup=nobody

The next commands will create Windows groups that will match the Linux group accounts:

root #> net groupmap create ntgroup="Scientists" unixgroup=scientists type d

Addition of Windows user accounts

The following steps create the SambaSAMAccount entries in the passdb backend that was chosen in the last article (tdbsam). The use of the pdbedit tool will demonstrate the account information that can be managed.

root #> pdbedit -a tbryant
new password: XXXXXXX
retype new password: XXXXXXX
root #> pdbedit -a mstone
new password: XXXXXXX
retype new password: XXXXXXX

Both accounts have now been added. This is demonstrated here:

root #> pdbedit -Lw
tbryant:6001:0E74DBD7B1C3CC88AAD3B435B51404EE:72BBEAA844F6E0E9CE546CE7E61C963F:[U ]:LCT-439530A0:
mstone:6002:744F505BEE319723AAD3B435B51404EE:5F7D844D278CD6898B4022AE766CCA1A:[U ]:LCT-43953197:

SambaSAMAccount information can be displayed using the pdbedit tool:

root #> pdbedit -Lv tbryant
Unix username: tbyrant
NT username:
Account Flags: [U ]
User SID: S-1-5-21-726309263-6128913605-1168186429-13000
Primary Group SID: S-1-5-21-726309263-6128913605-1168186429-1201
Full Name:
Home Directory: \\violetsblue\tbryant
HomeDir Drive: H:
Logon Script: scripts\logon.bat
Profile Path: \\violetsblue\profiles\tbryant
Domain: ROSESARERED
Account desc:
Workstations:
Munged dial:
Logon time: 0
Logoff time: Mon, 18 Jan 2038 20:14:07 MST
Kickoff time: Mon, 18 Jan 2038 20:14:07 MST
Password last set: Mon, 05 Dec 2005 23:37:11 MST
Password can change: Wed, 07 Dec 2005 23:37:11 MST
Password must change: Thu, 19 Jan 2006 23:37:11 MST
Last bad password : 0
Bad password count : 0
Logon hours : FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF

The use of the tdbsam backend account database makes it possible to override any of these settings.

Had the smbpasswd backend database been used, it would not be possible to change these settings, because this database does not store the advanced attributes of the SambaSAMAccount data structure.

It is possible to set a forced password change for just one user in the following:

root #> pdbedit --pwd-must-change-time="2006-01-31" \
--time-format="%Y-%m-%d" tbryant
Unix username: tbryant
...
Password last set: Mon, 05 Dec 2005 23:37:11 MST
Password can change: Wed, 07 Dec 2005 23:37:11 MST
Password must change: Tue, 31 Jan 2006 00:00:00 MST
...

In many cases, it is more desirable to create a global network access policy than to be required to set each account separately. The global setting must be in place before user accounts are created, otherwise the old settings will continue to be used. The per-user settings override the global settings.

In the following example, a global policy will be implemented. The policies that can be set include:

min password length
password history
user must logon to change password
maximum password age
minimum password age
lockout duration
reset count minutes
bad lockout attempt
disconnect time
refuse machine password change

In our example, stringent Sarbanes-Oxley compliance settings will be implemented.

root #> pdbedit -P "min password length" -C 8
description: Minimal password length (default: 5)
account policy "min password length" value was: 5
account policy "min password length" value is now: 8

The policy was set to eight characters. Security conscious sites may want to set this to 14 characters:

root #> pdbedit -P "password history" -C 5
Length of Password History Entries (default: 0 => off)
account policy "password history" value was: 5
account policy "password history" value is now: 5

In the example above, the last five passwords will be remembered. This means that an old password can be reused only after six unique passwords have already been used. We want a minimum password age of 45 days (3888000 seconds).

root #> pdbedit -P "minimum password age" -C 3888000
description: Minimal password age, in seconds (default: 0 => allow immediate password change)
account policy "minimum password age" value was: -1
account policy "minimum password age" value is now: 3888000

A password can be changed only after 45 days:

root #> pdbedit -P "maximum password age" -C 3888000
account policy "maximum password age" description: Maximum password age, in seconds (default: -1 => never expire passwords)
account policy "maximum password age" value was: -1
account policy "maximum password age" value is now: 3888000

Passwords must be changed every 45 days. This means that an old password can be re-used only after 270 days. If the minimum password age is left at the default, a smart user can simply enter six new passwords and then reset the original password in rapid succession, thereby defeating the controls.

root #> pdbedit -P "bad lockout attempt" -C 3
description: Lockout users after bad logon attempts (default: 0 => off)
account policy "bad lockout attempt" value was: 0
account policy "bad lockout attempt" value is now: 3

The setting above ensures that the user account will be locked out after three failed logon attempts. This is an important means by which password cracking attempts may be thwarted.

Where an LDAP password back end is used, the policy settings must be exported to the LDAP directory. This can be done by executing:

root #> pdbedit -y -i tdbsam -e ldapsam

It must be noted that the full capabilities mentioned in this article were stabilized in the Samba-3.0.21 release. Where Sarbanes-Oxley compliance is required please use the most recently released stable version.

The use of the Samba pdbedit tool has been shown to be a simple matter. Once you have mastered it, you'll be ready to move on to the next phase of mastering Samba management: the practical use of the net command, both for initiation configuration as well as for on-going system maintenance. That's the topic of the next episode in the Managing Samba series.

Managing Samba (Part 4 of 6): The pdbedit utility, part 1

2010-10-27T17:35:00.000-06:00

Administrators who want to make the most of Samba-3 need to know their PDCs (Primary Domain Controllers) . If you're migrating from Windows to Linux, or remotely managing Samba-3, knowing how to prepare and use Samba-3 PDC is a must.

Remote management of Samba is easily enabled by use of appropriate configuration settings. To achieve proper usage, you have to prepare the Samba-3 PDC for use. This article discusses the first steps, those that involve the pdbedit command. Detailed examples will be provided to demonstrate how this tool can be used to set password expiration limits, reconfigure the location of a users home folder, the location of the users roaming profile, and so on.

Additionally, the article will explain how to disable roaming profiles for all or some users, and will provide guidelines for Microsoft (MS) Windows workstation configuration to bring sanity to network management.

Before we get started, let us consider again some key facts regarding the way the Samba integrates with the server operating system that is hosting it.

Samba interaction with the host operating system

A few simple facts must be stated. There are many postings to the Samba mailing lists in which the author is perplexed that certain commands have failed to produce the behavior that was expected.

Samba implements MS Windows file and print services without overriding Linux system security. What does this mean? It means that:

MS Windows users and groups must map to local operating system UIDs and GIDs.
Therefore, Samba does not create Linux (POSIX) user and group accounts, but passes this responsibility off to interface scripts.
All file system access is limited by the normal Linux security constraints that apply to the mapped UID/GID information. Where in the smb.conf file share stanza parameters are specified to force a particular action. For example: force user, force group, force create mode, etc. Basic Linux file system access controls remain in effect.
Samba tools and utilities do not interfere with Linux system administration. The tools never add user or group accounts, and that is why the administrator can specify scripts for all such operations. If the scripts are not specified in the smb.conf file, Samba will reject an attempt to add a use for which there exists no POSIX (system) account. Samba tools do not interfere with file and directory access controls.
Because Samba must deal with the semantics that are used by MS Windows networking, it provides a method by which network access can be controlled. The pdbedit tool is the primary command-line tool by which these control may be affected.

The pdbedit tool

The pdbedit tool can only be used by root. It is used to manage the passdb backend, and domain-wide account policy settings. In general, pdbedit can be used to:

list user and group accounts
migrate user, machine and group accounts
manage account policies
manage domain access policy settings

Under the terms of the Sarbanes-Oxley Act of 2002, American businesses and organizations are mandated to implement a series of internal controls and procedures to communicate, store and protect financial data. This legislation has far-reaching implications, with respect to the following issues:

who has access to information systems that store financial data
how personal and financial information is treated among employees and business partners
how security vulnerabilities are managed
security- and patch-level maintenance for all information systems
how information systems changes are documented and tracked
how information access controls are implemented and managed
audit ability of all information systems in respect of change and security
disciplinary procedures and controls to ensure privacy.

In short, the Sarbanes-Oxley Act of 2002 is an instrument of law that demands accountability in regards to business-related information systems, so as to ensure the compliance of all information systems that are used to store personal information and, particularly, for financial records processing. Similar accountabilities are being demanded around the world.

The need to be familiar with the Samba tools and facilities that permit information systems operation compliance with government laws and regulations is clear to all. The pdbedit utility is currently the only Samba command line-driven tool that provides the capacity to manage account, systems access controls and policies. During the remaining life-cycle of the Samba-3 series, it is possible the new tools may be implemented to aid in this important area.

The pdbedit tool is the lone device that can manage account security and policy settings. It is capable of superset of the old Samba smbpasswd capabilities. One particularly important purpose of the pdbedit, is to allow the migration of account information from one passdb backend to another.

So, let's get down to brass tacks, or brass code. In part two of this tip, I'll show you how to create Linux system user and group accounts; create Windows group accounts; map Windows group accounts to Linux group accounts; add Windows user accounts; and establish network access policies and controls.

Continue to part two of this tip.

Managing Samba (Part 3 of 6): Configuration for remote management, part two

2010-10-27T17:34:00.000-06:00

You tasted basic Samba configuration in part one of this tutorial. Now, let's get into something much meatier: Samba PDC (primary domain controller) configuration.

The following smb.conf file contents demonstrate the use of the key parameters as well as a few additional ones required to round out a Samba primary domain controller. The key point to observe is the settings for the interface scripts. The example uses the passdb backend = tdbsam, and this means that this PDC is not -- I repeat, not -- suitable for use with a BDC (backup domain controller).

A PDC that has the configuration file shown in this example is a subset of the configuration that is depicted in the book, Samba-3 by Example, second edition. The extracted example does not make use of WINS (the Windows Internetworking Name Server) for NetBIOS to IP address name resolution and eliminates a few more refinements. In this article, my objective is to demonstrate a more basic configuration. The example file is fully functional, but it lacks some of the bells and whistles that can be used to create a more stable environment. The latter Samba environment can be obtained by carefully following the examples in the book.

The [homes] share is properly called a meta-service and is used to dynamically create a user home folder share from the Unix system account information.

The [printers] share is another example of a meta-service. In this case, it defines the generic Unix directory into which a print job will be spooled until it is ready to be sent to a Unix system printer by way of a standard operating system print command. Where CUPS (the Common Unix Print System) is used, Samba will pass the print job directly via a built-in CUPS library interface.

The permissions on the /var/spool/samba directory should be set by executing:

chmod 1777 /var/spool/samba
chown root /var/spool/samba
chgrp users /var/spool/samba

The [netlogon] share is necessary for domain controller operation. The logon.bat script specified in the smb.conf file must be stored in DOS file format in a directory called scripts inside the /var/lib/samba/netlogon directory.

The [profiles] directory will permit the storage of Windows roaming profiles. The permissions on this directory are important and should be set so that the default group can write to the directory. An example of suitable configuration is:

chown root /var/lib/samba/profiles
chgrp users /var/lib/samba/profiles
chmod 2775 /var/lib/samba/profiles

As domain users log onto and then out of the Windows workstations that are domain members, the users profiles will be written automatically to the directory /var/lib/samba/profiles/'username'.

[global]
workgroup = ROSESARERED
netbios name = VIOLETSBLUE
passdb backend = tdbsam
username map = /etc/samba/smbusers
enable privileges = Yes
log level = 0
log file = /var/log/samba/%m.log
max log size = 50
add user script = /usr/sbin/useradd -m '%u'
delete user script = /usr/sbin/userdel -r '%u'
add group script = /usr/sbin/groupadd '%g'
delete group script = /usr/sbin/groupdel '%g'
add user to group script = /usr/sbin/usermod -G '%g' '%u'
set primary group script = /usr/sbin/usermod -g '%g' '%u'
add machine script = /usr/sbin/useradd -s /bin/false -d /dev/null '%u'
logon script = scripts\logon.bat
logon path = \\%L\profiles\%U
logon drive = h:
logon home = \\%L\%U
domain logons = Yes

[homes]
comment = Home Directories
valid users = %S
read only = No
browseable = No

[printers]
comment = SMB Print Spool
path = /var/spool/samba
guest ok = Yes
printable = Yes
use clientdriver = Yes
default devmode = Yes
browseable = No

[netlogon]
comment = Network Logon Service
path = /var/lib/samba/netlogon
guest ok = Yes
locking = No

[profiles]
comment = Profile Share
path = /var/lib/samba/profiles
read only = No
profile acls = Yes

Before starting Samba's smbd and nmbd daemons, the contents of the smb.conf file can be validated by executing testparm. This is a very useful tool to verify that there are no critical errors or typos.

It may be useful to rename smb.conf file to smb.conf.master, and then let the testparm utility generate the smb.conf file by executing:

testparm -s smb.conf.master > smb.conf

The resulting smb.conf file will be fully optimized. Additionally, if there are any significant errors, these will be reported as the file is created.

If all went well, go ahead and start Samba. Check the documentation for your operating system platform for information regarding how to start the smbd and nmbd daemons. Alternatively, refer to chapter 1 of the book, The Official Samba-3 HOWTO and Reference Guide, 2nd Edition, for further Samba start-up information.

Operation of Samba can be validated by executing:

#> smbclient -L localhost -U%
Domain=[ROSESARERED] OS=[Unix] Server=[Samba 3.0.21]
Sharename Type Comment
--------- ---- -------
netlogon Disk
profile Disk
IPC$ IPC IPC Service (Main Server)
ADMIN$ IPC IPC Service (Main Server)
hp940 Printer HP DeskJet 940c
Cups-PDF Printer Cups-PDF
Domain=[ROSESARERED] OS=[Unix] Server=[Samba 3.0.21]
Server Comment
--------- -------
VIOLETBLUE Main Server
Workgroup Master
--------- -------
ROSESARERED VIOLETBLUE

If you've made it this far, then the Samba server is operative and it should now be possible to administer it remotely. Congratulations!

If you want to keep reading, go to the next article by clicking here.

Managing Samba (Part 3 of 6): Configuration for remote management, part 1

2010-10-27T17:31:00.000-06:00

Samba is the bridge between Linux and Microsoft worlds. Samba-3 can help administrators create transparent interoperability of Microsoft Windows client with a Unix or Linux server environment. Even so, seamless interoperability is not easy to achieve, but mastering some Samba configuration tasks helps. So, let's get started with this tutorial on smb.conf.

Use the configuration parameters in the Samba smb.conf configuration control file to permit the transparent management of Samba using tools that are native to the Microsoft Windows environment.

By the way, this is the third installment in a series on Managing Samba. The first article in this series explains how Samba handles Windows security identifiers. The second tip describes user rights and privileges in the Microsoft Windows networking environment.

Anatomy of a Samba configuration

To appreciate the complexity involved in creating transparent interoperability of a Microsoft Windows client with a Unix or Linux environment, open up a system shell on your Samba host server and execute this command:

testparm -sv | more

You will soon discover that there are over 300 parameters that are capable of being set in the smb.conf configuration file. While it may be tempting to discuss every one of them, this is not necessary. Even a complex Samba configuration can usually be accomplished using fewer than 30 specific parameters in the [global] section of the smb.conf file.

The parameters fit into a set of functional categories. The functional categories determine how Samba will interact with Windows networking protocols, how Samba will interact with the underlying Unix file system and how Samba will interface with the host operating system.

In this next section, I'll list some terms that summarize commonlyused configuration parameters.

Base parameters

netbios name - sets the machine name (i.e. the computer name). If not set, the name of the host server will be used
netbios aliases - can be used to present the Samba server under multiple computer names
workgroup - sets the workgroup name (the same as an NT4 domain name)
realm - is applicable only when Samba is an Active Directory domain member
interfaces - instructs Samba which network interfaces to use -- when used with bind interfaces only = Yes instructs Samba to provide file services (TCP ports 139 or 445) only on the specified interfaces. NetBIOS name resolution services (UDP ports 137 and 138) will work on all interfaces.

Lightweight Directory Access protocol (LDAP) controls

LDAP controls are only necessary when Samba is run as a primary domain controller/backup domain controller (PDC/BDC) combination. Basic mandatory controls when using an LDAP directory to store Samba account credentials include:

ldap admin dn - defines the administrative DN needed to write to LDAP
ldap suffix - defines the base directory information tree
ldap [user,group,machine,idamp] suffix - sets the container in which various accounts records are stored

Logging parameters

The default parameter values are usually adequate and normally used for debugging. Commonly used parameters include:

log level - sets the diagnostic intensity level.

Typical values are:
0 - Record only critical errors
1 - default
3 - For debugging simple file system access problems
5 - For complex system diagnostics
10 - Permit detailed diagnosis of authentication errors

log file - specifies the name of the log files.

Example: /var/log/samba/%m.log - creates a separate log per client

max log size - sets size limit at which Samba should rotate the log file. The file is renamed to oldfile.log and a new file is created.

System interface scripts

The system interface scripts are called when an external application requires management of host operating system environment information. An external application can be the NT4 Domain User Manager or the Samba net utility. When a Windows client is joined to the domain, the add machine script is called to create the Unix system account that is needed so that the appropriate SambaSAMAccount entry can be created to store the machine trust account information.

Examples of necessary actions include user/group account manipulation, add/edit/delete shares, add/edit/delete printers, call system initialization scripts, etc. The common parameters needed are as follows:

Account management: add group script, add machine script, add user script, add user to group script, delete group script, delete user from group script, delete user script, set primary group script, rename user script
Note: Where LDAP is used the scripts specified should be able to manage the directory contents. Where LDAP is not used the normal U system account management tools can be specified. For example: add user scriptl= /usr/bin/useradd -m '%u'.
Resource management: add share command, change share command, delete share command, add printer command, delete printer command, abort shutdown script, shutdown script

Samba operating mode parameters

Let's move on to Samba operating mode parameters:

domain logons - when set to Yes, Samba will enable the network logon service. This creates the domain control capabilities needed for a PDC/BDC.
enable privileges - when set to Yes, enables assignment of rights and privileges. Note: only Samba-3.0.11 and later.
logon drive - lets the drive that Windows NT4/200x/XP clients will associate with the execution of network logon scripts. This is the drive to which the home directory will be mounted if net use /home is executed on a Windows client, either in a command shell or from the logon script.
logon home - used only by Windows 9x/Me clients.
logon script - the logon script that is executed when a user logs onto a domain member client.
passdb backend - specifies which backend system Samba will use to store SambaSAMAccount information. The default is smbpasswd, the two common alternatives are tdbsam and ldapsam. Both the ldapsam and the tdbsam types are capable of storing the advanced SambaSAMAccount information. The smbpasswd method is not capable of storing the advanced account information.
security - this parameter sets the Samba server mode. Acceptable settings are:
user - means Samba is either a standalone server or a domain controller.
domain - means Samba is an NT4-style domain member.
ads - means Samba is an Active Directory member (uses Kerberos authentication and security, but only as a member -- not as an ADS domain controller.
server - means that Samba is not a domain member, but is using an external authentication server. Uses pass-through authentication that has been largely obsolete.
domain master - Leave this set at the default value on all systems except a BDC. On a BDC, this parameter must be set to No.

There are many additional parameters that are useful for specific purposes. Refer to the smb.conf online main page for detailed information regarding each parameter.

For the most part, the setting of unnecessary parameters in the smb.conf files is unnecessary and potentially problematic. It is very good advice to keep the smb.conf file as simple as possible.

Now that your appetite is whetted for Samba configuration, I've got a new treat for you. Move on to part two for an example of a Samba PDC configuration. I'm sure you'll find it very satisfying.

Click here to read part two of this tip.

Managing Samba (Part 2 of 6): User rights and privileges

2010-10-27T17:30:00.000-06:00

When faced with making Linux and Windows work together securely, system administrators can run into trouble if they don't understand or can't correctly use the administrative security controls that are present in the Microsoft Windows operating system (OS) and the Samba-3 equivalents of those controls.

So, let's take a look at the tools administrators should be familiar with and the rights and privileges that can be managed.

This is part two of a six-part series, titled Managing Samba. In part one, I covered Windows network identity basics. Stay tuned, as the next article in this series will review the process of establishing a Samba configuration that is able to be securely managed using traditional Windows administrative tools.

Why user rights and privileges matter

Many people get confused by the fact that, by design, Windows NT/2000/XP provides the capacity to restrict access to almost every part of the operating system, yet the facilities that permit files and directories to be secured through use of restrictive access controls are frequently ill-used, if they're used at all.

Microsoft Windows XP Home Edition is commonly configured so that the default user has full administrative privilege and thus bypasses most native operating system security provisions. It is, therefore, no wonder that virus and worm infections can march forward with the utmost pace and force.

Many users expect to have the ability to create accounts and to install software without constraint. My objective is not to discuss the defense mechanisms against virus and worm activity, but to focus on administrative rights and user capabilities.

Windows XP Home Edition does not provide the ability to manage file system access controls. In every respect therefore, the dominant operating system in the consumer market place, Windows XP Home Edition, provides little to no security control capability.

Windows XP Professional provides the capability to manage access control lists (ACLs) on file system resources, but few people learn how to manage them effectively. Despite this, ACLs are a valuable asset in a networked environment and are a common part of Windows security domains. Windows XP Professional also permits the assignment of administrative tasks to users and groups. These facilities are of significant importance in the creation and maintenance of a secure networking environment.

A key reason for the exercise of constraint in corporate networks is the protection of data and, in particular, the protection of confidential data. In the U.S., the Sarbanes-Oxley Act of 2002 is used to enforce restrictive data access in many industries.

Samba-3 is transparent in respect of file system security controls. So long as the underlying file system of the server on which Samba is being hosted has ACL support and, providing that Samba-3 has been compiled to support the handling of ACLs, Windows administrators and users will have the ability to manage ACLs on exported file systems on the Samba-3 server.

Windows domain membership and security

Machine (computer) accounts are used in the Windows NT OS family to store security credentials for domain member servers and workstations. A computer account is created when a Windows client joins a Windows NT4 domain, or when it is made a member of a Microsoft Windows 200x Active Directory domain.

As part of the startup process of a domain member machine, it goes through a validation procedure that includes an exchange of credentials with a domain controller. If the domain member fails to authenticate, using the credentials known for it by domain controllers, the machine will be refused all access by domain servers.

Within the context of a domain logon, the computer account is essential to the way that Microsoft Windows secures authentication. The computer account is used only to authenticate domain member machines during the startup, a security measure designed to block man-in-the-middle attempts to violate network integrity.

Windows Domain Administration Controls

By default, the ability to add machines to the domain is limited to the Windows domain administrator account. The right for other users or groups to add machines to the domain can be granted by the administrator using the appropriate tool.

The Windows NT4 server management tool can be used to manage domain member machines (servers and workstations). The right to add workstations to the Windows NT4 domain can be granted using the Windows NT4 domain user manager. This particular privilege that must be assigned to a user or group so that they can add machines to the domain is called SeMachineAccountPrivilege. The title in the NT4 domain user manager rights management panel is called Add workstations to domain. The list of available privileges that can be assigned in Windows NT4 is shown here:

Additional Windows privileges were added in Windows 2000 and are also available in Windows 2003 and Windows XP Professional. These may be found in Figure 2.

The Windows 200X privileges are a super-set of those available with Windows NT4. Some of the privileges present in the Windows NT4/200X environments are of unique relevance to those operating systems and have no direct equivalent in a UNIX- or Linux-based Samba environment.

The privileges that make sense within a Linux environment have been implemented in Samba-3.0.x since the 3.0.11 release. These are summarized below.

In comparing the privileges that are present in Samba with those supported by Microsoft Windows NT4/2000/2003, those found in Samba are also supported in Windows server products.

Windows administrative privileges can be assigned to Windows users and Windows groups, thereby permitting relatively fine-grained assignment of administrator privileges on an as-needed basis. For example, where it is desirable that a Windows user called joebartemy should be able to add Windows clients to the domain, his account can be assigned the SeMachineAccountPrivilege thus preventing this user from gaining any other administrative capabilities. Likewise, members of the Windows group Marketing Staff can be given the ability to manage printers and print jobs by granting this group the SePrintOperatorPrivilege.

Allocating administrative privileges

Samba versions prior to 3.0.11 required the use of a UNIX account that has UID=0 (the root account) for a number of core administrative operations. Operations that require this level of privilege within the Unix operating system environment include management of user and group accounts, change of file ownership, print job management and so on. Older versions of Samba provided work-around methods to permit some of these essential tasks to be completed. Those that could not be side-stepped simply had to be done by the root account, or its equivalent.

Samba-3.0.11 and later allow the creation and administration of a Samba server without needing to resort to the use of the root account. Users who are members of the Domain Admins group automatically have the necessary privileges to manage all aspects of Samba. Those who need to be assigned specific roles can be given the minimum level of access necessary to perform the tasks they are responsible for. Someone who has been given the SeMachineAccountPrivilege can add machines to the domain and those who have been given the SePrintOperatorPrivilege can now manage printers, print jobs and more.

Management of Windows administrative privileges

Samba-3 provides a command-line tool for the management of user rights and privileges. The use of the net utility, as well as the administration of user rights and privileges are documented in the Samba3-HOWTO book, and correct has also been documented in the Samba3-ByExample book.

An example of use of the net command to assign the ability to add machines -- Samba servers, Windows servers and clients -- to the Samba domain for the domain MIDEARTH and the user jerry is shown here:

net rpc rights grant "MIDEARTH\jerry"
SeMachineAccountPrivilege -S PDC -
Ujtadmin%password

This operation can also be performed by using the Windows NT4 Domain User Manager. This handy utility may be obtained from the Microsoft support Web site. A more recent version of this tool is also available.

An example of use is demonstrated in Figure 4.

The steps that lead up to this Windows panel are:

Log onto Windows XP Professional as a user who has administrative rights.
Launch the UsrMgr.exe.
Click on the Policy tab.
Click on the User Rights entry.

Another tool that can be used to manage user rights is a commercial package called Hyena. The latest at the time of preparation of this article was version 6.5.

The next article in this series will document the configuration of a Samba domain controller so that user rights and privileges can be managed using the tools we have reviewed here.

Managing Samba (Part 1 of 6): Choose your weapon

2010-10-27T17:27:00.000-06:00

Samba has become an integral part of a large portion of the networking marketplace. Despite its prevalence, not all IT shops currently utilize the facilities provided to fully secure the network or to drive operating costs to a minimum.

Samba-3 has grown in capability, as well as in complexity. Linux administrators are keen to learn how better to manage Samba and how to meet the growing demand of network security and audit-ability. This series of articles will help administrators master Samba-3 management. In this first article in a six-part, two-month series, you'll find a description of the series and a tutorial on Windows network identity basics.

Introduction

I recently asked a few Windows networking administrators what they believe is needed to help drive down the cost of using Samba. They all said better management tools would be a great help.

After further discussion with these administrators, it became clear that while they were comfortable with Samba, they were unclear where Samba stores user, machine and group information, and how these relate to the network environment. Had the relationships been better understood, the functionality of available administration tools might be better understood and the request for better tools might not be so strong.

This series of articles will provide a better understanding of the relationship between Windows networking accounts and their equivalent on the Unix or Linux server that is running Samba.

I've described the topics in the series below. You can also click here to go directly to page two for the tip on Windows network identity.

Series overview

This series will cover the following subjects:

Windows networking identity basics: The first article explores the ways in which Microsoft Windows users gain access to the operating system and how this affects access to network resources. It will also explore what a Samba server must do to handle incoming network connection requests from Windows workstations. Specifically, the article will show how Samba deals with identity issues, where it stores the data it uses and how each component of Windows networking identities can be managed. The article concludes with basic rules that make it possible to operate the Samba environment with a minimum of administrative overhead.
A comparison of user rights and privilege management in Windows and Samba: The second article will explain how Windows workstation users gain, or can be denied, various local and network-wide rights and privileges. It then will examine how this can be implemented in a Samba environment.
The creation of a network-wide remote management infrastructure: The next article focuses on how Samba can be configured to create an environment that is conducive to centralized network management. It will provide information on where to obtain detailed technical guidance, so that the necessary infrastructure can be implemented without undue difficulty.
Use of the pdbedit utility to manage user accounts and to configure network security settings: The pdbedit utility has become a strategically important tool for managing Windows network user accounts. It is also important to sites that need to comply with the requirements of the Sarbanes-Oxley Act. The article will discuss how this tool can be used to set password expiration limits and reconfigure the location of a user's home folder or the location of the user's roaming profile, and so on. Additionally, the article will explain how to disable roaming profiles for all or for some users and will provide guidelines for Windows workstation configuration to bring sanity to network management.
The use of the net utility to perform remote network management: The article will show how the net utility can be used to perform many essential network management and control operations. This versatile tool can be used to manage users, groups and group membership for Windows networking users. It can also be used to migrate servers and server resources across various Windows and Samba servers.
The use of various remote management tools: This closing article will examine the leading solutions to the remote network management challenge. An overview will be provided covering the NT4 SRVTOOLS, the LDAP administrator, the IMC console, the LDAP Administration Manager (LAM), and a commercial tool (yet to be chosen).

Continue to the next article for the tip, Managing Samba: Windows networking identity basics.

A collection of the top Linux command tips

2010-10-27T15:35:00.002-06:00

As a sysadmin, you’ll need to use the command line instead of the GUI from time to time. Do you know how to monitor what’s running on your Linux server through the command line? Can you confidently unlock the copying prowess of the dd command? Brush up on your command-line skills with this collection of expert advice -- a list of the 8 top Linux command tips.

Using grep and sed to find and replace
Using the Linux commands grep and sed can help search for a string of text within a file and replace strings.

Linux/Unix commands: How to get copy, conversion power with dd
The Linux command dd handles convert-and-copy tasks, and aids in cloning the hard drive, among other tasks.

How to manage software on Ubuntu Server with "aptitude" and "apt-get"
In this tip, learn how to use aptitude and apt-get to perform Linux package management.

Keep an eye on Linux processes with ps
By using the command-line tool ps, you can monitor what’s running on your Linux server.

Samba management: Configuration with the net utility, part 1
The first installment of a tip series that explains using the net utility to configure the primary domain controller and manage a server in Samba.

Samba management: Configuration with the net utility, part 2
Part 2 of a tip that explains using the net utility to configure the primary domain controller and manage a server in Samba.

Understanding Linux system performance management using top
Using the Linux command top can provide sysadmins with system performance data. Learn how to analyze these top results, including data such as load average values.

Analyzing Linux write performance problems with iotop
There could be write performance problems on your Linux system, and the iotop Linux command may be more detailed than top in getting to the root of the performance issues.

Be sure to also check out this comprehensive guide to 77 useful Linux commands and utilities, coming soon !!

Analyzing Linux write performance problems with iotop

2010-10-27T15:03:00.001-06:00

A common performance problem for servers is writing data. When the server writes too much data, the storage sub-channel isn't able to catch up. But, it can be difficult to discover the performance problems, so here we'll walk through how to analyze performance issues on Linux servers.

The problem with write performance-related issues, is that often they hide behind other parameters. In all cases however, the top command is a good place to start. The wa parameter in the CPU line indicates the amount of time that your system has been waiting for the I/O-channel. Typically, this indicates a slow storage channel.

A high value at the wa parameter indicates that the storage channel is suffering

Just looking at top however is not good enough. Let's have a look at a small test. In this test, we have written the current memory state to disk, using dd if=/dev/kcore of=/kcore.img bs=4096. On a 4 GB machine, that is a lot of work for the machine to do. While performing the job, everything else really slows down, which means there is a performance problem.

But, the problem with top is that it's not really easy to see the write performance problem. It all depends on the amount of CPU cores you have. On a 16-core server, the write problem may claim all CPU cycles on one CPU. You would however not see that from the generic top overview as this gives you the average for all CPU's together. So with a CPU core that is completely claimed by writes, you wouldn't see much more than 6% at the wa parameter in top. To get more detail, the first thing to do, is to press the 1 key in the top interface, which gives you a line for each CPU core. On the test system used for writing this article, there are two cores only, so the results are not spectacular. But, on a multi-core system the differences displayed may be important.

From top, press 1 to see performance details for each CPU core.

So, if only one core out of 16 cores is completely busy waiting for the slow storage channel, then the 15 remaining cores can do the work, right? Too often, the answer is "no." If you just have one storage channel, then all CPU's need to go over that one single channel. If one CPU is completely busy waiting for a storage channel, then the other cores won't be able to get prompt reactions from the storage channel either. So it may look all right from the top window, but performance could be terribly off.

Fortunately, there is iotop and it gives information about the most active processes with regard to I/O. Most Linux distributions don't install it by default, so make sure you install it manually, using your distributions meta package handler (for instance: yum install iotop if you're using RHEL). The good thing about iotop, is that it shows you which is the most active process with regard to I/O at the moment and how much I/O it is generating. If you compare the I/O load caused by this process with the capacity of your storage channel, you'll know immediately if you have a storage problem and if so, where it comes from. Then you can troubleshoot this issue, and optimize your Linux server write performance.

With iotop you can see exactly what process is claiming your valuable resources.

Understanding Linux system performance management using top

2010-10-27T14:52:00.001-06:00

If there's something wrong with the performance of your Linux server, chances are that you're already using top to find out what's happening. It seems however that few people really know how to tell what their system is doing from the information that top provides. Here I will explain how to understand the performance data that top provides.

When starting top, make sure that you are in root. To start, open a console on your favorite Linux distribution and enter the top command. The result should look similar to this:

host:~ # top
top - 12:41:34 up 1 day, 3:29, 6 users, load average: 0.00, 0.00, 0.00
Tasks: 99 total, 1 running, 98 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.1%us, 0.1%sy, 0.0%ni, 99.7%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 775064k total, 560056k used, 215008k free, 136216k buffers
Swap: 136544k total, 0k used, 136544k free, 275624k cached
   PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8830 root 15 0 2232 936 692 R 2 0.1 0:00.03 top
   1 root 16 0 728 284 244 S 0 0.0 0:01.25 init
   2 root RT 0 0 0 0 S 0 0.0 0:00.10 migration/0
   3 root 34 19 0 0 0 S 0 0.0 0:00.34 ksoftirqd/0
   4 root RT 0 0 0 0 S 0 0.0 0:00.04 migration/1
   5 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
   6 root 10 -5 0 0 0 S 0 0.0 0:00.33 events/0
   7 root 10 -5 0 0 0 S 0 0.0 0:00.10 events/1
   8 root 11 -5 0 0 0 S 0 0.0 0:00.01 khelper
   9 root 12 -5 0 0 0 S 0 0.0 0:00.00 kthread
   13 root 10 -5 0 0 0 S 0 0.0 0:00.13 kblockd/0
   14 root 10 -5 0 0 0 S 0 0.0 0:00.08 kblockd/1
   15 root 13 -5 0 0 0 S 0 0.0 0:00.00 kacpid
   16 root 13 -5 0 0 0 S 0 0.0 0:00.00 kacpi_notify
   227 root 20 0 0 0 0 S 0 0.0 0:00.00 pdflush
   228 root 15 0 0 0 0 S 0 0.0 0:00.93 pdflush
   229 root 17 0 0 0 0 S 0 0.0 0:00.00 kswapd0

The first part of relevant information that top provides can be found in the first line: the load average parameters. These describe how busy your computer is at the moment. The average workload of your server is always given in three digits. Each represents the load average for the last minute, the last five minutes and the last fifteen minutes. You should always start by interpreting these numbers, as they tell you if your system is overloaded or not.

To understand the load average values, you must relate them to the number of CPU's or CPU cores in your computer. If you're not sure, just press the 1 button when the top interface is active, this will give you a line for each CPU core that is present in your computer. When a CPU core has been completely busy in the last minute, top will show you 1.00 if it's a one core system. If you have eight cores installed in your computer, and one has been completely busy, while the others were doing nothing, top will show you 0.125 as the value in the load average. In order to interpret the value in the load average lines, you need to know the normal value for your server. For instance, on a four-core machine, that would be 4.00. Anything above that value is bad, as it indicates that queuing occurs and processes are waiting for their slice of system time. Anything below this value is good. If your system is getting beyond the ideal value for that system, the next step is to determine what exactly is happening. Listing 2 gives an example of a one-core system that is too busy:

top - 12:49:38 up 1 day, 3:37, 6 users, load average: 1.37, 0.34, 0.11
Tasks: 101 total, 4 running, 97 sleeping, 0 stopped, 0 zombie
Cpu(s): 7.1%us, 16.7%sy, 0.0%ni, 6.2%id, 67.3%wa, 0.4%hi, 2.4%si, 0.0%st
Mem: 775064k total, 767664k used, 7400k free, 514236k buffers
Swap: 136544k total, 0k used, 136544k free, 102744k cached
   PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8859 root 18 0 1788 524 448 R 46 0.1 0:21.25 cat
8860 root 18 0 1792 524 448 R 41 0.1 0:18.83 cat
  229 root 15 0 0 0 0 S 9 0.0 0:01.54 kswapd0
3695 root 16 0 1864 700 616 S 3 0.1 0:16.00 hald-addon-stor
4580 root 16 0 95916 14m 11m S 3 1.9 0:36.44 main-menu
4552 root 15 0 101m 24m 17m S 2 3.2 0:13.44 nautilus
   13 root 10 -5 0 0 0 S 1 0.0 0:00.70 kblockd/0
4270 root 15 0 146m 12m 5716 S 1 1.7 1:01.66 X
4536 root 15 0 29012 9688 7896 S 1 1.2 0:05.47 gnome-settings-
4578 root 16 0 18136 5500 4124 S 1 0.7 0:14.87 gnome-power-man
8861 root 16 0 2236 1036 780 R 1 0.1 0:00.64 top
   14 root 10 -5 0 0 0 S 1 0.0 0:00.49 kblockd/1
4575 root 15 0 95736 13m 10m S 1 1.8 0:07.39 application-bro
3131 root 16 0 4692 3244 1444 S 1 0.4 0:34.57 hald
4586 root 15 0 93868 11m 9796 S 1 1.6 0:06.62 mixer_applet2
  228 root 15 0 0 0 0 S 0 0.0 0:00.95 pdflush

If the workload is getting too high, you need to find out what is happening. To do this, you have to look at the CPU line(s). You will see no less than eight different parameters, and of these, only three really matter. First is the "us" parameter. This indicates the amount of time your system is busy handling requests that were made in user space. If a task is not in user space, it is a high-privileged task that runs in system space, which you can see reflected in the "sy" parameter. In kernel space, processes can communicate directly to the drivers. Therefore, you should worry more if your system gives a high load in system space. The third parameter that is important in the CPU line is "wa." This stands for waiting, and indicates the amount of time your system waits for I/O-devices. A high parameter here indicates a problem on the I/O-channel, normally this is a hard disk that is too slow or a misconfigured network.

The second listing example shows that the system is way too busy waiting for I/O. This is far too common, many times system performance problems are related to slow I/O devices. One solution is to install a faster hard drive, but before doing that, it is a good idea to check the BIOS of your server and see if there are parameters that you can tune. One of the most important candidates for that, is the write cache parameter. By writing data to write cache before writing it to the disk platters themselves, you can dramatically reduce waiting times. Since write cache is about 1,000 times as fast as the hard disk, chances are that you can win a lot by enabling this feature.

Use top to reveal memory efficiency on Linux servers

Apart from the information on how busy your system's CPU is, top also shows you how memory-efficient your server is. You can find information about this in the lines that start with Mem: and Swap:. Let's start discussing swap. This is RAM that is emulated on the hard drive that your computer should never use. There are some exceptions though: if your server runs Oracle, SAP or any other specific application that is built to use swap. But normally, Linux starts swapping only if it is totally out of normal memory. In an exception, your server could pre-allocate some swap so that it can use it faster if it's needed. But in most cases, you should install new RAM on a server that starts swapping.

After you have verified that your system isn't swapping, you should find out what it is doing with available memory. To understand memory, you should know that Linux uses memory quite efficiently. If there's no real need for memory to service processes, it will be used as read cache or write buffers. The read cache contains files that were recently read from your computer's hard drive. The kernel just keeps them in RAM, because you might need them again and if you do, it's a lot faster to serve these files from RAM than from hard disk. The write buffers on the other hand, are used as a waiting room for your server's hard drive. Instead of offering data directly to the hard drive, the operating system places them in the write buffers where they can wait until the hard disk decides it has time to flush these write buffers (e.g., writes them to disk). This also gives you a performance benefit.

The nice thing about read cache and write buffers is that the operating system can make them available instantaneously when it needs memory. Therefore, you should add the read cache and write buffers to the total amount of available memory. A nice way of doing this, is by using the free -m command. On the +/- buffers/cache line, you can see how much free memory your computer really has.

host:~ # free -m
   total used free shared buffers cached
Mem: 756 746 10 0 595 40
-/+ buffers/cache: 110 646
Swap: 133 0 133

As you can see in this listing, at first sight it looks as if this server almost has no more available memory, but if you know that buffers can be flushed immediately, you can see that it has largely enough available memory.

Determining active processes on a Linux server

The last interesting part of top is where it shows the most active process on your server. This is not hard to determine: the most active process is listed first on the process list. If this process uses too much system resources, top offers some options to handle it. You can terminate it by pressing the k-key from the top interface. Top will then ask you what signal you want to send to this process. You should always try signal 15 first, this represents the nice way of asking the process to please stop its activity. If that doesn't work, use signal 9, which just terminates the process without further waiting.

Another way of taming a process, is by "renicing" it – i.e., adjust the priority the process is using. To do this, press the r-key from the top interface. By giving a process a negative nice value, you increase the priority with regard to other processes. By assigning a positive nice value, you give more room for the other processes. The values you can use are between -20 and 19. It's a good idea not to assign the value of -20. By doing this, you would give the highest possible priority to a process, thus allowing it to leave no time for the other processes (if it is a busy process).

Managing Samba (Part 1 of 6): Windows network identity basics

2010-10-27T11:10:00.001-06:00

In heterogeneous IT environments, Samba has to deal with two entirely dissimilar environments: the Unix or Linux host on which Samba is being run as a service and a Windows network that is made up of workstations and servers. Understanding Windows networking identity basics can make Samba management a lot easier, and this article -- part one of a series – offers my insights on that subject.

Microsoft Windows workstations users practice the browsing of network file and print resources as the natural way to achieve daily tasks, yet few are aware of how this works. The ability to browse network resources via the Microsoft Windows Explorer is an expected capability. When this ability is impaired, the user's frustration is readily elevated. What a user considers as simply the process of browsing is comprised of a large number of complex processes, all of which must function correctly so that the user can experience expected behavior.

The browsing of network resources is possible by way of creating:

Special broadcast announcements made by Windows network clients and servers.
Machines -- selected through an election mechanism and called browse masters -- on the network that listen for the special broadcast messages.
Name resolution processes that permit the translation of a NetBIOS machine name to a TCP/IP address.
Establishment of a connection to a Windows file share.
All these mechanisms must work in harmony; otherwise, the expected user experience is lost.

The establishment of a connection is the vital step that enables access to a remote file. The Windows client provides a user name and password as part of the data request to create the network connection. The user name and password may both be empty (the null account), or the password may be empty (null password).

The use of the null account is common when creating a connection to the IPC$ share on a target Windows networking server like a Samba server. The IPC$ share is used by the network client to obtain the list of resources (shares and printers) that are available on the server. When a resource has been selected, a connection is created between the client and the server to enable the resource (share or printer) to be used by the network client. The connection request is refused if the credentials that are provided by the client are refused by the server.

This article is primarily concerned with the behavior of a Samba server and does not deal with Microsoft Windows network servers.

Samba authentication

The authentication methodology that is used by a Samba server depends on how it has been configured, as well as whether or not Winbind is in use. If Winbind is in use, smbd, the Samba process that performs all network connection handling, will ask Winbind to process the authentication request for the network connection setup request. If Winbind is able to perform authentication, handling it will return to the smbd process that called it either a pass or fail.

If Winbind is not able to resolve the validity of the network user's credentials or is not available -- for example, it is not running -- the responsibility for determining whether or not the network connection request should proceed is handled by smbd. In this case, smbd will attempt to look up the user in the passdb back end that has been specified in the smb.conf file. If the smb.conf file does not specify a passdb back end, smbd will use the default passdb back end (smbpassswd). The user name and password are validated against the encrypted credentials that are stored in this back-end database.

As I said earlier in this article, Samba has to deal with two dissimilar environments: the Unix or Linux host, where Samba runs as a service, and the Windows network of workstations and servers. Samba must obey the rules that are imposed by both environments. The authentication of Windows network operations requires access to the Microsoft encrypted passwords that are stored in Samba's own password back end.

Samba refers to this as part of the Samba security account manager (Samba SAM) data. The SAM information database (passdb back end) contains all the Windows-specific account information that is not used by the Unix/Linux environment.

User/group name and security identifier handling

When the connection to the share has been established, further access to shared network resources requires an user identifier (UID) and a group identifier (GID) on the Unix or Linux host server for the remote Windows user. This requires fully functional forward and reverse name to Unix or Linux identifier resolution. This requirement exists because a Unix or Linux host requires a UID and GID pair to determine access rights that are applicable to the remote Windows network user for all Unix file system access operations.

The windows user has the capacity to modify file security settings and access controls using the Windows Explorer. For this to function correctly, there needs to be a reliable and reproducible method for resolving user names to the matching Windows system identifier (SID). The reverse is also necessary. The Windows user SID is critical to the use of a tool, such as Windows Explorer, for the handling (setting and reporting) of file and directory security settings. This requires the ability to map a UID or a GID from the Unix/Linux file system to a unique Windows SID, and demands a consistent mapping of the Unix/Linux user name or group name to a Window user name or group name. The mapping must be unambiguous.

Every Microsoft Windows operating system has a unique machine security identifier (the machine SID). Likewise, a Microsoft Windows security domain also has its domain SID. User and group accounts that are created within either context are assigned a sequential security identifier that is relative to the security context. This identifier is known as the relative identifier (RID) and is added to the end of the SID for the security context within which it has been created. This means that when a user account is created for a user called adaniel on a collection of Windows NT/200x/XP machines, each account will have its own unique user SID. The user SID thus being made up of the machine SID concatenated to the user RID.

Windows accounts (user, group, machine, trust, etc.) are assigned a RID. This is done automatically as an account is created. Samba produces the RID algorithmically. The Unix operating system uses a separate name space for the UIDs and GIDs, but Windows allocates the RID from a single name space.

A Windows user and a Windows group cannot have the same RID. Just as the Unix user root has the UID=0, the Windows administrator has the well-known RID=500. The RID is concatenated to the Windows domain SID, so administrator account for a domain that has the above SID will have the user SID S-1-5-21-726309263-4128913605-1168186429-500. Outside of the well-known accounts that have a globally consistent RID, the RIDs are allocated starting at RID=1000. The RID is appended to the SID of the machine with the result that every account in the Windows networking world will have a globally unique security identifier.

In the Windows environment, user, group and trust accounts are all allocated a sequential number from the same name space (the RID). In the Unix/Linux environment, the UID and GID are independent entities providing each its own name space.

Samba obeys the security constraints that are normative to the host operating system on which it is running, as well as that of the Windows clients it serves. This means that the translation (or mapping) of Windows users and groups, Windows SIDs, Unix UIDs and GIDs, and Unix account names must be possible in a manner that is entirely unambiguous at all times. This is an important matter that Unix administrators often overlook.

Samba SAM information location

Information that is specific to the local Samba server is stored in the file /etc/samba/secrets.tdb. The domain SID, as well as the local server SID, is stored in this file. When Samba-3 is used with the LDAP passdb back end, the LDAP administrator password is also stored in the secrets file. When Samba is a domain member server, the domain membership account secret password is also stored in the secrets file.

Where the tdbsam passdb back end is used, user account information is stored in the (default) file called /etc/samba/passdb.tdb and contains the extended Samba SAM account information that can also be stored in an LDAP directory in the event that the ldapsam passdb back end is chosen.

Samba-3 can make concurrent use of multiple passdb back ends, making Samba-3 highly scalable, thereby permitting it to meet the needs of small, as well as very large, sites.

Unix groups can be made available to Windows users by creating a group mapping. The group mapping is part of the Samba SAM account information data and is stored either in the file /var/lib/samba/group_mapping.tdb (when using tdbsam) or in an LDAP directory (ldapsam).

User rights and privilege information, together with domain access controls, are stored in the file /var/lib/samba/account_policy.tdb. ACL access controls on shares are stored in the /var/lib/samba/share_info.tdb.

As of Samba version 3.0.21, the account policy information will be stored in the LDAP directory when the primary passdb back end specified is the ldapsam. Samba versions earlier than 3.0.21 did not replicate policy information via the LDAP directory, thus making it necessary to manage policy settings on all domain controllers. Version 3.0.21 automatically replicates policy information to all domain controllers (as happens with Windows NT4).

The tdb files store binary hashed data, the contents of which can be dumped to a plain ASCII format using the tdbdump utility. For example, the content of the secrets.tdb file can be exposed by executing:

linux:/etc/samba # tdbdump secrets.tdb
{
key = "SECRETS/SID/LINUX"
data =
"\01\04\00\00\00\00\00\05\15\00\00\000\DE\06\F4\FB\C0\AD\D3d\D7\1E\9C\00
\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00
\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00"
}

Summary

This article has begun the process of untying the knot that protects those secrets of Samba domain administration. It has been established that the security identifiers differ greatly on Samba host platforms from that of the Microsoft Windows world. A brief explanation has been given to show how Samba manages security identifier interoperability.

The next article will compare Windows and Samba user rights and privilege management. Now that some of the darker secrets regarding Linux and Windows security identifiers have been covered, we are ready to move on to explore the tools that can be used to manage make efficient use of them.

Click here to go back to the introduction to this series.

Keep an eye on Linux processes with ps

2010-10-26T16:04:00.001-06:00

On Microsoft Windows, the Task Manager is responsible for reporting on and managing the running processes and applications on a server. Linux servers have a Task Bar (or equivalent) when they are viewed from desktops such as GNOME and KDE, but the main tools for process management are command-line ones. Here we'll discuss how to see what's running on a Linux server.

On Linux, processes have owners. Any user can view all the current processes, but only the root user has full management of those processes. On Linux every process runs in its own memory space and can't accidentally overwrite other processes. A process that crashes or dies due to a bug also can't ruin any other process. It's still possible that a process may cope poorly with the loss of another process on which it depends. The Linux kernel itself is not a process; it is special. It is the program that brings into being a playpen inside of which all processes must play.

The central tool used to display processes is the command line tool "ps". Type "man ps" for full details. "ps" stands for "process status". When run, it queries the Linux kernel for the current state of the machine, and based on the snapshot returned, outputs a formatted report. The plainest use of ps shows processes owned by the current Linux user.

In the normal output, CMD states the name of the program that the process is a currently running example of, and TIME is the hours, minutes and seconds of CPU time consumed by that program during that run. PID is the unique identifier for that process.

To see the processes owned by another user, try:

ps -u fred

where "fred" is a Linux /etc/passwd user name. For more detail, the long (-l) or full (-f) options can be used; but the same processes are discovered. Be careful concatenating options; "ps -lufred" is "ps -l -u fred", but "ps -ulfred" is "ps -u lfred"

Each process has both an identifying process ID (a PID), and a parent process ID (a PPID). The parent process is the process that was used to "spawn" (to start) the other process. Usually, the PPID is a command-line shell such as ksh or bash, but it can also be a server such as httpd or inetd.

PID numbers start at one and go up, so processes started more recently have higher numbers. If a process has PPID=1, that means it was either started by "init," the first process that runs when Linux starts up, or (rarely) it has partially failed and has lost its parent process. Such part-failed processes are called "zombies".

A very common use of ps is the command:

ps -ef

which reports every process running, regardless of owner. Also, ps output is highly customizable; the -o option can be used to report exactly the columns you want. Be sure to set and export the COLUMNS environment variable if you want a very wide report, eg:

COLUMNS=300; export COLUMNS

Finally, the -H option, usually used with -e, shows all the processes stacked in a Windows Explorer-like hierarchy. This gives you visual cues (indenting) showing relationships between PPIDs and PIDs.

Running ps hundreds of times a second is bad for general performance, because the kernel must spend time reporting on itself. To watch the process list change in real time, most Linux versions include the program 'top'. You'll find that top acts like ps repeated at fastest once a second. It sorts the output so that the most CPU-intensive options are, well, at the top. The sort order is configurable. Beware that top (and many other programs) expect that you are using a quality terminal emulation in your command window. Usually xterm is enough. Microsoft Window's simple telnet client is too basic. Try installing the free Windows tool PuTTY if you must telnet to Linux from Windows.

Finally, ps and top do their job by exploiting a special interface to the kernel. Many GUI and command-line tools exist that also exploit this interface. For a quick hands-on assessment of a server's condition, simple ps is usually the safest way to go.

How to manage software on Ubuntu Server with "aptitude" and "apt-get"

2010-10-26T15:35:00.001-06:00

If you are used to RPM-based Linux-distributions such as Red Hat and SUSE, it is quite a shock getting used to package management on Ubuntu Server. The most difficult part is the layered organization of Ubuntu package management services. Nevertheless, in many ways Ubuntu offers a much easier way to manage packages based on the Debian package format. Understanding how Ubuntu Package management works allows you to do your work as a Ubuntu Server administrator much more efficiently, and even update your system in one single command. In this article we'll explain how it works.

Ubuntu uses the same package management solutions as Debian. Package management on Ubuntu Server is based on a database that keeps track of all software packages that are installed. The most fundamental of all is the dpkg-database, which is based directly on the Debian package management utility. On top of that sits the apt database which is stored in /var/lib/apt. To manage packages in this database, you can use the "apt-get" command or the "aptitude" command. Both do more or less the same. If a graphical user interface is installed, the Synaptic utility can be used as well. In this article you'll read how to work with these tools.

Managing software repositories

Apart from the database that manages packages that are installed on your server, there is also the software repository as a second fundamental component that you need for package management. These repositories are basically lists of installation sources. One such installation source is the cd that you have probably used to install Ubuntu Server. Other sources are added automatically when installing Ubuntu Server. A list of all installation sources is found in the configuration file /etc/apt/sources.list.

Since these sources refer to Internet sites as well as local installation sources, they help you in keeping your server up to date. The "apt-get" command always uses this list of sources before installing new software. To make sure that the list is up to date, you have to use the "apt-get update" command. This makes sure that the indexes that your server keeps locally are updated so that they know about new software packages that have become available. Once the indexes are updated this way, you can use "apt-get upgrade" to upgrade packages on your server to their latest versions or "apt-get install package-name" which will install the most recent version of the package that you want to use.

Managing packages with "aptitude"

Another command that works on the same package database, is "aptitude". This is not so much a command that you would use to install or upgrade packages, but more a command that is used for package management. You would use this command, for example, to search for packages that match a given keyword, e.g.: "aptitude search xen". This displays a list of all packages that match the description xen. The command would give a result like in listing 1.

Listing 1: Searching packages with "aptitude"

user@host:~$ aptitude search xen
p ubuntu-xen-desktop
   - Xen software for running on servers.
p ubuntu-xen-server
  - Xen software for running on servers.
p xen-doc-2.6.16
  - Linux kernel specific documentation
p xen-docs-3.0
  - documentation for XEN, a Virtual Mac
v xen-headers
v xen-headers-2.6
p xen-headers-2.6.16
  - Header files related to Linux kernel
p xen-headers-2.6.19-4
  - Common header files for Linux 2.6.19
p xen-headers-2.6.19-4-gener
  - Common header files for Linux 2.6.19
p xen-headers-2.6.19-4-serve
  - Common header files for Linux 2.6.19
v xen-hypervisor
v xen-hypervisor-3.0
p xen-hypervisor-3.0-i386
  - The Xen Hypervisor for i386
p xen-hypervisor-3.0-i386-pa
  - The Xen Hypervisor for i386 (pae ena
v xen-hypervisor-i386
v xen-hypervisor-i386-pae
p xen-image-2.6.19-4-generic
  - Linux 2.6.19 image on PPro/Celeron/P
p xen-image-2.6.19-4-server
  - Linux xen 2.6.19 image on x86.
p xen-ioemu-3.0
  - XEN administrative tools
p xen-restricted-modules-2.6
  - Non-free Linux 2.6.17 modules on x86
v xen-source
v xen-source-2.6
p xen-source-2.6.16
  - Linux kernel source for version 2.6.
p xen-tools
  - Tools to manage debian XEN virtual s
v xen-utils
p xen-utils-3.0
  - XEN administrative tools
p xen-utils-common
  - XEN administrative tools - common fi
p xengine
  - A benchmark program for the X Window
p xenman
  - A graphical Xen management tool
v xenx-doc-2.6

From the list of all these packages, you can next use the command "aptitude show" to get more information about a package. For example, use "aptitude show xen-source" to get a description of the xen-source package if it is already installed, or use "apt-get install xen-source" to install it now.

If you don't like to manage packages from the command line, there is good news. The aptitude utility can work in menu mode as well, but to do that you must make sure that the right terminal setting is used. To do this, first type export TERM=vt100. Next run "aptitude" again, and you'll see the interface from Figure 1. From this interface you can check which packages are installed and which are not from an intuitive menu interface.

GUI package management

If managing packets from the console using "apt-get" or "aptitude" is not ideal for you, you might want to do it from a graphical user interface instead. One of the tools that comes with the graphical interface is the Synaptic package manager, which is based on the apt database as well. It offers an intuitive mouse-driven interface to help you install and manage software packages.

In Synaptic, the "Sections" button is a good starting point. This button allows you to see all available software, organized by software category. To see what's inside a category, click it, and in the right part of the Synaptic window you'll see a list of available packages. After clicking an individual package, you'll see a description of the package, allowing you to see exactly what is in the package. Next, select the "Mark for Installation" option and click "Apply." This will bring up the window that you see in Figure 3, asking you if you really want to install this package. From this window, click "Apply" to start the package's installation.

Another useful option from the Synaptic interface is the "Search" feature. It works very easily: Click "Search", and in the window that pops up, select the software that you are looking for. Click "Search" again and you'll see a list with all matching packages in it. Next, mark them for installation and click "Apply" if you really want to use these packages.

Conclusion

For first time users, the Ubuntu server package management utilities are quite intimidating. The most important reason for that is that a layered architecture with related commands is used to manage software on your server. The most important system is based on the apt package database. In this article you have learned how to use the commands "apt-get" and "aptitude" to perform package management from the command line. You've also seen how the Synaptic package manager can make package management easier if you are using a graphical environment.

Linux/Unix commands: How to get copy, conversion power with dd

2010-10-26T15:27:00.001-06:00

System administrators often have to copy data around. Copying and converting ordinary data is easily accomplished with the Linux command called cp. However, if the data is not ordinary, cp is not powerful enough. The needed power can be found in the dd command, and here are some ways to put that power to good use.

The dd command handles convert-and-copy tasks. Obviously, "cc" would have been a better name, but there already was a command with that name when dd was invented. It doesn't matter since it's a cool command.

Cloning a hard drive with dd

The dd command doesn't just copy files; it copies blocks, too. As a simple example, I'll show you how to clone your entire hard drive.

Assuming that /dev/sda is the drive that you want to clone, and /dev/sdb is an empty drive that can be used as the target, using the dd command rather easy:

dd if=/dev/sda of=/dev/sdb

In that example, dd is used with only two parameters: if, which is used to specify an input file, and of, which is used to specify the output file. In this case, both input and output files are device files. Wait until the command is finished, and you will end up with an exact copy of the original hard drive.

In the latter example, the contents of a device were copied to another device. A slight variation to that is the way that dd is used to clone a DVD or CD-ROM and write it to an ISO file. If your optical drive can be accessed via /dev/cdrom, then you can clone the optical disk using:

dd if=/dev/cdrom of=/tmp/cdrom.iso

You can also mount that ISO-file using:

mount -o loop /tmp/cdrom.iso /mnt

Next, you can access the files in the ISO-file from the directory where the ISO is mounted.

Creating a backup of the Master Boot Record

So far, we have used dd to perform tasks that can be done with other utilities as well. Now, we can go beyond that. In the following example, let's create a backup of the Master Boot Record (MBR).

Copy the first 512 bytes of your hard drive, which contains the MBR, to a file. For instance, you could do this with the command

dd if=/dev/sda of=/boot/mbr_backup bs=512 count=1

In this scenario, two new parameters are used:

First, there's the parameter bs=512, which specifies that a blocksize of 512 bytes should be used.
Next, the parameter 1 is used to indicate that only one such block has to be copied. Without that parameter, you would copy your entire hard drive.
The backup copy of your MBR may be useful if, some day, you can't boot your server anymore because of a problem in the MBR. In case that happens, just boot from a rescue disk and use the command:

dd if=/boot/mbr_backup of=/dev/sda bs=446 count=1

In this restore command, only 446 bytes are written back. This is because you may have changed the partition table since you've created the backup. By writing back only the first 446 bytes of your backup file, you don't overwrite the original partition table, which is between bytes 447 and 511.

Extending swap space

In the last example of how dd can save your life, I'll show you how to extend your swap space by adding a swap file.

Let's say that you're alerted at 3 a.m. by a message saying your server is about to run out of memory entirely, due to an undiscovered memory leak. All you have to do is create an empty file and specify that it should be added to the swap space.

Creating this empty file is an excellent task for dd:

dd if=/dev/zero of=/swapfile bs=1024 count=1000000

This would write a file of one gigabyte, and that can be added to the swap space using:
mkswap /swapfile

and

swapon /swapfile

In this article, you've learned how to do some basic troubleshooting using the dd utility. As you have read, dd is a very versatile utility that goes far beyond the capabilities of ordinary copy tools like cp. Its abilities to work with blocks instead of files are especially valuable.

Using grep and sed to find and replace

2010-10-26T15:19:00.001-06:00

This article covers two beneficial Linux tools, grep and sed. If searching for or handling text, grep and sed can increase your efficiency with Linux bash scripting and configuration files. We'll learn how to use these commands and get some helpful tips, too.

Using Grep

Need to find a single matching line in a 300 page script or a large configuration file that you've customized? Don't have the time to search line by line to find that single bug that is halting your system when trying to start the service? You can search for a string quickly using grep.

You can search with grep (an acronym for searching globally for lines matching the regular expression, and print them) many of the files on a Linux machine. When you give it a list of files to read, it will search for lines of text that match the expressions you're looking for and, if it find any, it will output the pieces you want. Just think of as a grip: using grep is a great way to grab a hold of specific lines or files. This command line utility was originally designed for Unix systems.

For example, let's say we have a list of numbers in a file, numlist.txt, and we want to see if the number 8723 is in the list without searching for hours. Using grep can easily find the string in the following command line:

grep –i 8723 numlist.txt

If we get an output of 8723, then it is in the list. If we do not get an output, it is not there. The tool will print the whole line where the text string is found. If the list contained 8723 297r, we would get:

[root@tower root]# grep 8723 numlist.txt 8723 297r
[root@tower root]#

Grep is case sensitive by default. To disable this, use the –i flag.

Here's something useful in a daily RHEL system checkup. Say you need to find out which version of sendmail rpm you installed. To list which rpm packages are installed on the system, you enter the following:

rpm –qa

If you have a system like mine, you have just too many packages to sort through. It might take forever to go through them line by line to find the package you're looking for. Wouldn't it be easier to find a line with a text string matching the word "sendmail?" Thankfully, we can use grep like this:

rpm –qa | grep sendmail

Which should give you:

[root@tower root]# rpm -qa | grep sendmail sendmail-8.13.1-3.RHEL4.5
[root@tower root]#

Don't know if the string is capitalized or not? Just turn off the case-sensitive default:

rpm –qa | grep –i sendmail

Here's another example of how handy grep is. If we want some information on a process that is running on our server, we would run the command ps, which returns a fairly large list of running processes. To find the certain process that we're looking for, put the grep command after ps. Say I need to make sure my DHCP Server is running. I'll run the following:

ps ax | grep –i dhcp

You should see something like the following:

3058 ? Ss 0:00 /usr/sbin/dhcpd -cf /etc/dhcp/dhcpd.conf -q -pf /var/run/dhcp/dhcpd.pid -user dhcp -group dhcp

All set. Our DHCP server is running thanks to the simple and efficient grep.

Using sed

The command sed (stream editor) is used to take a file, read a line, and change the line as specified by the command. This is another former UNIX utility available today in most Linux distributions.

Let's say we have a file called sedtest.txt, which contains the string "1.0". We want to modify this file, changing the string to "2.0". We can do this by doing the following:

sed -e 's/1.0/2.0/g' sedtest.txt > sedtest2.txt mv –f sedtest2.txt sedtest.txt

In this command line, s stands for substitute and g stands for global. Sedtest.txt is the input file (the one we're changing) and sedtest2.txt is the file we've created from the changes of sedtest.txt. The second command takes the output file and renames it to the input file name (replacing the changes made)

Besides substitution, there are other things sed is useful for. For example, the following deletes empty lines or lines that only contain spaces:

sed -e '/^ *$/d' sedtest1.txt

Notice how this one does not need an output file.

Just like the Replace function in Word and Wordpad, sed is the Linux version of this handy tool.

Creating an inventory with nmap network scanning

2010-10-25T15:19:00.000-06:00

Creating a system inventory is a common task for system and network administrators. Information security specialists often need inventories as well. Even in shops that have inventory tools, a method is often needed to validate the inventory and identify systems that the tool may not be aware of.

The open source security audit tool nmap has all the necessary features needed to run ad-hoc or automated inventory processes. However, it is not obvious which parameters and output format are ideal to quickly create an inventory of your environment in a useable format.

Generating usable nmap results

To create a usable inventory, the format of the generated data is crucial. If a report is being shared, a portable format is likely needed. Comma-separated values (CSV) are ideal, as this format can be loaded easily into spreadsheet and database programs. If the report is not shared, then a format that stands on its own must be included with the output. This article will provide example of the CSV generation.

The inventory created by nmap and other network polling tools is a network-based inventory. The inventory created provides information that is critical to system, application and protocol management, such as a system's IP address, its operating system and the applications that it is running on network ports. The inventory will not include information such as how much memory the system has, how many processors there are, etc. This type of hardware inventory requires either an SNMP agent on the system or some other program or script running on the system to determine the hardware in the system.

Creating systems reports

A systems report can be quickly generated by nmap. Simply by scanning a network, an immediate list of systems and their protocols can be seen by using operating system identification (-O) and possibly verbose output (-v) against the complete network (/24):

nmap -O -v 192.168.1.0/24

However, the format is unruly, as pages and pages of output are generated. What is needed is a very clean output that can be easily loaded into a spreadsheet.

Nmap supports the output parameter (-o) to influence how it should write data to standard out. By using it combined with G (-oG), nmap will create output that grep can work easily with, which makes our inventory creation much easier.

Using operating system identification and the “grepable” output formatting, the following command can be used to run the raw reports and output the report to report.txt:

nmap -O -oG report.txt 192.168.1.0/24

This report, however, is still difficult to read and is not easily readable by a spreadsheet program or database, and includes the IP addresses that are not assigned to a system, since it scans the complete network:

cat report.txt
# nmap 5.21 scan initiated Thu Sep 30 08:28:31 2010 as: nmap -O -oG report.txt 192.168.1.0/24
Host: 192.168.1.0 () Status: Down
Host: 192.168.1.3 () Status: Down
Host: 192.168.1.4 () Status: Down
Host: 192.168.1.5 () Status: Down
Host: 192.168.1.1 (router.domain.com) Status: Up
Host: 192.168.1.1 (router.domain.com) Ports: 80/open/tcp//http///, 443/open/tcp//https///, 4567/open/tcp//unknown///, 8080/open/tcp//http-proxy///, 8443/open/tcp//https-alt/// Ignored State:closed (995) OS: Linux 2.4.18 - 2.4.35 (likely embedded) Seq Index: 205 IP ID Seq: All zeros
…

To create a system report that includes IP address, hostname and operating system, we need to focus on the lines containing the information. The easiest way to identify these lines is to search for OS:

grep "OS:" report.txt
Host: 192.168.1.1 (router.domain.com) Ports: 80/open/tcp//http///, 443/open/tcp//https///, 4567/open/tcp//unknown///, 8080/open/tcp//http-proxy///, 8443/open/tcp//https-alt/// Ignored State: closed (995) OS: Linux 2.4.18 - 2.4.35 (likely embedded) Seq Index: 205 IP ID Seq: All zeros
…

We now have the information necessary to create the report. However, a lot of information exists in the entry (such as open ports, sequences, etc.) that is not needed and should be dropped during the report generation. Let's remove Host:, Ports:, and all the ports included; OS:, everything starting with Seq up to the end of the line; and, finally, the first set of parenthesis around the host name:

grep "OS:" report.txt | sed 's/Host: //' | sed 's/Ports.*OS://' | sed 's/Seq.*$//' | sed 's/(//' | sed 's/)//'

The ouput of this command is in this format:

192.168.1.1 router.domain.com Linux 2.4.18 - 2.4.35 (likely embedded)

Now the IP address, hostname and operating system are easily identifiable, and it's a matter of piping that output to awk to add quotes and commas for the CSV. Basically, quotes are added around the IP address, the host name and all the words that make up the operating system identification tag:

grep "OS:" report.txt | sed 's/Host: //' | sed 's/Ports.*OS://' | sed 's/Seq.*$//' | sed 's/(//' | sed 's/)//' | awk '{print "\"" $1 "\",\""$2"\"," $3 " " $4 " " $5 " " $6 " " $7 " " $8 " " $9 " " $10 " " $11 " " $12 " " $13 " " $14 "\""}' >report.csv

Looking at the newly created report.csv file, the output is now in CSV format and can easily be loaded into a spreadsheet:

cat report.csv
"192.168.1.1","router.domain.com","Linux 2.4.18 - 2.4.35 (likely embedded) "
"192.168.1.2","freddy","Linux 2.4.35"
"192.168.1.6","computer.home","Linux 2.4.21 (embedded)|MontaVista embedded Linux 2.4.17 "
"192.168.1.7","IMAC.home","Apple Mac OS X 10.5 - 10.6.1 (Leopard - Snow Leopard) (Darwin"
"192.168.1.8","new-host-3.home","Apple iPhone mobile phone (iPhone OS 3.0 - 3.2, Darwin 10.0.0d3) "
"192.168.1.11","new-host-5.home","Apple Mac OS X 10.5 - 10.6 (Leopard - Snow Leopard)"
"192.168.1.15","AppleTV.home","Apple Mac OS X 10.4.8 - 10.4.11 (Tiger) (Darwin 8.8.0 – 8.11.1)"

Conclusion

Using the output of nmap, combined with the power of grep, sed, and awk, a complete network inventory can be generated in a matter of seconds on a small environment, and in a matter of minutes even in a very large environment.

These same tools can be used by the information security specialist seeking to identify changes such as additions and removal of systems within the environment.