c0t0d0s0.org (Entries tagged as lksf)

Less known Solaris Features: CPU resource Pools

nospam@example.com (Joerg Moellenkamp) — Wed, 16 Jun 2010 09:16:42 +0200

After writing about the processor sets and their role in computing the number of garbage collection threads recently, there were some questions about configuring processor sets. In this article i want to explain the usage of resource pools in Solaris in regard of CPU resources. You don't configure the processor sets directly. Instead of this the pool facility does this for you.

I want to explain how to configure, how to use and how to monitor them. However i can just scratch the surface in such an article.
Continue reading "Less known Solaris Features: CPU resource Pools"

Less known Solaris features - logadm

nospam@example.com (Joerg Moellenkamp) — Tue, 02 Mar 2010 21:19:45 +0100

The mighty root couldn't sleep at night, so root walked around the castle. Deep down in the foundation of super users castle there was a strange room. It was filled with scrolls and some of the serving daemons of root filled this room with even more scrolls while root was standing there. So root looked in some of them: There were new ones, interesting ones. Then root took some old ones, blew away the dust and after a short look root thought "Damned ... those scrolls are so old, there aren't true anymore. Someone has to clean up this place".

So root spoke a mighty spell of infinite power and another daemon spawned from the ether: "You are the keeper of the scrolls. But don't keep everything. Just the last ones." And so it was done since the day of this sleepless night.

Continue reading "Less known Solaris features - logadm"

Less known Solaris features - IP Multipathing (Appendix A): if_mpadm

nospam@example.com (Joerg Moellenkamp) — Wed, 03 Feb 2010 14:20:00 +0100

Of course it's a little bit tedious to zip cables from your switch to test IPMP or to deconfigure an interface to force traffic on different interfaces when you want to do some maintainance work in your network. There is an command to on- and offline physical connections in an IPMP group. It's called if_mpadm.
Continue reading "Less known Solaris features - IP Multipathing (Appendix A): if_mpadm"

Less known Solaris features - Persistant routes

nospam@example.com (Joerg Moellenkamp) — Wed, 03 Feb 2010 07:01:42 +0100

When you work with the interesting features of Solaris, you forget about the small features in commands you use day by day. In my my IPMP tutorial i used an SMF script to make some host routes persistent ... at foremost to show you that SMF scripts are really easy. But, well ... this script isn't nescessary because of the -p option of the route add command.
Continue reading "Less known Solaris features - Persistant routes "

New version of the "Less known Solaris features" ebook

nospam@example.com (Joerg Moellenkamp) — Tue, 26 Jan 2010 11:20:30 +0100

I've just uploaded a new version of the "Less known Solaris features" ebook. As the IPMP tutorial is a part of the new document the page count reaches 379 now. You can download it at the usual location.

Less known Solaris features: IP Multipathing

nospam@example.com (Joerg Moellenkamp) — Fri, 22 Jan 2010 14:51:44 +0100

It isn't really a less known feature for many experienced Solaris admins. On production system they already use IPMP. But many people new to Solaris aren't aware of the existence of an IP multipathing framework in Solaris and Opensolaris. This tutorial consisting out of eleven parts wants to give an brief introduction to this topic. Additionally it covers the new IPMP implementation, that found it's way into the code at build 107. Thus it's interesting for long-time admins as well.

I hope, this tutorial gives you a good insight into this interesting topic.

Less known Solaris features - IP Multipathing (Part 11): Conclusion

nospam@example.com (Joerg Moellenkamp) — Fri, 22 Jan 2010 14:03:55 +0100

The nice thing about IPMP is: It's simply there. You can use it. When you have more than one interface in your system and you care about the availability of your network, it takes you just a few seconds to activate at least the link-based variant of IPMP. This fruit is really hanging just a few centimeters above the ground.
Continue reading "Less known Solaris features - IP Multipathing (Part 11): Conclusion"

Less known Solaris features - IP Multipathing (Part 10): Tips, Tricks and other comments

nospam@example.com (Joerg Moellenkamp) — Fri, 22 Jan 2010 13:31:22 +0100

At the end of this tutorial i want to talk about some tricks, tips and comments in regard of new and classic IPMP.
Continue reading "Less known Solaris features - IP Multipathing (Part 10): Tips, Tricks and other comments"

Less known Solaris features - IP Multipathing (Part 9): Classic and new IPMP compared

nospam@example.com (Joerg Moellenkamp) — Fri, 22 Jan 2010 13:09:04 +0100

We've configured both mechanisms now. Let's summarize what we have seen so far. When you use the new IPMP the data address is bound to its own interface. Whatever happens to the physical interface there are no changes to the binding of interfaces to ip addresses.
Continue reading "Less known Solaris features - IP Multipathing (Part 9): Classic and new IPMP compared"

Less known Solaris features - IP Multipathing (Part 8): Classic IPMP

nospam@example.com (Joerg Moellenkamp) — Fri, 22 Jan 2010 12:40:54 +0100

IPMP itself is a really old feature. It's in Solaris for several versions now. Just the implementation I've described before is a new one. But in Solaris 10 you don't have this new IPMP implementation. Solaris 10 still uses the old implementation. I will call the old implementation classic IPMP. The basic mechanism of new and classic IPMP is pretty much the same: Providing failure detection mechanisms and switch something to do a failover thus the data address stays available. But internally it's a completely different implementation. While the new mechanism is certainly the future of IPMP, I' m pretty sure you will old mechanism more often in the wild.

Continue reading "Less known Solaris features - IP Multipathing (Part 8): Classic IPMP"

Less known Solaris features - IP Multipathing (Part 7): New IPMP and link aggregation

nospam@example.com (Joerg Moellenkamp) — Fri, 22 Jan 2010 10:59:19 +0100

As I wrote before there is another way to protect your system against the failure of a network connection - Link Aggregation. As i've explained before, there are failure modes that can't be addressed by link aggregation. But you can use both in conjunction. This makes sense, when your main connection is a 10GBe interface and you don't want to plug a second one into the system and use already existent 1GBe Interfaces as a backup for it instead.
Continue reading "Less known Solaris features - IP Multipathing (Part 7): New IPMP and link aggregation"

Less known Solaris features - IP Multipathing (Part 6): New IPMP

nospam@example.com (Joerg Moellenkamp) — Fri, 22 Jan 2010 00:06:00 +0100

When you want to try new IPMP, you need a fairly recent build of OpenSolaris. New IPMP was integrated into Build 107 for the first time.

At first: If you have already a working IPMP configuration, you can simply reuse this config. However it yields a different looking, but functionally equivalent result compared to your system with Classic IPMP. This is made possible by some automagic functions in New IPMP. One example is the implicit creation of the IPMP interface with the name of the IPMP group when there isn't already an IPMP interface in the group. However explicit creation should be prefered as you can choose a better name for your IPMP interface and the dependencies are much more obvious.
Continue reading "Less known Solaris features - IP Multipathing (Part 6): New IPMP"

Less known Solaris features - IP Multipathing (Part 5): Prerequisites

nospam@example.com (Joerg Moellenkamp) — Thu, 21 Jan 2010 22:01:06 +0100

At first you need a testbed. In this tutorial I will use a system with three interfaces. Two of them are Intel networking cards. They are named e1000g0 and e1000g0. The third interface is an onboard Realtek LAN adapter called rge0.

Continue reading "Less known Solaris features - IP Multipathing (Part 5): Prerequisites"

Less known Solaris features - IP Multipathing (Part 4): Foundations 3

nospam@example.com (Joerg Moellenkamp) — Thu, 21 Jan 2010 21:32:03 +0100

IPMP vs. Link aggregation

Link aggregation is available on many switches for quite a while now. With link aggregation it is possible to bundle a number of interfaces into a single logical interface.
Continue reading "Less known Solaris features - IP Multipathing (Part 4): Foundations 3"

Less known Solaris features - IP Multipathing (Part 3): Foundations 2

nospam@example.com (Joerg Moellenkamp) — Thu, 21 Jan 2010 21:15:10 +0100

As I wrote before, there are two methods of failure detection. Link based failure detection and probe based failure detection. Both have advantages and disadvantages.
Continue reading "Less known Solaris features - IP Multipathing (Part 3): Foundations 2"

Less known Solaris features - IP Multipathing (Part 2): Foundations 1

nospam@example.com (Joerg Moellenkamp) — Thu, 21 Jan 2010 20:16:45 +0100

The fundamental concepts of both implementations are pretty much the same, thus I will start with a short introduction into the nomenclature of IPMP. I scarified some precision to make it easier to understand, but for our purposes the precision is more than sufficient.
Continue reading "Less known Solaris features - IP Multipathing (Part 2): Foundations 1"

Less known Solaris features - IP Multipathing (Part 1): Introduction

nospam@example.com (Joerg Moellenkamp) — Thu, 21 Jan 2010 20:09:54 +0100

root was sitting in SuperUser castle and everything was fine in the kingdom. But then a loud squeaking and creaking noise found root's attention. The demons of hypertext wrote into the Scrolls of Log, that they couldn't fulfill their work any longer, as the sole bridge into the vast kingdom of root lowered at this moment was broken.
Continue reading "Less known Solaris features - IP Multipathing (Part 1): Introduction"

Less known Solaris features: The curious case of the /tmp in Solaris

nospam@example.com (Joerg Moellenkamp) — Fri, 31 Jul 2009 11:13:00 +0200

This article isn't really about a feature, it's about a directory and its misuse. Furthermore it's an article about different default configurations, that lead to misunderstandings. This is a pretty old hat for experienced Solaris admins (many of them learned it the hard tour), but it seems to be totally unknown to many admins new to the business or for people switching from Linux to Solaris, as many distribution are configured in a different way per default. A reader of my blog just found a 2GB .iso in /tmp on a Solaris system and that's not really a good idea. A few days ago, a user in twitter had vast problems with memory usage on a system which boiled down to a crowed /tmp.

Continue reading "Less known Solaris features: The curious case of the /tmp in Solaris"

(Upcoming) Solaris Features: Crossbow - Part 2: Limiting and Accounting

nospam@example.com (Joerg Moellenkamp) — Sat, 20 Jun 2009 18:05:11 +0200

One of the basic objects in the new Crossbow stack is the flow. Any network traffic is separated into such flows. And with this flows you can do several interesting things. In this article i want to present two usages of them: Bandwidth Limiting and Flow Accounting

Continue reading "(Upcoming) Solaris Features: Crossbow - Part 2: Limiting and Accounting"

Less known Solaris Features ebook - Version 0.5

nospam@example.com (Joerg Moellenkamp) — Sun, 31 May 2009 20:56:34 +0200

After a long time i found time to integrate all the typo and error corrections provided by Ceri Davis into the main LKSF document. Thus i´ve updated the document today. You can download it at the usual location at c0t0d0s0.org. I´ve added the kssl tutorial, too.

Less known Solaris Features: kssl

nospam@example.com (Joerg Moellenkamp) — Fri, 22 May 2009 13:52:00 +0200

There is an interesting feature in Solaris. It is called kssl. One component of this feature is obvious: SSL. So it has something to do with SSL encryption. As you may have already guessed, the k at the beginning stands for kernel. And kssl is exactly this: A proxy to do all the encryption stuff in a kernel module.

This feature is already four years old. I´ve reported about kssl back in December 2005 for the first time. After talking with a customer two days ago and looking at some new material about it, i thought it could not harm to write a new tutorial about this topic. So much appearances in such a small timeframe are a hint .
Continue reading "Less known Solaris Features: kssl "

Less known Solaris Features: About crashes and cores - Appendix B: Live crashdumps

nospam@example.com (Joerg Moellenkamp) — Mon, 29 Dec 2008 22:11:51 +0100

With gcore you can create core dumps of processes without the need to stop them. You can do the same with the crash dumps as well. You can trigger a crash dump without rebooting the system. The command for this task is the savecore command. The normal task of this program is to take the content of the crash dump device and to create files in the regular filesystem to make them persistent for further studies after the reboot of a system.

Continue reading " Less known Solaris Features: About crashes and cores - Appendix B: Live crashdumps"

Less known Solaris features: About crashes and cores - Appendix B: ::status

nospam@example.com (Joerg Moellenkamp) — Sun, 19 Oct 2008 11:30:38 +0200

You are logged into your system and doing a little bit of house keeping (archiving of old logfiles, deleting the rubbish on you system like backup files) and suddenly you see a core file. Heck ... you wasn´t aware of the fact, that an application wrote such a file. How can you get some basic information about it? With Solaris you can use the mdb for this task.

Let´s assume you find a core file at / on the system master:

# uname -a
SunOS master 5.11 snv_97 i86pc i386 i86pc
# ls -l core
-rw-r--r--   1 root     root     5073385 Oct 10 18:26 core

Now we can start the Modular Debugger mdb with the core file.

# mdb core
Loading modules: [ libc.so.1 ld.so.1 ]
>

After a short moment the mdb command will give you a prompt. Now just type in the ::status command

> ::status
debugging core file of sshd (32-bit) from master
file: /usr/lib/ssh/sshd
initial argv: /usr/lib/ssh/sshd
threading model: native threads
status: process core file generated with gcore(1)

Now you know, that you can savely ignore the file, as the system created it on your order while you was playing around with gcore.

My first article has gone live on SDN

nospam@example.com (Joerg Moellenkamp) — Tue, 14 Oct 2008 06:07:45 +0200

With a huge amount of help from Marina Sum the first article from the LKSF book made it to the Sun Developer Network: Introducing pfexec, a Convenient Utility in the OpenSolaris OS.

Less known Solaris features: pfexec

nospam@example.com (Joerg Moellenkamp) — Sat, 20 Sep 2008 20:09:00 +0200

One of my first tutorials was the tutorial about RBAC. In this new tutorial i want to come back to this topic. In the RBAC tutorial i used su to assume a different role. But Solaris offers an additional way to work with the privileges of a different role.
Continue reading "Less known Solaris features: pfexec"

Making an one dot zero out of the LKSF book

nospam@example.com (Joerg Moellenkamp) — Fri, 12 Sep 2008 19:53:13 +0200

I´ve decided to stop the writing of new parts for the LKSF for a while and want to make a version 1.0 of of this document. There is one major showstopper on the way. I had just an mediocre B in English as i found physics, chemistry and biology and micro/macro economics much interesting than languages. I know there are several typos in the document and i assume many errors regarding the grammar. Thus i ask for your helping hands for proof reading the document. To give you something back for your help, i´ve introduced the "Credits/Kudos" section at the end of the book. Well, when you want to help me at this task, just drop me a mail at joerg@c0t0d0s0.org !

PS: The missing apostrophe in "aren´t" or "don´t" are not typos ... it´s an error in the .tex-source i´ve made consistently in the document. I´m writing a perl script for a mass subsitition at the moment to fix this.

LKSF restructured

nospam@example.com (Joerg Moellenkamp) — Sun, 07 Sep 2008 20:27:26 +0200

I´ve uploaded an restructured version of the "Less known Solaris features" pdf book today. Well, the order of the tutorials is still the same, but i promoted each structural unit by one step. So a chapter is now a part, a section is now a chapter. This was nescessary as the new Kerberos tutorial (one of two projects in progress) was in need of a different organisation in this section. As a subsubsubsection wasn´t an option, i took the other way.

BTW: The tutorials about killing zombies and getting command line/environment are in this version, too.

Less known Solaris Features: Getting the command line and environment of running processes

nospam@example.com (Joerg Moellenkamp) — Tue, 02 Sep 2008 08:09:25 +0200

There is a really interesting tool to gather informations from a running process. Let´s assume a fellow admin is in her or his well-deserved vacation and now you have to reboot the new machine, because you have to service the memory. Of course you look into the documentation and you find a process that isn´t covered by the documentation and there doesn´t seem to be an startup script or SMF manifest that starts this application. D'oh ... yet another last-minute pre-vacation change ... shouldn't happen, but it does happen.

Let´s assume you found an process called application in the process list:

bash-3.2# ps -ef | grep "application" | grep -v "grep"
    root 27943 27888   0 23:08:23 pts/1       0:00 /bin/perl ./application --start --debug --foobar=e3421
jmoekamp 27945 27824   0 23:08:28 pts/2       0:00 grep application

You look into the application source code (it´s a perl script, so this isn´t a problem and you see, that the application uses environment variables for config information. Now you have a problem. What are the correct settings of the environment variables? How can you gather the correct information to startup the application again. Solaris can help here with a programm called pargs.

At first let´s check the environment of the application first. It´s important to know, that you just can lookup this informations for processes you own. You need to get root privileges to see this information for other processes. Okay, the pargs command offers the -e switch for this task.

# pargs -e 27943
27943:  /bin/perl ./application --start --debug --foobar=e3421
[...]
envp[3]: DATABASEHOST=foobar.server.de
[...]
envp[6]: USERNAME=snafu
[...]
envp[13]: DATABASEDB=importantdatabase
[...]

There is another interesting switch. You can use pargs with the -l switch to get the full command line that was used to start the script, albeit this isn't that interesting, as you could get similar information with /usr/ucb/ps -auxwwww. But with a known process id pargs is much more comfortable.

# pargs -l 27943
/usr/perl5/5.8.4/bin/perl ./application --start --debug '--foobar=e3421'

Now you should have all informations to restart the script and -by the way- to update the documentation for the fellow admin.

Less known Solaris features: Getting rid of Zombies

nospam@example.com (Joerg Moellenkamp) — Mon, 01 Sep 2008 19:56:23 +0200

Once in a while you will see some strange processes with instead of a process name. This happens when a child process terminates, but the parent process isn´t interested in the outcome because it didn´t waited for the childs response. Almost all resources of the child process are freed up at the moment with the exception of the entry in the process table. The parent process need it to get the exit code from it´s child, thus you can´t simply delete it on the termination of the child. The remaining process table entry will be delete, when the parent proccess reaps the child process by gathering the exit code. But when the parent forgets to reap the child, it´s undead, it´s defunct. Or to stay in the terminology: You´ve produced a Zombie process.

Let´s create such a process. It´s really easy, we just have to create a long running process forking away a child but we don´t use the wait() system call to gather it´s response at the exit.

bash-3.2$ nohup perl -e "if (fork()>0) {while (1) {sleep 100*100;};};"&

Okay, let´s check for our processes. In the output of ps -ecl the zombie processes are marked with a Z:

bash-3.2$ ps -ecl |grep "Z"
 F S    UID   PID  PPID  CLS PRI     ADDR     SZ    WCHAN TTY         TIME CMD
 0 Z    100 27841 27840    -   0        -      0        - ?           0:00 
bash-3.2$

A kill -9 to this process is without effect. Obviously, a zombie will go away when you terminate the parent process, but that isn´t alway an option. How can you get rid of this Zombies? Okay, with Solaris you can reap such processes manually. The preap forces the parent to reap the child by calling wait() system call on the child.

bash-3.2$ preap 27841
27841: exited with status 0

And when you look in the process table again you will see, that the zombie founds it´s peace...

bash-3.2$ ps -ecl |grep "Z"
 F S    UID   PID  PPID  CLS PRI     ADDR     SZ    WCHAN TTY         TIME CMD

Obviously, you should ask yourself, why an application leaves such zombie processes, when the task of reaping them away manually gets a frequent task. Often it´s because of bad programming style.

Some new fragments for the Kerberos Tutorial

nospam@example.com (Joerg Moellenkamp) — Tue, 26 Aug 2008 21:28:35 +0200

I´ve uploaded another early pdf of the tutorial. I´ve added some text and the first to figures. Kerberos is just one of the topics where you should explain only the configuration or you have to explain it almost completly. You can´t explain just a few things and still hope your explanation makes sense.