Beginning in OS X 10.5 (Leopard), Apple built in the ability to run validity checks on any font file. Those checks can be done when the fonts are loaded into Font Book, but they’re also run beforehand when your disks are indexed by Spotlight. By using mdfind, one of the command line tools that works behind the Spotlight interface, it’s easy to find all the indexed files that may contain bad font data. Open the Terminal and type:

mdfind “com_apple_ats_font_invalid == 1″

This short and simple trick will return a list of all the suspect font files stored on any machine, whether they’re loaded in one of your user’s font folders or just stored on a spare external drive. Push the command out with Apple Remote Desktop, and you’ve got a list of every questionable font on every machine at your whole company. Once you’ve located these potential problems, it should be easy to round them up and remove them before they cause additional frustration.

Listening For Syslog Data:

Like all Unix systems, Mac OS X logs it’s system activity through syslogd, the system logging daemon. This facility keeps track of all the system activity specified in /etc/syslog.conf, which details the kind of information to log (based on its process of origin) and its level of priority (set by its parent process). This system is well documented by simply typing “man syslog” at the command line. What isn’t so easy to find is how to configure OS X clients to send this data to a central collection server for analysis.

The secret is hidden in /System/Library/LaunchDaemons/com.apple.syslogd.plist. The last item in the file is a key named NetworkListener, and by removing the comment characters around it you can tell your Mac server to listen for any and all logging information sent to it via UDP port 514. Once that’s done, you’ll need to restart the syslog mechanism by opening the Terminal and typing:

sudo launchctl unload \
/System/Library/LaunchDaemons/com.apple.syslogd.plist
sudo launchctl load \
/System/Library/LaunchDaemons/com.apple.syslogd.plist

With syslogd restarted, your server can now receive and store remote logging data from Macintosh clients, networking devices, and other Unix-compatible systems.

Sending Remote Syslog Data:

Now you’ve got a brand new syslog server. It’s listening, but nothing’s talking to it yet. For that, we’ll need to edit /etc/syslog.conf on your client machines, telling them what (and where) to report.

Open the file in any text editor, and you’ll see the following format on the very first line:

auth.info;authpriv.*;remoteauth.crit /var/log/secure.log

On the left side are a series of “selectors”, each separated by a semi-colon. Each selector is made up of a “facility” (before the period), which indicates the category being logged to, and a “level” (after the period), which indicates the level of importance that a message from that category needs to reach before it’s logged. An asterisk acts as a wildcard, including any possible facility or level.

On the right side is an “action”, preformed when syslogd receives a message matching the specified selector. This is most often expressed as a local log file, but can also be another machine listening for syslog data.

So if you wanted to log every possible message to the syslog server, you could simply add the following line (replacing server.example.com with the name of your local server):

*.* @server.example.com

That would send all that messages from any facility at any level of priority to your new syslog server. That configuration’s fine for testing a single machine, but unless your goal is to completely flood your local network with logging traffic, you’ll need to narrow down your selectors significantly before you push your revised file out to all your client machines.

Once you determine what information is important to your organization, you can build a custom syslog.conf file to install across your whole network, and begin collecting system log information for all your machines.

Recommended Reading: For in-depth information on configuring remote logging, check the manpages for syslogd, syslog, syslog.conf, and logger.

Leopard brought a number of improvements to Spotlight, the OS X search mechanism, including system-wide integration with the Finder and native applications. This is accomplished with an indexing process, mdworker, that runs in the background at all times organizing file metadata. While this feature has proven to be quite powerful, it’s also proved quite troublesome, as issues that would previously effect only Spotlight can now disable the ability to search the content of email messages and calendars as well.

When Spotlight attempts to scan a corrupt file, it can stall or crash, failing to properly index your disks and (as a result) disabling the search functionality in other Apple applications. To figure out what Spotlight’s choking on, you’ll first need it to stop indexing entirely. Make sure you’ve quit out of all your effected applications, then open the Terminal and type:

sudo mdutil -i off /Volumes/*

Once the Spotlight process is disabled, remove the old index files Spotlight built of your existing file system, replacing DISKNAME for the name of each mounted volume:

sudo rm -r /Volumes/DISKNAME/.Spotlight-V100

Next, open the Console application in the Utilities folder. View “All Messages” in the left hand column, and use the “Filter” field in the top right to search for “mdworker” (the behind-the-scenes process that indexes data for Spotlight). If the remaining errors end in file names, you’ve found a likely source for your Spotlight woes.

Make sure these corrupt files are safe to move (and not within Application bundles or required by the OS), then relocate them to a removable drive or erase them entirely. With your suspect files out of the way, you can restart Spotlight indexing:

sudo mdutil -E -i on /Volumes/*

Once the indexing is complete, check the Console logs again to make sure the errors haven’t repeated. You can now reopen your applications, and the ability to search messages and appointments should be restored.

At the root of the problem is the /etc/authorization file, which outlines unique situations where users are granted escalated privileges, and which should be altered as part of the 10.5.7 update process. It appears, however, that the file is updated only on Intel-based machines, leaving managed users on the PPC architecture unable to login on their workstations or laptops.

The solution is to copy the file to a PPC machine booted into target mode from an updated Intel installation, taking care that the ownership and permissions remain the same as on the Intel version. Alternately, if you have multiple PowerPC machines updated and booted, the same idea can be applied en masse by pushing an updated Intel file out via Apple Remote Desktop, JAMF Casper Suite, LANrev, or your preferred third-party distribution tool.

Once the corrected file is in place, reboot the afflicted machines, and login should be restored.

Keep in mind that those files hold comments and folder views for the Finder, so you’ll be annoying your Mac users by disabling them. But if your own obsessive-compulsive tendencies outweigh other people’s convenience (and whose don’t?), it’s as simple as running the following command on each client machine:

sudo defaults write /Library/Preferences/com.apple.desktopservices \
DSDontWriteNetworkStores true

If you push this out with ARD, Casper Suite, or LANrev, you can leave off “sudo” as you’ll be running the command as root. Once the prefernces have changed, just reboot your machines for it to take effect. Just like magic, no more .DS_Store!

Special Thanks: It’s a beautiful lazy holiday weekend in Seattle as I write this, and there’s nothing lazier than stealing an old trick from the Creativetechs Tips blog.

Now FSEventer is not a polished tool. The interface is crude and unfinished in some places. The documentation is sketchy. Developer Robert Pointon’s focus isn’t “look and feel”, and it shows.

Instead, the focus is on power, and it doesn’t take long to recognize FSEventer’s enormous potential. Open the application, authorize root access, and hit the triangular “play” button. As you preform tasks on your machine, you’ll see each file that’s utilized drawn into an enormous tree. Temporary files, file locks, preferences, and more all appear in real time.

Once you’ve finished your work, hit the “pause” button, and you’ve charted all the activity that’s taken place. You can inspect file information, location, and even double-click to see them in the Finder. Want to filter your results? Click the funnel. Want to start over? Just click the broom. It’s a simple interface, but one that covers all the bases.

There are several command line utilities that do similar work to FSEventer, but having a graphical representation in this case makes your data much easier to work with. If you’re responsible for pushing out configurations across your network, or just need to quickly understand how something works on your machine, there aren’t many tools that are as useful or as convenient.

FSEventer is donationware.

Configuring Secondary DNS:

Now that you’ve got a functioning DNS server inside your network, the next thing to do is consider what happens if that server is interrupted. Once your internal DNS becomes the center of your network, it’s hard to make an argument against the importance of providing redundancy. Unless you expect your main server will never go down for even a second, you’ll want a backup plan.

Fortunately, a second OS X Server can act as a secondary DNS source with little configuration. This secondary server will provide the same information as the primary (synchronized periodically to catch any changes), and

First, on the primary server you’ve just configured, go back to the “Zones” pane in the “DNS” settings of Server Admin and highlight your domain. Then further down the window, check the box marked “Allows Zone Transfer”. With the default DNS settings Apple provides, other servers within your network should now be able to inherit and host the zone file for this domain.

Now, on the server that will act as your secondary DNS, open Server Admin, and browse to the same “Zones” pane. This should be empty on an unconfigured machine. Click the “Add Zone” button at the middle of the window, and select “Add Secondary Zone (Slave)”. Enter the domain you’ll be handling secondary DNS for, and the IP address of your primary DNS server. Save, then start the DNS server.

DNS Client Configuration:

In order to get the domain information your new servers provide, client machines need to be told where to look. And since the DNS servers at your ISP likely see an external server as authoritative for your domain, you’ll need to make sure your internal clients (including your server itself) look to the internal server first.

For a single machine, this is as easy as opening the “Network” pane of the System Preferences application and replacing the current setting with your DNS server’s IP. When you’ve got to make this change on a couple hundred machines, though, even a minute each will make for hours of work. In most cases, this means some kind of network settings deployment.

If you’re already utilizing DHCP (that’s Dynamic Host Configuration Protocol) to distribute IPs on your LAN, it makes sense to use the same mechanism to distribute DNS settings as well. If, instead, your network setup somehow precludes this, Mac OS X has an easy way to push DNS settings out to your machines. Using Apple Remote Dektop’s “Send UNIX Command” feature, just select your new client machines and enter:

networksetup -setdnsservers “NETWORKSERVICE” PRIMARY SECONDARY

In this setup, NETWORKSERVICE describes the network interface on the client machine. It’s typically “Ethernet 1″ (or just “Ethernet” for laptops), but you may wish to run networksetup -listallnetworkservices if you’ve got an unusual configuration, just to see what options are available to you. PRIMARY and SECONDARY are simply the IPs for your new DNS server(s).

This will reset the servers your client machines look to for DNS information, and allow them to find domain information specific to your internal network.

Recommended Reading: For a complete understanding of the BIND software that runs Mac OS X DNS, there isn’t a more definitive text than the venerable DNS and BIND, now in it’s fifth edition by Paul Albitz and Cricket Liu.

Before we begin, a quick word of warning: If you don’t already understand DNS, you really shouldn’t be allowed to make changes to it. Nothing will screw up your day, or your employment situation, faster than taking your whole network down. So if you’ve found yourself in the unenviable position of having to set up DNS without experience or guidance, do yourself a favor and practice the procedure on a test network first. The job you save could be your own.

Basic DNS Configuration:

To start your setup, select the server you’d like to configure, and choose “DNS” from the list of available services in Server Admin. Choose “Zones” from the toolbar, then press the “Add Zone” button in the center of the window and select “Add Primary Zone”. An entry for the example.com zone will appear at the top of the window.

At this point, you’ll be tempted to hit “Save” before proceeding, but doing so triggers a terrible bug in some versions of Leopard that will break the configuration file that Server Admin edits (leaving it to permanently believe example.com is in fact your real domain). Instead, immediately change the “Primary Zone Name” to the name of your own domain, followed by a period to indicate that the entry is “fully-qualified”. A fully-qualified domain name is one that doesn’t require the host’s domain to be appended to it.

You’ll also want to change the name and IP of the first machine record (also known as an A record) to those of the server you’re configuring, this time without a period after it to indicate that it’s part of the larger domain.

You’ll then need to add what’s called an MX record, indicating the host (or hosts) to which mail is sent for your domain. Do so by clinking the plus symbol beneath the “Mail Exchangers” field, and input each hostname followed by it’s priority (traditionally 10 for the first mail server, 20 for the second). Then, and only then, will you want to save your work.

Now that there’s a record of your domain, and one of its primary server, you’ll need to tell client machines where to look for information on other domains. Click “Settings” in the toolbar at the top of the window, then click the plus symbol beneath the section marked “Forwarder IP Addresses”. Add the IP addresses of the DNS servers at your internet service provider (those pictured are mine from Comcast here in Seattle), then hit “Save” once more. This will allow your client machines to receive DNS information for the remainder of the internet.

If your domain exists only within your own network, you’re ready to click “Start DNS” and configure your client machines to use your new DNS server. If your company has any kind of internet presence, however, you’ve got a little more work ahead of you.

Split DNS Configuration:

When DNS was originally designed, every machine on the internet used a static IP, and address translation from internal to external networks didn’t exist. Twenty-five years later, the internet is a very different place, and most corporate servers are well protected from it. This can create a problem when you try to use the same domain name both inside your own network and outside on the internet, as local users won’t be able to find externally hosted services like email or websites.

The most common solution is configuring a “split” DNS, where servers both inside and outside your network control customized resolution of your domain for only those machines that can see them. This would, were I hosting my own email for instance, let internet machines find mail.makemacwork.com at an external IP of 64.13.192.203 routed through my firewall, while still pointing internal clients at 192.168.0.250 on their local network. In this case, both IPs would be the same machine, but with the external record on the DNS server at my ISP and the internal record on my own internal server.

Even if you’re not hosting any internet-available services, though, your internal machines will see your new DNS server as the authoritative source of information on your domain. So if you’ve outsourced any kind of hosting with the same domain name, you’ll need to put that information in your internal zone record as well.

Go back to “Zones” in the Server Admin toolbar, highlight your domain, and choose “Add Machine (A)” again from the “Add Record” button. Repeat this until each external server has it’s own listing. If more than one hostname resolves to the same IP, all but one should be configured using the “Add Alias (CNAME)” option instead.

With all the hostnames on your domain configured, you can finally start the DNS service.

Next week in part two, we’ll look at configuring your client machines to find your new DNS server, and setting up a secondary DNS server for redundancy.

Recommended Reading: For a simple overview of How Domain Name Servers Work, there’s a great (if slightly web-centric) synopsis at the appropriately named “How Stuff Works”.

The issue wasn’t reported in the Retrospect Knowledge Base or even the Retrospect Twitter Feed, nor was it emailed to registered Retrospect 8 users. If you happened to be lurking on Retrospect’s support forums, however, you’d find their handy Alert for Tape Library Users posted April 3rd.

According to the notice, “When highlighting a group of tape slots or a magazine and clicking Erase, EMC Retrospect 8.0 incorrectly sends the Erase All command, commanding the tape library to erase all the tapes contained in the library, instead of only those tapes in the group/magazine”. They suggest that instead, you erase each tape individually, or remove any tapes you don’t want erased from your loader.

So if you’re thinking about upgrading to Retrospect 8, you might want to wait until this (and a host of less catastrophic issues) have been resolved.

Update: This issue was addressed in Retrospect 8.0.608. We’d recommend you update immediately to use Retrospect 8 safely.

ODBackup is a script which creates valid, restorable Open Directory archives in .dmg format. In an emergency, these archives can be restored via the Archive tab in the Open Directory options of System Admin, exactly like those created via the GUI tool. Just install the script on your Open Directory master and type “ODBackup -h” for help configuring your own archive scheme.

The script allows you to either choose the archive’s password on the command line (leaving it in your command history) or have a randomly-generated password mailed to you (leaving it in your mail spool on another server). Neither choice is perfect, but the latter is arguably more secure, in that you’re metaphorically keeping the bullets and the gun in separate drawers. The random password generation also allows you to automatically create archives on a reoccurring basis, either via launchd or third-party management tools, without leaving your password in a script.

This isn’t a tool for those uncomfortable with the command line (in fact, I’d recommend you never run a script you don’t fully understand), but for those of us responsible for multiple Open Directory environments ODBackup is an incredibly generous gift.

ODBackup is available for free, under the terms of the GNU GPL.