This last week Zimbra released its new version 6.0.8 which is the first version that supports Ubuntu 10.04 (still in beta, but package was released). Being a sucker for upgrades, I decided to give it a try tonight and upgrade my Zimbra server. The upgrade actually went pretty smooth, despite the fact that I’ve had bad luck with Ubuntu distro upgrades in the past.

To start, I had a server at Zimbra 6.0.7 on Ubuntu 8.04 (64-bit). I upgraded Ubuntu to 10.04 first and then Zimbra to 6.0.8.

The upgrade for Ubuntu is pretty straightforward and is outlined in their docs. Basically, make sure the install manager is up-to-date:
sudo apt-get install update-manager-core

Then, make sure Prompt is set to “lts” in /etc/update-manager/release-upgrades. (These two steps weren’t necessary for me and I didn’t do that on purpose, so it’s likely they are ok ‘out of the box’)

Then, perform the upgrade:
sudo do-release-upgrade

I did mine from an SSH session (gives a warning and starts another SSH server on an alternate port, but I didn’t have to worry about that). When the upgrader prompted, I took the defaults except for /etc/pam.d/common-*.

Once your new Ubuntu 10.04 server is up and running, we have to grab an additional dependency:
sudo apt-get install libperl5.10

I then just ran the install like normal, but skipped the integrity checks. The first time I let them run and hit this error:

perl: symbol lookup error: /opt/zimbra/zimbramon/lib/x86_64-linux-gnu-thread-multi/auto/Data/UUID/UUID.so: undefined symbol: Perl_Tstack_sp_ptr

The second time I just skipped the integrity check and everything went as normal.

I lost a few settings, the same as the 6.0.7 upgrade . I wrote about them last time here. I also lost my zimbraMailMode setting again, same thing as the upgrade to 6.0.7.

After the settings were back, I noticed mail wouldn’t send out properly. /var/log/zimbra.log complained of the antivirus scanner being unavailable. A run of ‘zmcontrol status’ showed that clamd wasn’t running. When I tried to start the antivirus, it failed on starting clamd, showing this in /opt/zimbra/log/clamd.log:

/opt/zimbra/clamav/sbin/clamd: error while loading shared libraries: libltdl.so.7: cannot open shared object file: No such file or directory

Turns out it’s easy enough to fix:
apt-get install libltdl7

A restart of zmantivirusctl and all was good. (update: submitted bug for this)

.. my desktop kernel panics while I’m trying to read a stack trace from another machine.

I was poking around on the Zimbra forums today during some downtime and I ran across a little gem regarding configuration around a few options regarding spam delivery to users. The question was about disabling of spam filtering on a per-user basis (Yes, not really ideal, but sometimes you have to give users something I like to call ‘exactly what they asked for’), but I found even more than I expected.

Anyway, I thought I would highlight a few of the gems that I discovered.

Disabling mail to the Junk folder

This was actually the original question that I was trying to answer. Most Zimbra admins know how to whitelist/blacklist senders and other various tweaks in the Zimbra wiki, but I didn’t realize until today that we can actually configure Zimbra to not send mail filtered as junk to the Junk folder. This can be done per user, per domain, or per class of service (COS).

zmprov ma user@whatan00b.com +amavisSpamLover TRUE +amavisBypassSpamChecks TRUE

To disable, just make the + a -:
zmprov ma user@whatan00b.com -amavisSpamLover TRUE -amavisBypassSpamChecks TRUE
(or you can just set to false, but the default is false – I do like me some clean configs!)

And of course, change the command ma (modifyAccount) to match whatever type of object you want if you’re not modifying a user.

Letting banned files through to users

Every once in a while, I’ve struggled with users needing to get things like encrypted zip files or other suspicious-looking files. Not wanting to let all kinds of viruses though by disabling virus scanning and file extension blocks, it always ends up in a battle. Perhaps I’m the last to learn this one, but it’s actually easily bypassed on a per-user, per-domain, or per-COS basis.
zmprov ma user@whatan00b.com +amavisBannedFilesLover TRUE

You can also disable virus scanning for those various levels as well:
zmprov ma user@whatan00b.com +amavisBypassVirusChecks TRUE

Whitelisting and Blacklisting via zmprov (upgrade-safe!)

One of the new, well-known, features of Zimbra 6 is that you can now allow users to have their own white and blacklists, controllable within the user interface. Whitelisting and blacklisting has, of course, always been supported, but it’s been a pain as long as I’ve been managing Zimbra servers. If you’re one that already knew which wiki article I was referring to above, you know what I mean! To make matters worse, those settings have to be re-applied after upgrades.

While adding blacklist and whitelist senders to a user’s list via zmprov is expected (after all, you can configure it in the web client), what hasn’t really been touted – as far as I’ve read – is that those filters can easily be applied per domain as well. While that’s not really a new feature by any means, it does mean that those whitelist and blacklist lists are in Zimbra’s LDAP – where they should’ve been all along. That, and LDAP doesn’t get wiped during upgrades / service restarts like some config files do.

zmprov md whatan00b.com +amavisBlacklistSender @exchange.microsoft.com

Loosening up spam tag levels

You can also adjust the spam scoring levels to mark messages as junk/not junk on the various levels as well. The properties to adjust are: amavisSpamTagLevel, amavisSpamTag2Level, and amavisSpamKillLevel.

Warning: with these settings, be sure that you know exactly what each of these mean. That rule always applies, of course, but getting those settings wrong could cause your mail server to drop messages without bouncing them, thus neither sender nor receiver knows!

So many more!
This turned into a much longer post than I expected, but there are so many more options available! Check out /opt/zimbra/conf/attrs/amavisd-new-attrs.xml on your Zimbra server for even more config options.

Extra Credit
Also, check out /opt/zimbra/conf/attrs/zimbra-attrs.xml for even more goodness (unrelated to junk mail).

On my WordPress install, I use the WP-SuperCache plugin to be nice to my web server. I was shocked when trying to show a coworker something on my site (he typed in whatanoob.com – what a n00b! ) to see my homepage content very old and the redirect was no longer working (it worked some time ago when I setup WordPress). It appears that sometime after installing the WP-SuperCache plugin, my website no longer redirects the aliased site to the “real” domain, but instead just serves up the same content (this is very, very bad for SEO). The redirect removing the ‘www.’ still worked.

The fix is simple.. I just added the lines:
RewriteEngine On
RewriteCond %{HTTP_HOST} ^(www\.)?whatanoob\.com [NC]
RewriteRule ^$ http://whatan00b.com/$1 [L]
.. to my .htaccess file.

Two lessons (re-)learned:

1. Test everything – This WP-SuperCache is definitely not something that you can install, activate, and expect to “just work” with your setup. When you assume.. well, you know the rest.
2. Monitor, monitor, monitor

FYI: still not sure why cache was so stale, but going into the plugin settings and deleting the cache cleared it up. I also changed it to clear cache whenever posting new posts.

I upgraded my Zimbra server at home from 6.0.6 to 6.0.7 this weekend and during the upgrade, it seems to have lost a few settings for my outbound SMTP relay configuration (I relay through my ISP so the server can be at home). I thought I’d share what I had to go through to get it back. FYI, I followed this Zimbra doc to set it up the first time (which says it should carry across upgrades).

After the upgrade, when I tried to send to any external addresses, I got a bounce back from my ISP SMTP servers complaining that it required authentication:

Jul 3 14:32:03 mail postfix/smtp[14324]: F2F91B6002: to=, relay=smtp.relay.com[xx.xx.xx.xx]:587, delay=0.54, dela
ys=0.01/0.01/0.39/0.13, dsn=5.1.0, status=bounced (host smtp.relay.com[xx.xx.xx.xx] said: 550 5.1.0 Authentication required (in reply to MAIL FROM command))


Turns out SMTP auth had been disabled:

zimbra@mail:~$ postconf | grep smtp_sasl_auth
[snipped]
smtp_sasl_auth_enable = no
[snipped]

Easy enough to fix:

postconf -e smtp_sasl_auth_enable=yes

After a postfix reload, messages still wouldn’t relay but I got a different error in the logs:

Jul 3 14:34:17 mail postfix/smtp[15078]: warning: SASL authentication failure: No worthy mechs found
Jul 3 14:34:17 mail postfix/smtp[15078]: F0386B6002: to=, relay=smtp.relay.com[xx.xx.xx.xx]:587, delay=0.43, delays=0.01/0.02/0.39/0, dsn=4.7.0, status=deferred (SASL authentication failed; cannot authenticate to server smtp.relay.com[xx.xx.xx.xx]: no mechanism available)


A quick Google search later, I came across my problem in this forum post. The setting smtp_sasl_security_options had been set back to its default:

zimbra@mail:~$ postconf | grep smtp_sasl_security_options
smtp_sasl_security_options = noplaintext, noanonymous

It’s easy enough to fix:

postconf -e smtp_sasl_security_options=

postfix reload

After that, messages went through no problem.

I got the opportunity to attend the Velocity 2010 Web Performance and Operations Conference last week with most of the rest of the guys from the Ops team at SugarCRM. It took place right around the same time as some other stuff I had going on, so I wasn’t able to hang out “after hours” as much as I would have liked, but it was great to listen in on some great sessions and talk to people far smarter than I!

Some of the sessions to highlight:

• “A day in the life of Facebook operations” – Gathered some info on what happens when you scale bigger than pretty much everyone else in the world.
• “Datacenter Infrastructure Innovation” by James Hamilton – I’ve seen a video of an older version of his presentation, but still some new info coming in about stupid things we’ve been doing for years in our datacenters.
• “In the belly of the whale: Operations at Twitter” – similar to the Facebook presentation, but the name of one of their monitoring tools was worth the mention: “Whale Watcher”

There were also a number of sessions on different performance tools. Hopefully posts on some tested out for real are soon to come.

On the slightly less technical side, there were some great talks about culture, including quite a few talks on DevOps (more on that in a later post):

• “Creating Cultural Change” by John Rauser – some great examples on how to create change of culture within your organization by trickery, humor, and other methods
• “Ops Meta-Metrics: The currency you use to pay for change” by John Allspaw – was a great talk that could also be a blog post of its own. Basically, as operations people, we tend to be afraid of change because we associate it with downtime. To get around that, we change more often and track just how often things really do go wrong.
• “The Upside of Downtime: How to Turn a Disaster Into an Opportunity” – Talk on how to use communication and transparency from downtime to create trust from your customers.
• “The Mobile Web: The next frontier” – presentation by Strangeloop on the monetary value of performance

Another great set of videos from the conference is the “Choose Your Own Adventure” talks with Adam Jacob from Opscode. You can head over to one of the videos and see them all listed in the related videos. I didn’t get to that session, but watched all the videos and wish I had now .

It was a great few days and was followed by DevOpsDay USA over at the LinkedIn campus, more on that to come!

There’s been a significant amount of chatter lately around the Adobe v Apple we’re-more-open-than-thou pissing match, but AT&T has proved once again that vendor lock-in and control in the software industry is nowhere near as bad as the cell phone industry (heck, Verizon will even disable GPS devices in phones to make sure users have to pay for their navigation service).

AT&T has done so by announcing it is nearly doubling its early termination fees on smartphones and netbook data plans. This announcement has some seriously unfortunate timing after rumors of an iPhone able to run on the Verizon network emerged once again and seem to have been validated by AT&T’s rate hikes. As if the fee hikes weren’t enough, AT&T also seems to have taken a really idiotic stance on how it will keep its customers as well.

The money quote from the Wall Street Journal from this last week is that “about 80% of AT&T’s customers are on family-talk plans or business-discount plans, which are very ’sticky.’” That’s it? That’s the reason AT&T isn’t scared? Because it’s a big pain to switch away? Because I’d have to switch more than one phone at once and that would be annoying? How about: “We’re not scared because we have better coverage and our customers know it” or “We’re not scared because our network is faster” or even “We’re not scared because our customer service is the best”? Instead, we get: “We’re not scared because it’s way too freakin’ expensive to switch.”

Why doesn’t AT&T list any of those reasons as the primary reason? Well, I think the answer is obvious. I work about 2 miles from the Apple headquarters, where you would think coverage on the iPhone with its exclusive provider would be top-notch, if nowhere else. But, alas, I drop calls and don’t get audio on calls all the time. This just doesn’t happen with my Verizon phone (I have an iPhone from work and a Verizon phone for personal use).

All right, now that the rant is over, when I imagine AT&T execs making this decision, all I can think of is:

Over time my desktop has become a box that just sits in the corner that I boot up using wakeonlan and SSH to when needed. Tonight I wanted to jump onto the console to test a few things, but really didn’t want to go through the trouble of hooking up the monitor that now is connected to my laptop (waay too much work, I know ). Anyway, it turns out enabling VNC isn’t too bad over SSH.

First, enable it for your user:
gconftool-2 -s -t bool /desktop/gnome/remote_access/enabled true

If you’re like me, you probably enabled it at one point, set the password, disabled it and forgot what you set the password. To set it, we use the base64 utility to convert it and set it using gconftool:
gconftool-2 --type string --set /desktop/gnome/remote_access/vnc_password $(echo -n 'dontstealmysupersecretpassword!'| base64)

I then just connected with my VNC client (I used Chicken of the VNC from my MacBook) by connecting to the IP of my desktop on display 0 and the password I had set.

Troubleshooting
If you’re like me even more, you probably ran into trouble getting connected. If you’re having troubles getting connected, you can see additional settings for the GNOME remote_access using the gconftool-2 utility:
gconftool-2 -a /desktop/gnome/remote_access

There are a few key settings including “local_only”, “enabled” (of course), “prompt_enabled” (makes VNC not prompt on the desktop to allow connection – something that would obviously be a problem if you didn’t have access to the console to begin with!), and “use_alternative_port”.

I am playing with a new install of Ubuntu 10.04 in a minimum virtual machine and noticed there was no killall utility installed. In case you run into this, don’t fret:
sudo apt-get install psmisc

The package to install from wasn’t overly obvious, but not difficult to install

To say that Facebook has come under fire for its privacy (or lack there of) would be an understatement. They’ve been opening up users’ data to the world with changes to privacy settings after the introduction of several new features. All of that seemed to come to a point for me today when I logged in to my profile and was greeted with a new feature to find pages I might be interested in linking to. Handy? Well, maybe. The feature scans your profile for key elements and finds pages “for you” so you can join. Problem is, it does things like search hometowns, jobs, etc. which is all the information in our profiles that we’re trying to keep private in the first place. Keep in mind that pages are public to the world, and for me personally, I’ve become very selective in the pages that I link to. But what was scary were the options at the bottom of the prompt:

If you’ll notice, there’s a very important option that’s missing: “No”. Perhaps this is a misconception due to a poorly-worded prompt, but it was a bad one to screw up at this stage in the privacy game.