Configuration management / orchestration software already knows how to handle systemd and they do it well. Take advantage of that.

Unfortunately for me, there is a Chef server in our infra so old that doesn’t know how to handle systemd, and with the release of CentOS 7 (1511) it finally broke.

================================================================
Error executing action `restart` on resource 'service[iptables]'
================================================================

Chef::Exceptions::Service
-------------------------
service[iptables]: unable to locate the init.d script!

It expects that all services have a script in /etc/init.d, so here is my quick hack around it.

$ ls -l /etc/init.d/iptables
lrwxrwxrwx 1 root root 32 Dec 17 17:34 /etc/init.d/iptables -> /usr/local/bin/systemctl-wrapper

$ cat /usr/local/bin/systemctl-wrapper
#!/bin/bash
# workaround for old school SysV persons
# stsimb dec 2015

SERVICE="$(basename $0)"
exec /bin/systemctl "$@" ${SERVICE}.service

Now we can spend more time migrating to our new Chef, that knows how to handle systemd.

delv works very much like the already know dig. It is like dig with special DNSSEC validation powers. delv checks the DNSSEC validation chain using the same code that is used by the BIND 9 DNS server itself.

But keep in mind that delv, as any standard unix tool, will use the resolvers in /etc/resolv.conf to perform all lookups. If all your resolvers listed  are DNSSEC validating, delv will not be able to lookup a non-dnssec validating RR, and will not help you debug the problem.

[stsimb@jumbo ~]$ delv www.spbet.eu. +multi +rtrace
 ;; fetch: www.spbet.eu/A
 ;; resolution failed: failure

You need to have access to some non-DNSSEC validating resolver, to kick start the trace..

[stsimb@jumbo ~]$ delv www.spbet.eu. @4.2.2.1 +multi +rtrace
 ;; fetch: www.spbet.eu/A
 ;; fetch: eu/DS
 ;; fetch: ./DNSKEY
 ;; fetch: spbet.eu/DS
 ;; fetch: eu/DNSKEY
 ;; fetch: eu/DS
 ;; fetch: ./DNSKEY
 ;; fetch: www.spbet.eu/DS
 ;; fetch: eu/DS
 ;; fetch: ./DNSKEY
 ;; fetch: spbet.eu/DS
 ;; fetch: eu/DNSKEY
 ;; fetch: eu/DS
 ;; fetch: ./DNSKEY
 ;; no valid RRSIG resolving 'www.spbet.eu/DS/IN': 4.2.2.1#53
 ;; no valid DS resolving 'www.spbet.eu/A/IN': 4.2.2.1#53
 ;; resolution failed: no valid DS

Personally I don’t consider this a bug. It’s expected behaviour, a unix tool to use the resolvers in /etc/resolv.conf for resolution.

delv cannot determine for sure if it’s your resolver’s fault, and then try another one (and if it did, which one should it choose?). It’s your job to have this in mind, and ask some other non-dncsec validating resolver that you have access to.

Nevertheless, this situation may come up many times. Troubleshooting broken trust chains is one of the reasons someone will use delv, and gives you a reason to keep a non dnssec-validating resolver available somewhere.

Update: Thanks to Peter van Dijk‘s tweet I discovered that +cdflag works in delv (although it’s not listed in delv -h).

Size information is buried in the database, and needs a few queries per folder to dig..

Here is a script to do it for you.. zimbra-size.sh

[zimbra@zimbra ~]$ /tmp/zimbra-size.sh sakis@zimbra.gr
sakis@zimbra.gr's max mailbox size = 20480 MB, current mailbox size = 343.02 MB.

size (MB)  msgcount     unread folder
--------- --------- ---------- ----------------------------
        0         0          0 /Chats
        0         0          0 /Drafts
       12       221        221 /Inbox
       15      1001       1001 /Inbox/Tickets
      310      7158       7158 /Inbox/Syslog
        3        78         78 /Inbox/Wiki
        0         0          0 /Junk
        3        21          0 /Sent
...

If you don’t want to increase it globally using sizelimit, then you can do it for specific users.

All you have to do is add something like the following in slapd.conf.

limits dn.exact="uid=serviceuser,ou=users,ou=domain,ou=com" size=unlimited

(manual reference)

/etc/sudoers contents looked like this

%sudoers hostname=(root) NOPASSWD: /usr/bin/chef-man-provision

and it actually executed rvim

rvim -u /root/.rvimrc /etc/munin/munin.conf

Contents of /root/.rvimrc look like this

" all following options are for securing vi for chef-man-provision script. 
" Security option: disable file opening with `:r` command
cmap r <Nop>
cmap o <Nop>

This «security option» had a pretty nasty effect… We couldn’t use characters «r» and «o» in vi command mode, even in simple search strings…

Enter sudoedit! To quote the sudoedit(8) manpage

-e‘ The -e (edit) option indicates that, instead of running a command, the user wishes to edit one or more files. In lieu of a command, the string «sudoedit» is used when consulting the security policy. If the user is authorized by the policy, the following steps are taken:

1. Temporary copies are made of the files to be edited with the owner set to the invoking user.

2. The editor specified by the policy is run to edit the temporary files. The sudoers policy uses the SUDO_EDITOR, VISUAL and EDITOR environment variables (in that order). If none of SUDO_EDITOR, VISUAL or EDITOR are set, the first program listed in the editor sudoers(5) option is used.

3. If they have been modified, the temporary files are copied back to their original location and the temporary versions are removed.

If the specified file does not exist, it will be created. Note that unlike most commands run by sudo, the editor is run with the invoking user’s environment unmodified. If, for some reason, sudo is unable to update a file with its edited version, the user will receive a warning and the edited copy will remain in a temporary file.

So, /etc/sudoers was modified to look like this

%sudoers hostname=(root) NOPASSWD: sudoedit /etc/munin/munin.conf

We now type «sudoedit /etc/munin/munin.conf» and everything works as it should work in the first place, i.e. securely and we can use all the letters!

.@stsimb Any chance you're going to change your cvi tool to use #git ?

— Yiorgos Adamopoulos (@hakmem) April 5, 2013

cvi was published in an earlier post: version control for important files

Going distributed instead of local can be tricky, because somebody else may have changed the file in the master repo.. but here is a cruel attempt..

#!/bin/sh
# https://stsimb.irc.gr/2013/04/07/cvi-going-distributed/

if [ -z "$1" ] ; then
  echo edit a file stored in a git repo
  echo usage = $0 filename
  exit 1
fi

git pull origin master || exit $?
vi $1
git add $1
git commit
git push

Intro

I wanted to check how often a domain gets updated, so I quickly put together soacheck.

I assume that everybody increments the SOA serial number when they make a change or generate the zone file. But strictly speaking, they don’t have to do it (they can use some other way to sync changes to all their NSs and make them available).

Usage

List the zones you’re interested in file «zones-to-check». Execute soacheckd in your crontab (no need to run as root), as often as you like. e.g.

* * * * * /path/to/soacheckd > /dev/null

Check the results in soacheckd.log.

[stsimb@pipitsa soacheck]$ grep " gr. " soacheckd.log | tail
Fri Feb 1 04:51:01 EET 2013 soacheckd[22093] gr. serial changed from 1301312194 to 1302010099
Fri Feb 1 07:51:02 EET 2013 soacheckd[19481] gr. serial changed from 1302010099 to 1302010401
Fri Feb 1 10:52:01 EET 2013 soacheckd[18500] gr. serial changed from 1302010401 to 1302010695
Fri Feb 1 13:52:01 EET 2013 soacheckd[19079] gr. serial changed from 1302010695 to 1302011000

Ideas

• Record answers from authoritative servers (answers now are from system resolver’s cache)
• Record answers from multiple servers
  • use a set of public dns servers
  • use some dns-lg instances around the world
• Make multiple checks in parallel
• Record changes of other RR types, e.g. A, AAAA, CNAME, PTR etc
• Trigger some event on change

Thanks to carinsurance.org for this work and to Allison M. for bringing it to my attention :)

Προαγωγή και πάλι!

So I did the experiment, for a week of email traffic at a certain mail server, and here is the result:

Amazing, the law stands for sizes as well!

Ξαφνικά, αρχίζει βροχή από SMS alerts για services που πέφτουν ..

Ανοίγεις το netbook και έχεις δίπλα την κεραία με το σήμα 3G στο full..

Priceless!

(a reminder to myself)

I visited a test-ipv6 page .. but the page took too long to open, had broken images, timeouts and readiness score 1/10 !

So I changed my MTU to 1280 and tested again

(In linux you might type: /sbin/ifconfig eth0 mtu 1280)

I then found this relevant discussion: IPv6 and path MTU discovery black holes

UPDATE 4/6/2011: An MTU of 1492 also seems to work correctly in my setup (as @chatasos prefers http://goo.gl/KRgm5)

I found the lab based exams much better than other multiple choice exams. It doesn’t matter if you do something using the command line, a gui tool, or if you write a program yourself to do the job. It’s real world simulation!

The preparation was self-study using the following:

• Exam objectives for RHCSA and RHCE
• RHCE Rapid Track Course Outline (RH299)
• Red Hat online skill assessment questionnaires
• Lots of RHEL documentation
• A few VMs to destroy while experimenting ;-)

No book covering RHEL6 existed while I was preparing, but I believe one is available now (ISBN 0321767950) and others are on their way (e.g. ISBN 0071765654).

I also recently discovered OpenRHCE by @texastwister, an effort to collaboratively prepare study materials to help candidates achieve Red Hat’s RHCE certification, under a CC license.

I found the journey of answering all the above quite a rewording experience. I have learned new things and I can definately troubleshoot SELinux issues faster than before ;-)

Let «host» do the dirty work for you!

sotiris@jumbo:~$ host 2001:db8:2:2001::11
Host 1.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.0.0.2.2.0.0.0.8.b.d.0.1.0.0.2.ip6.arpa not found: 3(NXDOMAIN)

The zone for your /32 is 8.b.d.0.1.0.0.2.ip6.arpa

$TTL 1d
@    IN    SOA ( 42 1h 15m 30d 10m )
     IN    NS    localhost.
;
1.0.0.2.2.0.0.0    IN    NS    localhost. ; delegate 2001:db8:2:2001/64

The zone for your /64 is 1.0.0.2.2.0.0.0.8.b.d.0.1.0.0.2.ip6.arpa

$TTL 1d
@    IN    SOA ( 42 1h 15m 30d 10m )
     IN    NS    localhost.
;
1.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0 IN PTR mydevice.local.

Reconfig and run host again

sotiris@jumbo:~$ host 2001:db8:2:2001::11
1.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.0.0.2.2.0.0.0.8.b.d.0.1.0.0.2.ip6.arpa domain name pointer mydevice.local.