Natural Networks

F5 BIGIP LTM Reboot Script

2011-09-03T16:58:00.002+10:00

In an effort to ensure the best performance and stability of our two BIGIP LTM 6400 Load Balancers I have created a script to synchronise and reboot the units regularly.

This script runs a series of checks before rebooting the unit.

Check Active/Standby state based upon the output of bigpipe failover show
Check Peer status (up/down) - based upon the result of ping -c 1 -w 5 peer ('peer' is the hostname of the peer BIGIP)
Check the uptime to see when the last time the unit was started, if under a given period then don't reboot
Check configuration synchronisation status based upon the output of bigpipe config sync show

If the configuration is not in sync then it will attempt to synchronise the configuration using bigpipe config sync all and check the status of the synchronisation again. If the configuration is still not in synch it will exit and not reboot the unit.

Each check/task will output to STDOUT and syslog (facility: local0.notice tag: BIGIP-ADMIN-SCRIPT). Also a result file (/tmp/reboot-cron-job-result) that will be left in place until next run and will also be e-mailed to 'user@domain.tld' (change this to suit your environment).

The same reboot.sh script is used on each unit.

Still some tidying up to do - like using a lockfile and better error handling like using 'set -e' and 'set -u' and traps.

/home/admin/reboot.sh

I run this script using a cron job that occurs at the start of our weekly maintenance window. You can also use this script as a safe way to force a failover and reboot.

F5 BIGIP LTM Maintenance Page Update for v10

2011-07-04T09:15:00.003+10:00

The folks at F5 devcentral have kindly provided a number of 'Maintenance Page' examples that allow you to host a page directly from the BIGIP LTM and display it automatically when all pool members go off-line. The example I used is http://devcentral.f5.com/wiki/default.aspx/iRules/LTMMaintenancePage.html (login required, registration is free).

However there are a few changes required to get it working with the latest version of TMOS (v10).

Follow the instructions provided in the aforementioned link and change them as follows:

Create iRule Data Groups with the following information:

maint_index_html_class

General Properties
Name: maint_index_html_class
Partition: Common
Type: (External File)

Records
Path/Filename: /var/class/maint.index.html.class
File Contents: String
Key/Value Pair Selector: :=
Access Mode: Read/Write

The file will need to look like the following (add "index.html" := to the beginning of existing example):

main_index_logo_class

General Properties
Name: maint_index_logo_class
Partition: Common
Type: (External File)

Records
Path/Filename: /var/class/maint.logo.png.class
File Contents: String
Key/Value Pair Selector: :=
Access Mode: Read/Write

The file will need to look like the following (add "logo.png" := to the beginning of the existing example):

generic_irule_maintenance_page

Change [lindex $::maint_index_html_class 0] with [class element -value 0 maint_index_html_class]
Change [b64decode [lindex $::maint_logo_png_class 0]] with [b64decode [class element -value 0 maint_index_logo_png_class]]

F5 BIGIP and Blackboard Collaboration Server

2011-05-31T20:51:00.021+10:00

Blackboard Collaboration Server is a separate, optional, web server that provides virtual classroom and chat tools. As part of the university’s Blackboard application upgrade I have been asked to develop a way to add resilience to the collaboration server side of the application where possible.

The brief is to provide failover only. The reason for this is that the collaboration server is not “load balancing aware” in that it assumes that it will be hosted on a single host. To provide rudimentary fail-over capability I have set up a method that will switch all sessions to another host should the active host fail. However clients will stay on the new host until it fails and only then switch all sessions to the other. The key word here is ‘all’ because it’s important to keep all sessions on the same host.

From a users perspective; in the event of an active host outage they will lose connectivity but will be able to log back in straight away and continue until such a time the alternative host fails. This prevents them from being switched over only to be kicked again when the prior host has been restored and also ensures that ALL sessions are sent to a single host and not spread across multiple host, so everyone is in the same chatrooms.

My first idea was to adapt BIGIP’s Priority Group capability however this presented the same problem where I could not ‘stick’ the clients to a server. As soon as a same or higher priority server was restored the sessions would be sent to the new host effectively splitting the chat rooms. Also load balancing will take place on member servers of the same priority.

So I did a bit of digging around and discovered a method of using an iRule to provide me with the capability to ‘stick’ sessions based upon an arbitrary number in this case I used the TCP Port number.

The iRule is as follows:

CLIENT_ACCEPTED is an event that is triggered when a connection has been established between a client device and the BIGIP.

‘persist uie’ is where I am manipulating the connection persistence and in this case the Universal Inspection Engine. Here I am simply setting a integer, can be any number but I have chosen to use the connecting TCP port number ([TCP::local_port]). This fixes the session persistence to a single host, preventing load-balancing.

The following BIGIP configuration has been tested as working by business systems analyst using a combination of application logs, BIGIP statistics and packet captures. He confirmed what traffic was being sent on which ports - Port 8010 is used for the majority of user generated traffic that must be kept on a single host. Port 8443 is used to transport application specific information but does not carry anything that is user generated and therefore does not require persistence.

The aforementioned iRule is referenced by a ‘Universal Persistence’ profile as follows:

And then reference that Univeral Persistence profile from a Performace Layer 4 type Virtual Server like so::

Another Virtual Server is required for HTTPS traffic however this does not require any special configuration and is set up as a typical HTTP type Virtual Server e.g:

The above configuration refers to a six member/node pool. Each member runs both the general Blackboard application and the Collaboration Service. We have yet to load test the combination of the application and collaboration services and how they influence how the BIGIP balances the load across the members - considering using ‘Observed (node)’ as opposed to the current ‘Observed (member)’ method since the same nodes are used in multiple pools. Although at some stage I would like to look at uses Dynamic Ratio if it can play nicely with persistent connections.

References:
http://devcentral.f5.com/wiki/default.aspx/iRules/CLIENT_ACCEPTED.html
http://devcentral.f5.com/wiki/default.aspx/iRules/persist.html
http://support.f5.com/kb/en-us/archived_products/big-ip/manuals/product/bigip4_5admin/BIGip_uie.html

Also take note of:
http://support.f5.com/kb/en-us/solutions/public/4000/100/sol4166.html

Windows Wireless Clients and the X6148V-GE-TX Ethernet Switching Module

2011-05-28T23:06:00.036+10:00

Burnt hard by a bug that exists in a place that makes plenty of sense when you find it but not so much when you’re looking at the symptoms.

I was tasked with establishing an EduRoam presence at a University. Since there was already a suitable wireless infrastructure in place all I needed to do was build a FreeRADIUS server, hook it into the EduRoam federated RADIUS and point the two Cisco 4404 controllers dressed as a WiSM (Wireless Services Module) at it so they authenticate EduRoam clients. Easy!

Getting FreeRADIUS communicating nicely with EduRoam was made more difficult than it needed to be. The configuration information provided from EduRoam was sketchy and inaccurate. It wasn’t until I decided to chuck it out and build the FreeRADIUS configuration from scratch that it worked. EduRoam have some strange ideas on what should be sent on the outer TLS tunnel... it’s the inner tunnel that’s important, the other is just establishing an anonymous TLS connection to the local RADIUS server which will then pass the inner-tunnel to their home campus RADIUS.

Okay, that was a bit tedious however that should be the hard part over with. Authentication was working nicely with the local LDAP directory (Novell eDirectory) and with other federated entities, tested with accounts from James Cook University, AARNET and the Australian Catholic University. Just the simple task of setting up a WLAN on the WiSM and confirming that it works with EduRoam as I had been using my trusty Mikrotik RouterBoard RB433 for testing. Associate a laptop to the new wlan, go to open google and was presented with a rather slow web experience that would basically stall on the first image that tried to load. However pings were fine so end to end connectivity was all there.

Odd. Maybe I left something out/in or perhaps the RADIUS was setting some kind of QoS value on the controllers that I wasn’t aware of. Checked all that out, nope all good. Maybe it’s the laptop? Try a little netbook running Jolicloud - works fine. Okay, lets check with another laptop - win7 - fail! Macbook - works! A Windows wireless client + WiSM + EduRoam problem?? Hang on, lets try the Intranet, works! Lets try a proxy server, works! This is getting annoying, so it’s a Windows wireless client + WiSM + EduRoam + FWSM/NAT + Internet problem??

The next 8 months consisted of running every conceivable check on the data path between a Windows wireless client and the Internet. The Cisco TAC had crawled over the WiSM - all good, the FWSM, hmm old untrusted software, install another one! test again - all good, even the ASR - nope, all good.

So I figured that it must be something I’m just not doing right. I blew away my test environment which consisted of a C4402 wifi controller and C1131AG/C1142N LWAPs, and the second FWSM running the latest software and rebuilt it. However when I did this I had physically relocated all the kit (except FWSM of course) from the data centre to the foyer just outside. In doing this I had disconnected the C4402 from the C6513 and plugged it into a C3750 I had set up for the link between the APs and controller and the trunk back into the general network. This configuration worked!

The test environment at this stage

So what did introducing a C3750 or simply moving it elsewhere on the network do to fix the issue? This made me think there was something suss going on with the chassis and/or connecting switching modules.

By now the TAC had grown tired of my pokes and prods so I gave our Cisco account manager a nudge and the SR was escalated and an e-mail that was CC’d to ‘Cisco Australia’ popped into my inbox from the Cisco Switching team asking for a webex session so they could waterboard the 6513 chassis that housed the WiSM and FWSM.

The phone call started at 10am Monday morning and didn’t end until 3pm.

We worked through each stage of the data path again. Luckily they had the history of all the other tests I had done so I didn’t have to do many of the captures again. We narrowed down to the X6148V-GE-TX switching module. This was the one element that shared something in common with all the different combinations I had tried. The C4402 test controller was connected to it along with the link to the ASR/Internet. So I connected the C4402 to a port on the module (issue present, not working), ran a capture. Then moved the C4402 to a X6724-SFP module (no issue pressent, working) and ran another capture. Then the TAC guys ran a comparison between the two caps. It seems the X6148 was silently dropping packets, small ones, particularly ACKs from the client - egress to the ASR/Internet.

Gentlemen, we had hit Cisco bug CSCeb67650:

WS-X6548-GE-TX & WS-X6148-GE-TX may drop frames on egress

Packets destined out the WS-X6548-GE-TX or the WS-X6148-GE-TX that are less than 64 bytes will be dropped. This can occur when a device forwards a packet that is 60 bytes and the 4 byte dot1q tag is to added to create a valid 64 byte packet. When the tag is removed the packet is 60 bytes. If the destination is out a port on the WS-X6548-GE-TX or the WS-X6148-GE-TX it will be dropped by the linecard....

WLC drop TCP ack from wireless client to wired

Symptom: Wireless client has problem loading certain web pages. Conditions: client connected to wireless controller, and has problem loading web pages from certain web sites. Specifically has problem loading pictures. A wired packet capture shows the ack coming from the wireless client are been drop on the controller. Workaround: None

Since there was no workaround the only option was to shift the ASR/Internet link from the X6148 to a X6724. Fixed!

I plan to remove the X6148V-GE-TX from the chassis anyway along with a CSM. These are both ‘classic’ modules that don’t use “fabric switching” (2 x 20Gb dedicated) but instead use an older “bus” method (32Gb shared) thus causing the chassis as a whole to not run as well as it could. However if X61xx modules were all I had then I would be in a pickle.

Note:

Wondering why this only affected Windows clients? So am I.

ACKs aren't all the same 'size' given the comparisons between pcaps I've grabbed from public repos. However ACK frames during a HTTP transfer all seem to be 60bytes long no matter the OS.

I think it could be related to the differences between the Slow Start/Congestion Avoidance algorithms. The ACKs are probably being dropped no matter which OS is sending them, however some OSs might be better at recovering. Something to test. Although this problem shows indiscriminate dropping of 60byte frames so how can they recover??

I haven't been able to find a decent comparison between *nix/BSD/MacOS and Win* TCP stacks. It would be an interesting test to get a Linux box running the same algorithms as a Windows box. When I pull the X6148 out I'll toss it into the test 6509 and hang a test webserver off of it.

Zimbra Part II

2010-08-08T09:01:00.000+10:00

I mentioned a while ago that I will be rolling out a Zimbra mail server. It's been a hard slog but I think I've got it together enough to roll out into production.

I had some grief with bad sectors showing up on the system disk I used. They showed up in the swap partition. When bad sectors show up here applications that are using that virtual memory will show behaviour akin to faulty memory. This meant that when I was migrating e-mail from the old mail server to the Zimbra server it would cause Zimbra to use a bit of swap memory for various things and come across these bad sectors in turn causing it to crash.

Having pinpointed the fault to bad sectors (using dmesg to see the disk errors) I went about imaging from the old disk to a new one. The system disk isn't mirrored, Zimbra lives on a mirror but the system disk stands alone - you can call it a compromise of costs if you like. However the cheap onboard RAID controller is either set to all SATA ports as RAID or none. I have to set up single disk stripes in order to add a single disk. This means that I have to contend with the obscure device mapping between the BIOS, RAID BIOS and the linux device mapper. Juggling all these around I managed to get the new disk in and booting without failed mounts and what not.

I decided after all this to clear the user accounts and aliases and refresh them. The reasons for doing this is that I modified the zmprov script for converting the passwd file to a zmprov command list to include UID and GIDs plus the SambaID. I could have created another script that simply modified each user however I felt it better to run thought the process of clearing and restoring users again just to be sure.

I also updated Zimbra to the latest 6.0.7 release. I also tested the shared calendaring and resource scheduling a bit further to make sure it fits the requirements - all works quite well and I like the different permission levels for managing resources.

I think it would be possible to make a Zimbra appliance of sorts so I wouldn't mind having a go at scripting the installation and packaging it up into a small power efficient server that can be easily used by small businesses or as departmental mail servers. Could be something to add to my consulting work on the side along with cheap and efficient network consulting.

One Of The Situations I Find Myself In

2010-07-26T11:36:00.006+10:00

The other day at work reception called and said that they had an Ian from Summerville High School on the phone wanting to know about what kind of software packages were in use by the Zoo. Straight away I had my doubts because we receive many calls from sales people wanting to get a foot in the door which is made easier by developing an understanding of our IT environment. Since it was still a doubt I decided to take the call.

Speaking to Ian on the phone he said that he would like to know more about the database systems in use by the Zoo. He said that he would be bringing in about 16 girls from Summerville High in Brisbane and part of their current education is about databases. It was strange because I would consider the zoo to be the last place to go on an excursion to learn about databases. Unless he had the foresight of knowing that we use databases to track our animals?

I compromised - I told him that he could call me and I would go out and talk about how the zoo uses various software packages with a database back-end.

The day came and I didn't receive a call at 9am when he said he would. I put it down to a failed sales guy and went on with my usual tasks. However at about midday I receive a call from him and it turned out that he was there but with only six girls. So I had to give an impromptu talk about database systems in the Zoo environment to six high school girls. It's not something I had expected to do while working as a IT guy in a zoo.

I think I taught them something. I haven't given a talk to a group about IT topics for sometime so I forgot to do things like gauge their existing knowledge or try to obtain more feedback from them in the form of questions and revisions. It did remind me how much I liked talking about the subject to others and I miss the training side of what I do.

It was but a small break from the mundane.

Zimbra, once more

2010-05-24T08:37:00.003+10:00

I remember playing around with the Zimbra Collaboration Suite when it first came into public existence sometime ago. I was working for a different company back then and was looking at it from an ISPs perspective, it was good but wasn't exactly there yet but development was well underway in that regard. I played around with it a bit and stuck it on the 'neat tech to check out later' pile.

Later on and a change of jobs I looked at it again, this time from a medium enterprise perspective. This time I was looking specifically at the per user licensing for use of the Outlook Connector. The costs were okay but the limited testing in our environment proved it to be a bit hit and miss, although I contribute a large portion of the blame towards the lack of any formal directory service or structure.

Now that I'm presented with a rapid upgrade requirement to save our e-mail services (due to shortsightedness of management) I'm having to jump straight into rolling out Zimbra with limited testing. So I blew away a idle Win2k3 SBS server and installed Ubuntu 8.04 LTS Server and tossed on Zimbra 6.0.6.1. I did do some testing beforehand on a xen vm just to make sure it would install and operate okay before I wasted a good 2k3 install.

Currently I'm impressed with how far ZCS has come along. There's plenty of documentation available in the wiki and forums and the 'zmprov' provisioning utility is working wonders with shifting user accounts over from the Sendmail/Dovecot/PAM setup on the old mail server. I am using imapsync to copy the 143GB of email over thanks to the handy scripts provided by the ZCS community.

One thing I like in particular is the ability to dump the {crypt} passwords straight from the shadow file into Zimbra's LDAP - no need to have everyone change their passwords. Although it is recommended and I will get them to do so after I'm satisfied that Zimbra is working okay.

I will update this post with a run down on the scripts I used with any modifications I made.

Simultaneous TV Output

2010-01-22T10:12:00.002+10:00

Why can't current graphic adapters support simultaneous TV-Output these days? What changed for them to disable this feature which was common on older chipsets/cards?

Currently suffering from a problem where one of our Audio Visual PCs has died and required a new mainboard etc. Stick in a current PCI-E nVidia something a rather with it's TV out only to discover that it can not run both displays at once. We use the TV-Output to send video to a monitor which in turn feeds a visual mixing desk and outputs to the big screen in the Crocosium.

Now we're trying to track down a card that allows this which isn't as easy as we had hoped.

Notes about MythTv and Nambour

2009-08-17T17:20:00.003+10:00

Just throwing this here so I remember in future.

The 'Sunshine Coast' channel list provided by shepherd causes me greif. I have a rather substantial antenna on my roof and the inclusion of ABC's Brisbane band causes myth to use the Brisbane ABC channels over the local ones. I get tuner lock, but in myth-frontend it shows no lock with 10% signal.

So I removed the 226.5MHz Transport and corrected the 767.5MHz transport to read the correct frequency of 767.625MHz (Bald Knob and Dulong towers). This fixes my ABC1/ABC HD/ABC3 channels. Probably don't need to correct the existing freq but meh.

Weather Station

2008-08-13T12:59:00.004+10:00

Installed a Fine Offset Electronics WH1081 Weather Station on my roof the other day. Purchased the device from ebay for $85 and $25 delivered.

The station consists of the following sensors:

Thermo-hydro transmitter
Wind speed
Wind direction
Rain gauge

The console is a touch screen LCD panel. Apart from the LCD being difficult to read due to being too light (needs a contrast setting) it works reasonably well. The best part about it is that it has a USB connection for plugging into a PC.

To go with the USB conneciton the station comes with a software package called 'EasyWeather' which is functional and maintains a log with various graphs showing historical data. I wasn't too fussed on it though since it did things like give false readings and has really bad memory leaks.

I tried Cumulus, which is another weather station application that can upload results to a FTP site for Internet access. However it doesn't recognise the WH1081 natively and instead relies upon EasyWeather to gather statistics - not good for the above reasons of instablity and memory leaks.

Currently I am using the Linux console version of Weather Display. I like it because its a no fuss application with no frills and supports uploading results to Weather Underground natively. However it doesn't support 64bit at all, even when using IA32 libraries. This made me install 32bit Ubuntu (CLI only) within a VirtualBox - had to use the closed source version for the USB support. This is on my Mythbuntu 8.04 HTPC too, so the weather console sits on top of the TV in the loungeroom which worked out well.

You can see my weather data on Weather Underground here.

Network upgrade

2008-04-05T12:10:00.004+10:00

And the upgrades continue.

The network here is expanding something chronic so I needed something that could push the vlans harder. It's basically replacing the local Mikrotik/WRAP1-1 Router and Asus GigaX2024 L2 switch with a single Asus GigaX3112 L3 switch. It certainly tidied up the rack by removing two switches and a stack of patch leads.

Now I have two Asus 2024 L2 switches to stick in Admissions and the Crocosium. This will give me a gigabit vlan trunk to the locations and allow me to create some more subnets to reduce some of this needless traffic off the main networks. I'm trying to have a L2 managed switch on the end of every fibre link to get some flexibility into the network and get things into this decade...

I'm not all that impressed with the Asus network kit so far. It's okay for the price but it's buggy as hell and the 3112 tends to crash due to kernel panics and reboot due to buffer overflows or memory errors. I'm hoping future firmware updates will come and fix things. Not that I had much more luck with Netgear and Linksys stuff. There's a reason why Cisco can charge so much.

The campus wireless network is almost completed. Awaiting a cable run from the Machinery shed to the Conference center to install the AP on the roof there. Also need to install the 12th AP at the Tiger Temple to complete the 'ring of coverage' - full wireless coverage of the safari shuttle track and nearby walkways and buildings. Will put off any more expansion until the Hotel is built.

Terminating fibres

2008-02-09T15:05:00.001+10:00

Been busy terminating a few new fibre runs throughout the Zoo. This particular run goes from the Taj (Crocosium) through to the Dingo Diner and then onto the Snake Shed and Compound. 12 cores to the Diner, 6 cores each to the shed and compound. Single mode glass fibre is used throughout however we may start using Multi-Mode within buildings.

Terminating fiber isn't that difficult these days. Basically you just need cutters and strippers, cleaning wicks, guillotine and fusion splicer. We hire the fusion splicer and guillotine and purchase pre-made tails with ST connectors from AFC. Ends up being around $50 an end instead of the usual $100 an end for a contractor.

Pics:

An A/V Perspective

2008-01-20T09:32:00.001+10:00

Here's a picture of the A/V crew operating the two remote PTZ cameras and 'house' audio gear during the Veronica's gig at Australia Zoo.

The panel between them is the Panasonic Visual Mixing desk. The smaller monitors are the cameras and the screen on the top is what's currently showing on the big screen (a rather large Panasonic LED screen outside).

Asus Eee PC 4G (701)

2008-01-14T21:02:00.001+10:00

Today I purchased a Asus Eee PC 4G from the local Myers department store (Sunshine Plaza).

The only colours they had were black and white. I didn't want to be mistaken for jumping on the iWagon so I got the black one. It happens to look identical to my work laptop, a black LG LS70 Express, so I have a matching pair of portables :/

So far I've played around with the default OS - Xandros with the K Desktop Environment installed along with a custom Asus application launcher. I have also installed a live version of eeexubuntu onto a USB stick (Myers threw in a 2GB Toshiba USB stick for nothing, valued at $28AUD) and it ran fine in its 'live' mode. I will get a 2/4GB Secure Digital card to install xubuntu onto and then set up a dual boot with Xandros. That way I can use Xubuntu for my work and Xandros to demo it to the uninitiated.

I'm not going to go into details about its performance etc - plenty of reviewers out there already demonstrating that and I agree with their findings. I can confirm that the build quality is good and that it will probably handle rough treatment a lot better than other notebooks - or most 'non-ruggidised' portable electronic device for that matter.

I'm looking forward to setting this up as a my ultra-portable network tool. Testing wireless will be made a lot easier, also cheaper than those dedicated PDA based wifi test units. I can also do my usual router configuration using a USB serial adapter and its 10/100 network adapter.

I will consider the purchase of a long life battery when they become available - they apparently add a lot of extra bulk to the package but it would be worth it for extended trips or major network problems.

A Wikipedia article describes the various details of the Eee PC range can be found here.

Asus Eee PC size compared with Sony Ericsson P1i

Home wireless

2008-01-13T22:40:00.001+10:00

Today I finally set up my little public access wireless network at home. I've had a Mikrotik RB133 with RB52 and Senao radios laying around in an outdoor box for sometime now. Today I fixed up the roof cable run and installed it on my existing mast with Cisco 21dBi solid dish.

I used a homebrew 8dBi collinear omni-directional antenna for the public side (Senao 200mW/23dBm) radio - that puts output at around 30dBi in a 360°H/25°V area. I've set up a rudimentary hotspot service on the interface - I will set it up with PayPal etc shortly so the neighbors can gain ad-hoc internet access and not feel guilty for it. Not everyone wants to pay for a permanent connection unbelievable huh.

I attached the dish to the RB52 radio which is adjustable - so I'll set that down to give no more than the regulated 36dBi. I'm hoping to establish a Point-to-Point link to a friend in Buderim. Failing that maybe a repeater link via Kiel Mountain which sits in between. Mainly to share links and to muck around with routing etc.

I'll go for a cruise around the local area and see how much coverage my little 8dBi is providing - might even be able to pick it up at Kiel Mt with the laptop radio...

Progress

2008-01-11T21:16:00.001+10:00

Wireless Network

How things move slower the lager the enterprise. Might have to start pushing to get things done faster and more efficiently in future.

I installed the last two Access Points in from the first batch of six we received all those months ago. I guess a fairly large hold up was the flaky firmware of the Symbol WS5100 I mentioned earlier. It's going very nicely now and I can't wait to start tweaking it and including it into a radius/ldap domain.

One AP was installed in the main warehouse and another under the Taj for coverage of the main hallways. Both work quite well using the supplied 6dBi collinear antennas.

Today a 9dBi collinear from the RFShop arrived with two made up cables to suit the AP300. Another three access points will be ordered shortly to provide coverage of the Foodcourt - using the new 9dBi, open area between the warehouse and Taj - using existing 10dBi yagi and indoors are at the Croc's Lair shop - AP with integrated antennas. That will be the coverage are for now with the possibility of a few more sites later on.

Telstra IP WAN

Almost 6mths in the making and the completion date is within sight. The last week I have sorted out the issue Mooloolaba Travel were having with running their VPN to Galileo running over the new Telstra WAN. It simply wasn't connecting. Discovered that the 1-to-1 NAT addressing that was also used for IPSec and PPTP traffic did not like traffic coming from the direction of Mooloolaba. So they changed it to a Many-to-1 NAT and things are looking good apart from frequent dropouts of the IPSec VPN which I believe are due to the 'keepalives' being blocked - so I turned them off and I will see what happens.

WhaleOne still hasn't been changed over. I had a 3hr crack at it today with all the information I had scrounged from Telstra and the 'net in general. Turns out that their NextG account hasn't yet been set up for use with the IP WAN. What happens is that the APN Name changes from 'telstra.internet' to 'telstra.corp' and this dictates the gateway used for the connection. Before this can occur the SIM card/Account needs to be set up so that the calls are allowed through or something along those lines anyway - the request goes to the Telstra Mobile team. Hopefully it will be fixed up by Tuesday next week.

Myth Frontend

Ages ago I talked about building a mythbox out of a Epia M10000. Well last weekend I finally did it. Epia support is quite good with linux these days, it was easy to set up Mythbuntu on the box and have it connect to the existing Soltek Qbic 3401 Mythbox. It runs okay apart from stutter after a channel change for about 5 minutes - I think more RAM will fix that.

I might upgrade my PC and use the old bits to build a better backend someday.

Shoveling Data

2007-12-01T13:20:00.001+10:00

Because the 'zoo keeps many operations such as Graphic Design and Marketing in-house it generates a considerable amount of data on a day-to-day basis. It's a challenge to keep all this centralised and backed up. What I have done to achieve ample storage with basic redundancy is use a 'front end' NAS (Network Attached Storage) combined with a 'back end' NAS located elsewhere from the front end serving as the primary backup/archive.

The users access the front end NAS directly and generally work from its shares. This will change in future as I intend to access it as a iSCSI mount on a server. This NAS is a standard box housing JBOD and runs OpenFiler - you have probably read about it here earlier. The performance of this NAS is fairly ordinary but since its the network that presents the bottleneck its not something to be concerned about at this stage.

The back end NAS is a purpose built NAS from a company called Thecus, the N5200. It houses up to 5 SATA disks and supports RAIDs 0 thru to 10. I've set up this particular one with 5 x 750GB disks with RAID5. This provides enough space to backup the front end NAS at maximum capacity - about 2.5TB total.

I'm currently backing up the front end NAS via rsync to the Thecus. I had to find the rsync 'module' to install on the Thecus first as it doesn't support it by default however it wasn't a difficult process.

I will consider a Thecus 1U4500 NAS to go with a future Novell OES2 server - mounting it as an iSCSI volume for localised e-mail/data archiving. This will probably use another 5200 for backups.

Overall this provides us with a sizable storage pool at a very reasonable cost. I would like to implement a proper SAN however our needs aren't that great at this stage and a single form of redundancy appears to be acceptable to management. I'll always plan for the upgrade though.

Changed Service Providers once again

2007-11-29T21:33:00.001+10:00

Changed from Internode to Comcen - mainly so I can access the Australian PIPE peering network unmetered again. There's simply too much useful content to be had. I'm on the Comcen 10+10GB 8000/386kbit/sec plan now.

Also because I was using Internode's NodePhone service I changed VoIP providers too. So I went with MyNetFone. No problems thus far - quality is fine and the price is good.

The transition from Internode to the latter services took no more than two days. I cancelled the accounts on Monday noon and was changed over and functioning by Thursday noon.

OpenDNS Media Release

2007-11-29T18:12:00.001+10:00

Thought I'd link this here:

Australia Zoo Conserves Bandwidth, Enjoys 100 Percent Network Uptime with OpenDNS

A new era of IT focused policy

2007-11-25T19:27:00.000+10:00

Firstly I am excited at Australia's change of government, especially with fellow Queenslanders Kevin Rudd as PM and Wayne Swan as Treasurer. I'm sure they will do an excellent job of managing the country and will prove that QLD can produce what it takes. I found that this article gave words to my thoughts.

Now one of the most interesting topics of this election that I found was that Information Technology became a key policy area for all parties. Sure in the last few elections it was there but it was always one of those background things that parties flaunted so they could look 'modern.' However this time IT was front and center.

Labor brings with it the proposal to roll out a national broadband network that will provide Fibre to the Neighborhood (FTTN) to 98% of the population, and provide broadband services to the remaining 2% via wireless and other alternatives. More information about the policy is available on the ALP website here.

As part of my usual blogging I will be covering the progression of this new broadband network. To provide an industry perspective by offering my opinions.

For starters I believe the catalyst for this initiative is the selling of Telstra without splitting it up. In my opinion it was wrong to sell Telstra as a single entity. The government should have split Telstra's retail and wholesale operations first and kept the infrastructure at the very least. This would have provided exactly what this new ALP policy is going to provide - a state owned communications infrastructure that is accessible to anyone for cost price plus future investments.

If you want a example of a similar exercise you need not look further than Queensland where the state government have maintained ownership of the electricity network and generators while privatising the retail operations.

So what's done is done and the government won't get Telstra back. So this leaves the task of producing an alternative communications infrastructure in the hope of removing Telstra's monopoly particularly in rural areas where the private sector can not easily move into.

Going into detail; a key term used by the ALP in this policy is "in partnership with the private sector" as this is crucial for the success of this project. There are many organisations that are interested in the removal of Telstra's monopoly - reducing wholesale rates for network access suits anyone who is trying to compete in a tight market. With this support the ALP gain the additional funds necessary to build out the network nation wide - it has been said that it could cost an extra $20 billion on top of the initial $4.7 billion (I'll find the source of that claim...).

However I believe the policy is lacking detail with regard to Australia's international connections. Our links to other continents are already reaching their peak capacity and to increase the available bandwidth throughout the country will only make matters worse. These links are certainly not cheap and to simply rely on supply and demand to upgrade them will shift the monopoly from the access and distribution back on to the core supply. Its akin to building a million homes and running power to them all and not bother building more power plants until the demand, thus price, is through the roof.

Interesting times ahead.

Links:
Broadband: an election issue (Whirlpool, 23rd November 2007)
Federal Labor Broadband Policy (ALP Website, 21st March 2007)

Posting profile pic here so I can link it :/

Now we're getting somewhere

2007-11-24T09:30:00.001+10:00

It has been a hectic few weeks for me. Major projects for the period include the preparations for Steve Irwin Day, the roll out of the new Telstra Next IP managed WAN and the slowly but surely deployment of a campus wireless network.

I'm also feeding in various nifty services into the Zoo network - such as a OpenFire Jabber server, Twiki wiki, One or Zero Helpdesk and a few other things. Although I'm going to hold off on their deployment to users until I have a LDAP directory of some sort in place for all these things to authenticate against.

I updated the firmware on my Sony Ericsson P1i too - the difference in performance and stability is night and day - and it was pretty good to begin with! The Opera browser and unified messaging apps have been improved quite a bit. I'm actually encoding TopGear episodes to 3gp format on my MythTV box and watching them during my lunch breaks on it - I didn't think I'd be using it like that. nb: I could just watch the xvid/dvix encoded eps but they're a tad large...

Steve Irwin Day
Steve Irwin Day went well as far as the web servers went - they handled a doubling of traffic without a hitch. I will be sad to see the replication servers go, they've done their job well and I'm kinda proud of them. I'm starting an upgrade of massive proportions of the two main web servers this week - hopefully once I'm done they'll be more than capable of handling the load without the need for replicas. More on that later.

Telstra NextIP
I've cut everything over to the zoo's shiny new Telstra Next IP WAN (to use their marketing spin). Speeds are good, response times are awesome. I'm also using Telstra's Proxy Caches too as they're very snappy and are used by many - plus there's a discount on data used through them apparently.

As part of the WAN, each SHDSL connected site has a managed Cisco 1801 router and SHDSL TA, some yumcha device. The NextG connection is as per usual, but when it is connected it has a L2TP into the WAN and thus has access to all the same routes as the other sites. I've set up Mikrotik routers at each site including the NextG connection - the Zoo has 2 x Yawarra WRAP1-2s in a rack enclosure still, Mooloolaba has a Yawarra WRAP1-1 with wireless and WhaleOne has a Mikrotik RB133 in an indoor enclosure.

Campus Wireless
Not too much progress - the Admissions indoors area now as its own AP and the Taj (crocosium buidling) offices have coverage too. I'm waiting for a sparky to run new cabling to key points so I can locate a few more APs in good coverage areas. Also waiting on a $1k order for various bits and pieces from the RFShop so I can start making up tails and prepare the splitters for installation. Also getting some antennas from them that are cheap and appear to have excellent performance - looking forward to trying them out.

Symbol/Motorola WS5100

2007-11-06T07:37:00.001+10:00

If you use these Wireless switches and are still running pre-3.0 firmware, UPDATE! Huge changes made and I suspect it's Motorola weaving its magic. I had all sorts of issues running with Spectralink VoWiFi sets and coverage - updated firmware to 3.0.2.0 and everything is happy now.

Other benefits of the firmware is that the CLI now mimics Cisco's IOS in many ways and the Java/Web interface is greatly improved - information is readily available and the controls make sense...

It's changed my view on this kit I was almost about to turf it in exchange for some Cisco gear or even Mikrotik (but I didn't really want to configure each AP individually).

It's been more than a month

2007-10-23T19:23:00.001+10:00

So I guess I should post an update.

Ubuntu Gutsy Gibbon
I've updated the laptop from Fesity Fawn to Gutsy Gibbon the other day. Apart from fiddling with the sound card settings and installing the XGL version of XServer with the ATi RADEON restricted drivers it all went okay... Do you still want to use Linux? But really, I'm actually noticing a improvement in performance from Feisty - hard to put a finger on it but it just seems more "responsive". Maybe all that talk about improving the Linux kernel for the desktop has paid off and they've included various tweaks to better handle the scheduling?

The 3D desktop is nicely done - just a good sprinkling of bling to improve the overall functionality of the interface, not just a bunch of unstable power sapping eye candy like what Beryl throws up. Also it seems to be more friendly with this ATi Mobility X600 as beryl used to crash, although that could be all sorts of UE causing that - re-enforces the case that its better if such things are included in the distro.

So you can say that I'm happy with the direction of Ubuntu and if you're going to use it, upgrade to Gutsy.

Telstra Next IP WAN
All the ground work has been done and apparently this is the week for the physical rollout of the new service. Given all the problems I have been having with the wireless link into the Zoo it will be a breath of fresh air to have a decent high capacity link to use. I still have some reservations about it being presented as a "BDSL" (Business Digital Subscriber Line, Telstra's term) service though - it will be using a dedicated copper service from the exchange (not a RIM) and I think it might also be using two pairs - sounds very much like a G.SHDSL service which isn't bad at all.

I think they will be using Cisco 18xx series routers. Meaning that all these 857 and 877 routers will soon become redundant. I would like to use them elsewhere but much of the other locations the Zoo is concerned with are too remote for a DSL service. Maybe I can give them to staff who require a reliable VPN back into the Zoo WAN.

So I guess I should be devising a service change-over plan. Such things used to be second nature to me when working for AccessPlus but I have to think about it since my mind isn't buried in networks 90% of the time. I'm sort of split between networks, web servery things and desktop. I'll make an effort to get something together shortly, I guess I am still waiting on IP information from Telstra anyway.

Also once the WAN is in place I will re-establish my trusty webserver.

Novell Work Group Suite - Small Business Edition
I'm pretty keen on seeding Novell in this environment. It's just crying out for some decent administration and services.

The Small Business Workgroup suite is basically Novell OES2 with Groupwise. E-mail is a great introduction for a 'green fields' environment so I'm hoping management are impressed by it and won't be afraid to continue our foray into Novell. I'm hoping the new OES2 and Groupwise lives up to the hype, at least the cost is reasonable.

Facebook
Even though I have a blog I caved in and created a facebook account. I can see why it's addictive to many people - it's pretty bloody easy to find others on it and all of those widgets that seem to spread throughout profiles like some kind of nasty virus.

I predict two things to happen - someone will be assaulted/murdered as a direct result of having all their information in one easy to reach location and/or some kind of worm weaves its way through it, data-mining all the way and people start finding credit accounts opening up in their name. Call me a pessimist.

Election'07
Federal elections are fun aren't they, seeing what policies will be bandied about and then abolished as soon as they are elected. At least we get to see the pollies get their arse kicked on more than one occasion. As far as my political persuasion goes; I'm center-left which puts me on top of the Democrats, somewhat close to the Greens with Labor as the primary. That as close to a preference you'll get from me.

Here's a few blogs and things that I use to follow the election:

I'll try and keep this updated more often.

Posting from a P1i

2007-09-05T17:16:00.002+10:00

Well, it'll take some getting used to but it isn't that hard.

Recording RouterOS's IP Accounting Data

2007-08-25T11:53:00.000+10:00

There are a number of ways to gather data from a Mikrotik RouterOS based router. The easiest would be it's 'Accounting Web Access' feature where you can go to http://routeros_addr/accounting/ip.cgi and view a list of ip pairs similar to a basic netflow output.

Using this feature I wrote the below perl scripts to collect the data into a DB file. To keep things reasonable I set it record the data per the hour, meaning my smallest unit of measurement is hourly. While I could have simply used a MySQL database to dump the data into, I wanted to maintain a level of portability and simplicity - it sucks having to install/configure/run a fully fledged RDBMS just to view some basic data usage statistics.

The first script is used to gather the data from the MT router and store it into the db_file, the second script uses GD::Graph to produce bar charts using the data stored in the db_file. I'll be writing more scripts that dumps the contents of the db_file into a .xls spreadsheet for manual reports - handy for tracking down heavy users and to use as evidence if there are any ISP account discrepancies.

Apologies for the untidy code and the lack of formatting. Blogger doesn't provide any 'code markup' function and I cbf'd looking for alternatives. I'll fix it up when I can.

Example graph output (graph.pl 8 hours):

gather.pl:


#!/usr/bin/perl -w

use strict;
use LWP::Simple;
use MLDBM 'DB_File';
use Time::Local;

my $arg0 = $ARGV[0];
my $arg1 = $ARGV[1];

my $ip_accounting_url="http://<routeros ip>/accounting/ip.cgi";
my $accounting_mldbm_data_db = "~/accounting_data.mldbm";

tie my %h, 'MLDBM', $accounting_mldbm_data_db or die $!;

my ($timestamp) = &time_stamp();
my $epoch = time();
# print "\n Epoch set to: $epoch\n";

&gather_ip_accounting($ip_accounting_url);

sub gather_ip_accounting {
  my $url = $_[0];
  my ($src, $dst, $bytes, $packets, $src_usr, $dst_usr);

  foreach my $line (split(/\n/, get($url))) {
      ($src, $dst, $bytes, $packets, $src_usr, $dst_usr) = split(" ", $line);

      if ($dst && $dst =~ /(192\.168\.)|(10\.2\.)|(172\.16\.)/){
          my $h_dst = $h{$dst . "_" . $timestamp};
          $h_dst->{dst} = $dst;
         # $h_dst->{src} = $src;
          $h_dst->{bytes} += $bytes;
          $h_dst->{packets} += $packets;
         # $h_dst->{src_usr} = $src_usr;
          $h_dst->{dst_usr} = $dst_usr;
   $h_dst->{epoch} = $epoch;
          $h{$dst . "_" . $timestamp} = $h_dst;
      }
  }
}

# use Data::Dumper;
# print Dumper(%h);

untie %h;

sub time_stamp {
my ($d_t);
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);

      $year += 1900;
      $mon++;
      $d_t = sprintf("%4d-%2.2d-%2.2d %2.2d:00:00",$year,$mon,$mday,$hour,$min,$sec);
      return($d_t);
}

graph.pl:


#!/usr/bin/perl -w

use strict;
use LWP::Simple;
use MLDBM 'DB_File';
use Time::Local;
use GD::Graph::bars;

my ($num_values, $period_type);
if ($ARGV[0] && $ARGV[1]) {
if ($ARGV[0] =~ /\d+/) {
 $num_values = $ARGV[0];
}
else {
 print "\nIncorrect value supplied for number of units\n";
 exit;
}

if ($ARGV[1] =~ /(hours)|(days)|(months)/) {
 $period_type = $ARGV[1];
}
else {
 print "\nIncorrect value supplied for type of units\n";
 exit;
}
}
else {
print "\nUsage: period units\nPeriod: The number of values\nUnits: Hours, Days, Months\n\n";
exit;
}
print "\n Gathering $num_values $period_type worth of data from db!\n";

my $epoch = time();

my $accounting_mldbm_data_db = "~/accounting_data.mldbm";
my $graph_image_file = "~/accounting_data_" . $num_values . "_" . $period_type . "_" . $epoch . ".png";

tie my %h, 'MLDBM', $accounting_mldbm_data_db or die $!;



#else {
# &print_period_summary($arg0, $arg1);
#}

my($graphvalues, @graphvalues_tmp);
my $period_total = 0;
my $i = 0;
   while ($i <= $num_values) {
# print "$i\n";
       @graphvalues_tmp = &print_total($i, $period_type);
my $data = $graphvalues_tmp[0];
my $epoch = $graphvalues_tmp[1];
my $HMS = &epoch_to_MDHMS($epoch);
push @{$graphvalues->[0]}, $HMS;
push @{$graphvalues->[1]}, $data;
$period_total = $period_total + $data;
$i++;
   }

my $graph = GD::Graph::bars->new(85*$num_values, 300);
$graph->set(
   x_label     => "$period_type (latest towards the left) Period Total: $period_total",
   y_label     => 'Mbytes',
   title       => "Total Mbytes (Over $num_values $period_type)",
   transparent => '0',
   show_values => '1',
   bar_spacing => '2',
) or warn $graph->error;

my $image = $graph->plot($graphvalues) or die $graph->error;

open(IMG, ">$graph_image_file") or die $!;
binmode IMG;
print IMG $image->png;

# use Data::Dumper;
# print Dumper($graphvalues);

untie %h;

sub print_total {
my $h_total=0;
my ($h_row, $h_column, $h_bytes, $h_dst);

my ($num, $period) = @_;
my ($epoch_start, $epoch_end) = &epoch_period($num, $period);

for my $h_row ( keys %h ) {
 if ($h{$h_row}{epoch} >= $epoch_start && $h{$h_row}{epoch} <= $epoch_end) {           $h_bytes = $h{$h_row}{bytes};    $h_dst = $h{$h_row}{dst};           $h_total = $h_total + $h_bytes;   }     }   my $formatted_total = sprintf("%.3f", $h_total/1024/1024);   return($formatted_total, $epoch_start); }  sub epoch_period {  my ($past_count, $period) = @_;  my ($epoch_period_start, $epoch_period_end);   my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);  my ($start_hour, $end_hour);   if ($period eq "hours") {   $start_hour = $hour-$past_count;   $end_hour = $hour-$past_count;  }  elsif ($period eq "days") {   $mday = $mday-$past_count;   $start_hour = '00';   $end_hour = '23';    }  elsif ($period eq "months") {   $mon = $mon-$past_count;   # $mday = '00';   # $hour = '00';  }   $epoch_period_start = timelocal(00,00,$start_hour,$mday,$mon,$year);  print "Start: $epoch_period_start\n";  $epoch_period_end = timelocal(59,59,$end_hour,$mday,$mon,$year);  print "End: $epoch_period_end\n";    # print "SUB EPOCH_PERIOD: $epoch_period_start, $epoch_period_end\n";  return($epoch_period_start, $epoch_period_end); }   sub epoch_to_MDHMS {  my $epoch = $_[0];   my ($sec, $min, $hour, $mday, $mon) = (localtime($epoch))[0,1,2,3,4];   my $mdhms = $mon+1 . "-" . $mday . " " . sprintf("%02d", $hour) . ":" . sprintf("%02d", $min) . ":" . sprintf("%02d", $sec);  return($mdhms); }