Rather than a security bug in WordPress exposing my internal network to the world I’ve decided to move everything to static pages. I don’t really post here any more, so it’s not like I need everything that WordPress offers and there are lots of places which will host static pages for either free or very cheap, so why not try it out? Here’s what I decided on and how I did it.

Hosting

I considered the following:

1. Creating a virtualhost in an already existing Apache set up
2. Uploading a static copy of these pages to S3
3. Uploading a static copy of these pages to Github Pages
4. Moving to a cheaper cloud provider

I ruled out 1, I don’t want any public facing services on my otherwise private servers. I know enough to know that I don’t know anything about securing web servers.

I think #2 and #3 are largely the same in the end. As you have probably worked out by now, I decided that Github was the one I would try. This was decided entirely on cost. S3 would have cost me a couple of pennies a month, Github was free.

I also considered spinning up a new instance in Digital Ocean or Linode (referral link, get $100 credit) which would have been about 20% cheaper for a t2.micro equivalent. Github pages are free though, and get a decent CDN behind them. WordPress on a small server is slow without doing work.

Getting the data out of WordPress

I used the Simply Static plugin for WordPress to export the pages and images to HTML. It worked pretty well, and I would recommend it. Simply Static produced a zip file with everything in. I unzipped it and copied everything to the root of a new Github repo, committed it, pushed it, and it worked.

Some tips for preparing your blog for export:

• Disable comments on all your posts. This is a faff as you have to do it on each existing post. In the end I followed this guide: https://themeisle.com/blog/disable-comments-in-wordpress/
  See section 3 on how to remove the ability to comment on existing posts.
• Remove all the meta links etc from the footer and sidebar. I did this through the WordPress “Appearance” -> “Customise” menu to remove widgets and links as required.

Adding new content in the future

I have zero interest in learning some new static blogging platform or writing posts in HTML or Markdown or using some automated jiggery pokery. I really like WordPress for it’s WYSIWYG writing interface.  I write about one post every two years. I intend to switch off the instance hosting WordPress until such time as I need to write a new post, then I’ll switch it on, write the post and export it using Simply Static. This will cost me a few cents for the disk storage, and zero effort.

I came down in favour of Pi Hole for a couple of reasons but the deciding factor was that Pi Hole felt a bit more open and that it was built on top of dnsmasq which allowed me to reuse config for TFTP which netboots some devices which needed it.

Now that I’ve been using Pi Hole for a few months I have a much better understanding of its limitations and the big one for me is performance. Not the performance when servicing DNS requests but performance when querying the stats data, when reloading block lists and when enabling and disabling certain lists. I suspect a lot of the problems I was having is down to flaky SD cards.

I fully expect that for most people this will never be a problem, but for me it was an itch I wanted to scratch, so here’s what I did:

Through the actually quite generous Amazon Alexa AWS Credits promotion I have free money to spend on AWS services, so I spun up a t2.micro EC2 instance (1 vCPU, 1GB RAM – approx £10 a month) running Ubuntu.

I installed Pi Hole on that instance along with Wireguard which connects it back to my local network at home. I used this guide from Linode to get Wireguard set up.

The Pi Hole running in AWS hosts the large block files and is configured with a normal upstream DNS server as its upstream (I’m using Cloudflare).

I use three Ad block lists:

• OISD: https://dbl.oisd.nl/
• Wally3k: https://v.firebog.net/hosts/static/w3kbl.txt
• Polish Filters Team: https://raw.githubusercontent.com/PolishFiltersTeam/KADhosts/master/KADhosts_without_controversies.txt

Pi Hole running on a t2.micro instance is really speedy. I can reload the block list in a matter of seconds (versus minutes on the Pi) and querying the stats database no longer locks up and crashes Pi Hole’s management engine FTL.

The Pi Hole running on my LAN is configured to use the above AWS based Pi Hole as its upstream DNS server and also has a couple of additional block lists for YouTube and TikTok.

This allows me use Pi Hole on a Pi as the DHCP server on my LAN and benefit from the GUI to configure things. I can quickly and easily block YouTube when the kids have done enough and won’t listen to reason and the heavy lifting of bulk ad blocking is done on an AWS EC2 instance. The Pi on the LAN will cache a good amount of DNS and so everything whizzes along quickly.

Pi Hole on the LAN has a block list of about 3600 hosts, whereas the version running in AWS has over 1.5 million.

All things considered I’m really happy with Pi Hole and the split-load set up I have now makes it even easier to live with. I would like to see an improved Pi Hole API for enabling and disabling specific Ad lists so that I can make it easier to automate (e.g. unblock YouTube for two hours on a Saturday morning). I think that will come in time. The split-load set up also allows for easy fallback should the AWS machine need maintenance – it would be nice to have a “DNS server of last resort” in Pi Hole to make that automatic. Perhaps it already does, I should investigate.

Why not just run Pi Hole on a more powerful computer in the first place? That would be too easy.

If you fancy trying out Pi Hole in the cloud or just playing with Wireguard you can get $100 free credit with Linode with the Late Night Linux referral code: https://linode.com/latenightlinux

With the recent news about the ISP UK association proposing Mozilla as “Internet villain of the year” for enabling DNS over HTTPS (and subsequently changing their mind and dropping the whole category of villain of the year. Good move I think.) I figured it was probably about time that I looked at enabling DoH at home.

Cloudflare have a suite of open source tools called cloudflared which has, among other things, a DNS over HTTPS proxy. By default it points at their 1.1.1.1 service, but you can change that if you want to. Note, at the time of writing there is a bug which seems to stop Google’s DNS service working. If you’re looking to stop people seeing your DNS traffic then Google probably isn’t the right DNS service to use anyway.

I already have dnsmasq running as my DNS server and I have quite a lot of config which I wanted to keep (e.g. adblocking) so I figured I would add cloudflared’s proxy-dns alongside dnsmasq and have dnsmasq use proxy-dns as it’s upstream server, which would in turn pass the DNS lookups to 1.1.1.1 over HTTPS. dnsmasq would then cache the results locally.

So far, so good. I’d built cloudflared on my desktop to test it, now I wanted to move it on to the Raspberry Pi, run it as a service, and ideally have a package so that I didn’t have to mess around rebuilding it in loads of places if I wanted to move to a different box.

Make a snap

Making a snap of proxy-dns would give the the package I wanted, and could allow me to run proxy-dns as a daemon with two words in the YAML. Snapcraft’s build service would build me an ARM binary, as well as loads of others, for free.

I downloaded the source for cloudflared and added three files:

1. A snapcraft.yaml which describes how to build cloudflared and sets it to be run as a daemon
2. A configure hook which lets me set some config options
3. A launcher script which sets the config at run time

None of these are very complicated, as you can see. Hat-tip to Popey for help with the snapcraft.yaml.

The I pushed these back to my project on GitHub and added that project to the Snapcraft.io build service. Now, whenever I push a new change back to GitHub the snap will get rebuilt automatically and uploaded to the store! All I would need to do is a snap refresh and I’d be upgraded to the latest version. All my requirements solved in one place.

How to use the snap

If your Pi is running snapd, it’s dead easy (e.g. Ubuntu MATE or Ubuntu Core):

sudo snap install cloudflaredoh --edge

The snap is currently in the edge channel, meaning it’s not ready for the main stage just yet. Once I’ve spent a bit more time on it, I will move it to stable.

sudo snap set cloudflaredoh address=127.0.0.1
sudo snap set cloudflaredoh port=5053

Configure proxy-dns to listen on 127.0.0.1. If you want it to answer DNS queries from other computers on your network try either the IP address of the box, or just 0.0.0.0 to listen on all interfaces. It will also configure proxy-dns to listen on port 5053. If you want it to answer DNS queries from other computers on your network, use the default DNS port of 53.

sudo snap get cloudflaredoh

This will show you the currently set config options.

sudo snap restart cloudflaredoh

Restart proxy-dns and use the new config.

Now you can use something like nslookup to query the DNS server and make sure it’s doing what you expected.

10 Steps To DNS-over-HTTPS

1. Get a Raspberry Pi
2. Download Ubuntu Core and write it to an SD card
3. Put the SD card in your Pi and boot it
4. Set up the network on Ubuntu Core (tip: register for an Ubuntu One account first)
5. sudo snap install cloudflaredoh
6. sudo snap set cloudflaredoh address=0.0.0.0
7. sudo snap set cloudflaredoh port=53
8. sudo snap restart cloudflaredoh
9. Configure your client’s DNS server as the IP address of your Pi
10. Have a cup of tea

Update 2019-08-01

I’ve got a new Github repo set up with an improved snapcraft.yaml which pulls directly from the upstream project. I’m aiming to get this hooked up to the Snapcraft build service so that we can package the latest version automatically. More on this later. In the meantime, you can clone this and build the latest version yourself:

https://github.com/8none1/cloudflarednsproxy

Like a lot of weekend hardware hackers I thought it would be fun to build a wireless temperature sensor. I could use it as a feedback mechanism for my Raspberry Pi heating controller, I could create graphs of temperatures in different rooms (everyone loves a graph right?), and with a little bit of forward planning I could make a fairly useful Arduino breakout board which could be used for lots of other fun wireless projects.

I’ve been through four iterations of the design before I finally found something that worked. I could have avoided some of this if I’d have spent more time testing on breadboard, but then I wouldn’t get to order colourful PCBs!

The Brief

Cheap

In common with a lot of my other weekend projects, this is very likely to be a white elephant, so it needs to be cheap to start with.  There are lots of excellent wireless sensors already available, the Moteino is well regarded.  Using that as a baseline, could I build something similar for less than £5 per unit?  The answer is “nearly”, see the Bill Of Materials below.

Use a ready made microcontroller board

I didn’t want to have to design a board which I would then have to solder an ATmega328/P to myself. Laying out that board would be hard, plus I would have to spec and solder all the supporting components. I could have done that with through-hole components I suppose, but then the board would have been massive, and I very much doubt it would have actually worked out any cheaper.  Instead, I decided that a better idea was to solder on a ready-made Arduino clone, specifically the Pro Mini. I buy them from this supplier on eBay and I’ve never had any problems. They are 3v / 5v switchable, very reliable and turn up quickly.  Power consumption was going to be an important metric and running them off of a 3V supply would be essential.

Use a ready made radio module

As with the MCU, I don’t want to be laying out a radio board or soldering surface mount components.  I needed a ready-made radio module which was cheap, readily available and had low power requirements.  I started off with the NRF24L01.  It had a small footprint, cost around 99p a unit, had a built in antenna and was supported by the RadioHead library.  Initial testing on breadboard showed that the range of these devices would be sufficient for my house.  I designed the first two revisions of the sensor board around this radio (and a CR2032, see the battery section below).  Initial current draw was a bit too high for the CR2032, but I could have worked around that by using AAA batteries instead – but more annoying was that the range of the radios was not at all reliable.  The 2.4GHz spectrum is very noisy and the addition of two baby monitors since the original breadboard test hasn’t helped, but in testing on the PCB I found these radios to be pretty much useless.  I also tried using the version with the power amp as the central transceiver, this helped a little bit but they were still failing to get at least 50% of their transmissions to the other end, even over a couple of meters.  Other people have reported really good range with these boards, but crucially those range tests were done outside.  It’s my considered opinion that these radios and Wifi in the house do not co-exist.

So I gave up with those, and tried the XL4432-SMT.  This is based on the Si443x chipset from Silicon Labs.  They’re readily available on eBay for around £.170 each, so nearly twice the price of the NRF24L01.  They’re well supported by the RadioHead library, can run down to 1.8V, have low current draw when on (virtually nothing when in stand-by), support a wide frequency range around the 433MHz ISM band and in range testing they out-performed the NRF24L01 by a long way.  The downside is that these pre-made boards use 1.27mm pitch / castellated connections.  I had to design an Eagle part to interface with them, but that wasn’t really too hard.  See below for links to the parts I made.  Another drawback was the antenna; being a much lower frequency means they need a much longer antenna so I would need to find a project box which could hold them.
The Si443x also has a temperature sensor and a wake up timer on board.  However, reading the errata from Silicon Labs it seems that the WUT is actually broken and the temperature sensor was returning very strange results.  The datasheet says that you need to calibrate the temperature sensor so I tried doing this but go nowhere fast, and so I opted to use a 1wire sensor instead.

The other option for a radio would have been an ESP8266.  You can get ready made boards cheap on eBay, and I could have done away with a separate MCU altogether but the power consumption of these devices is just too great for a project which needs to run of a couple of batteries for a year.

Run for a long time on batteries

What’s the point in a wireless sensor if you have to plug it in to the mains.  This project must run from batteries, and those batteries need to last a long time – having to change the batteries every few months would quickly get boring and the sensors would be left doing nothing.  Obviously having the MCU go in to a sleep state between runs is going to be necessary, plus a radio which has modest power requirements when running.  We can further reduce the power needs by making sure that the “work” that the sensor has to do is done as quickly as possible.  The default 1wire temperature sensor code you find will typically takes around 2700msec to read.  That’s a very long time.  I changed the code a bit by hard-coding the hardware address of the sensor on the board and by only reading two bytes of temperature data (good for 0.5 degree accuracy).  More information can be found below.

From a size perspective a CR2032 battery looks very appealing.  Some early testing made me think that they would work fine but in real life I had a lot of problems.  In hindsight I think I can put most of the problems down the Brown Out Detector on the Pro Mini being set to 2.8V, more on that in a moment.

UPDATE:  Whoops.  This post has been sat in drafts for nearly 2 years.  Guess it’s not getting finished then.  I’m posting this in case the above is interesting.  Topics that I wanted to cover but haven’t are:

• Fit neatly in a box
• Read the temperature
• Be extensible
• Reading 1wire temperature sensors quickly
• Lower the BOD to 1.8V
• Bill Of Materials

We can get some reasonable insights from the Stack Overflow Developer Survey, but I’m keen to really dig down in to the Ubuntu community specifically.

When I was chatting with Barton George a few weeks back he expressed the same interest; what are people doing with the Sputnik machines from Dell? We want to learn more about the sorts of software projects that you’re working on so that we can make the Ubuntu developer experience as good as possible.

To that end we put together the Ubuntu Developer Desktop Survey to help us understand more about what you’re doing and how you’re doing it. This survey is aimed primarily at people who are using Ubuntu to develop software targeting any platform. It doesn’t matter if you do that at work, at home, at school – if you’re building software then we’re glad to hear from you. To be clear: this doesn’t mean we’re abandoning our mantra of Ubuntu being for human beings, software developers are human beings too. Right now I want to get a better view in to what software developers are doing.

The survey will close on Friday 31st May 2019 and we will publish the results very shortly afterwards. We’ll then follow that up with some further analysis and some ideas as to how this will influence the desktop product roadmap in the future. Please take this opportunity to help shape Ubuntu for the better.

You can find the survey here:

http://desktopdevelopersurvey2019.ubuntu.com/

You don’t need to register or sign in to complete the form.

I’m going to start a weekly newsletter style update to keep people abreast of what’s been going on with Ubuntu Desktop as we move to GNOME Shell and build the foundations for 18.04 LTS.  Here’s the first instalment:

Friday 19th May 2017

GNOME

We’re on to the last few MIR (https://wiki.ubuntu.com/MainInclusionProcess) reviews for the packages needed to update the seeds in order to deliver the GNOME desktop by default.
We still have some security questions to answer about how we deal with updates to mozjs/gjs in an LTS release (where mozjs has a support period of 12 months but we need to offer support for a full five years). This is being looked at now, but for 17.10 we are set.
We are aiming to have the seeds updated next week, and this will be the first milestone on the road to a fantastic GNOME experience in 17.10 Artful.

We’ve been fixing bugs in the Ambiance & Radiance themes to make them look crisp on GNOME Shell.
http://www.omgubuntu.co.uk/2017/05/install-improved-ambiance-gnome-theme

We’ve also triaged over 400 GNOME Shell bugs in Launchpad to allow us to more easily focus on the important issues.

We have been working on removing Ubuntu’s custom “aptdaemon” plugin in GNOME Software in favour of the upstream solution which uses PackageKit. This allows us to share more code with other distributions.

LivePatch

https://www.ubuntu.com/server/livepatch

LivePatch delivers essential kernel security updates to Ubuntu machines without having to reboot to apply them. As an Ubuntu user you can sign up for a free account.
We’re working on integrating LivePatch in to the supported LTS desktops to provide a friendly way to setup and configure the service.
This week we started to investigate the APIs provided by the LivePatch services so we can report LivePatch activity to the user, obtain an API key on behalf of the user & set up the service. Work has also started on the software-properties-gtk dialogs (aka Software & Updates in System Settings) to add the options required for LivePatch.

QA

Added upgrade tests from Zesty to Artful for Ubuntu and flavours. Working on making all these tests pass now so that everyone will have a solid and reliable upgrade path.
Work is being done on the installer tests. This will extend the current installer tests to check that not only has the install completed successfully but that all desktop environment is working as expected, this had previously been covered with manual tests.

Package Updates

• GStreamer is now at 1.12 final in 17.10.
• Chromium: stable 58.0.3029.110, beta 59.0.3071.47, dev 60.0.3095.5
• LibreOffice 5.3.3 is being tested.
• CUPS-filters: 1.14.0
• Snapd-glib: 1.12

Snaps

More GNOME applications are being packaged as Snaps. There is still some work to do to get them fully confined and fully integrated into the desktop. We’re working on adding Snap support to Gtk’s Portals to allow desktop Snaps to access resources outside their sandbox.
We will start tracking the Snaps here:
https://wiki.ubuntu.com/DesktopTeam/GNOME/Snaps

In the news

Interview with Ken VanDine on the GNOME Desktop in Ubuntu:  http://www.omgubuntu.co.uk/2017/05/ubuntu-switch-to-gnome-questions-answered

There’s also a survey running to get feedback on some extensions which could be shipped with Ubuntu Desktop: http://www.omgubuntu.co.uk/2017/05/ubuntu-desktop-gnome-extensions-survey-1710

This was picked up by the Linux Unplugged podcast as their headline story: http://www.jupiterbroadcasting.com/114701/that-new-user-smell-lup-197/

First of all you need to make sure your skill is set up to use Login With Amazon.  I’ve covered this for Smart Home skills here but it works just the same for normal skills.

You also need to make sure your skill is configured to use the scopes “profile” and “postal_code“.  This is done in the Configuration tab in the developer console for your skill:

The Interaction Model for this skill is as follows:

{
  "intents": [
    {
      "intent": "AMAZON.HelpIntent"
    },
    {
      "intent": "AMAZON.CancelIntent"
    },
    {
      "intent": "AMAZON.StopIntent"
    },
    {
      "slots": [
        {
          "name": "Options",
          "type": "Options"
        }
      ],
      "intent": "GreetIntent"
    }
  ]
}

The Custom Slot type “Options” is:

First name
Last name
Post code
email address
user id

And the Sample Utterances are:

GreetIntent Tell me my {Options}
GreetIntent What is my {Options}

And here’s the code (also on GitHub here):

#!/usr/bin/python

import requests

def build_speechlet_response(title, output, reprompt_text, should_end_session):
    return {
        'outputSpeech': {
            'type': 'PlainText',
            'text': output
        },
        'card': {
            'type': 'Simple',
            'title': "Greeter",
            'content': output
        },
        'reprompt': {
            'outputSpeech': {
                'type': 'PlainText',
                'text': reprompt_text
            }
        },
        'shouldEndSession': should_end_session
    }


def build_response(session_attributes, speechlet_response):
    return {
        'version': '1.0',
        'sessionAttributes': session_attributes,
        'response': speechlet_response
    }
    

def get_user_info(access_token):
    #print access_token
    amazonProfileURL = 'https://api.amazon.com/user/profile?access_token='
    r = requests.get(url=amazonProfileURL+access_token)
    if r.status_code == 200:
        return r.json()
    else:
        return False

def handle_end():
    return build_response({}, build_speechlet_response("Session ended", "Goodbye!", "", True))

def launch_request(access_token):
    session_attributes = {}
    card_title = "Welcome"
    if access_token is None:
        speech_output = "Your user details are not available at this time.  Have you completed account linking via the Alexa app?"
        reprompt_text = ""
        should_end_session = True
    else:
        user_details = get_user_info(access_token)
        if user_details is None:
            speech_output = "There was a problem getting your user details."
            should_end_sesion = True
        else:
            print user_details
            speech_output = "Hello "+user_details['name'].split(" ")[0]+"!  I know all about you now.  We can be friends!"
            reprompt_text = "What can I tell you about your user information?  First name, last name, email address or postcode?"
            should_end_session = False
    return build_response(session_attributes, build_speechlet_response(
        card_title, speech_output, reprompt_text, should_end_session))

def intent_request(intent_request, access_token):
    intent = intent_request['intent']
    intent_name = intent['name']
    
    if intent_name == "GreetIntent":           
        if access_token is not None:
            user_details = get_user_info(access_token)
            if user_details is None:
                query_type = False
            else:
                query_type = intent['slots']['Options']['value']
        else:
            return handle_end()
        
        if query_type == "post code":
            speech_output = "Your post code is "+user_details['postal_code']+"."
        elif query_type == "first name":
            speech_output = "Your first name is "+user_details['name'].split(" ")[0]+"."
        elif query_type == "last name":
            speech_output = "Your last name is "+user_details['name'].split(" ")[1]+"."
        elif query_type == "email address":
            speech_output = "Your email address is "+user_details['email']+"."
        elif query_type == "user id":
            speech_output = "Your user id is "+user_details['user_id']+"."
        else:
            speech_output = "Something went wrong.  Goodbye."
        card_title = "What I know about you"
        return build_response({}, build_speechlet_response(card_title, speech_output, None, True))
        
    if intent_name == "AMAZON.HelpIntent":
        print "Help intent"
        card_title = "No help available!"
        speech_output = "Sorry, I can't help you."
        return build_response({}, build_speechlet_response(card_title, speech_output, None, True))
        
    if intent_name == "AMAZON.CancelIntent" or intent_name == "AMAZON.StopIntent":
        card_title = "Session Ended"
        speech_output = "Good bye!"
        return build_response({}, build_speechlet_response(card_title, speech_output, None, True))
    


def lambda_handler(event, context):
    print event
    try:
        access_token = event['context']['System']['user']['accessToken']
    except:
        access_token = None
    if event['request']['type'] == "LaunchRequest":
        return launch_request(access_token)
    elif event['request']['type'] == "IntentRequest":
        return intent_request(event['request'],access_token)

The important part is:

event['context']['System']['user']['accessToken']

Once that is available to your Lambda script then you know that the user has done the OAUTH account linking, and then you can query the Amazon APIs for the user’s info.

I’m not proud of this code, so I will tidy it up in to a better demo at some point and update this post accordingly.

Here’s a quick video of what it does:

Edit 2: Looks like the tinyupload file has gone missing too.  Try this one:
https://drive.google.com/file/d/1EBJ-s9TIMKM0JVmZFWRk2bKeGxR9srFv

I got a Cisco 7941 off eBay.  This is a phone which was £400 when new (some time around 2004) but can now be picked up for about £10.  These phones went End Of Sale in January 2010, so even if mine was one of the last phones to roll off the production line it’s still about 7 years old but it’s still working perfectly.  A testament to the good build quality of these phones, and perhaps the previous owner’s careful handling.

Since these devices are no longer supported many companies will be getting rid of them (or probably already have) so there should be some bargains to be had for phone geeks.

Q: Does the Cisco 7941 work with Asterisk?
A: Yes.  You need to load the SIP firmware (the focus of this post) or chan-sccp (out of scope for this post but I’ll check it out at some point).

Q: Does the Cisco 7941 work with SIP?
A: Yes.  You need to flash the correct firmware though.

Q: Is it really hard to get working?
A: No.  If you’re comfortable with Linux and a few command line tools.  And assuming you already have Asterisk set up.

Q: Is a lot of the information on the web about how to set up the 7941 wrong?
A: Yes.  There is a lot of confusion about config files (the 7940 and 7941 use different ones).

Q: Will you tell us how you got your phone to work?
A: Yes!  However – this is what works for me.  You will need to tweak the config in places.

The steps to getting this phone working as a SIP extension on Asterisk on Ubuntu / Raspberry Pi:

1. Set up a TFTP server
2. Download the SIP firmware from Cisco
3. Flash the phone with the firmware via the TFTP server
4. Configure the SIP extension in Asterisk
5. Write the config files for the phone and upload them via the TFTP server
6. Make a call!
7. Optional Extras
   1. Dial plan
   2. Ring tones
   3. Dial tones
   4. Wallpaper
   5. Telephone Directory
8. Final Tip

Set up a TFTP Server

The phone will download it’s firmware and config via TFTP.  It needs to download it’s config on every boot, so you will always need a TFTP server running.  I think that if the TFTP server is unavailable it will just use the previous config, so it’s possible that you can get away without it, but I haven’t tried.  My recommendation is that you install dnsmasq.  It’s a small and full featured DNS server which also includes a DHCP & TFTP server which are easy to configure and it’s almost certainly packaged for your distro.  You should also (temporarily) disable any other DHCP servers on your local network so that dnsmasq is the only thing offering DHCP addresses.  This will simplify the process of getting the phone to find the TFTP server, since with dnsmasq it will all be automatic.  If you later re-enable your original DHCP server, say on your router, then you will need to configure it to give out the address of the dnsmasq TFTP server and disable DHCP on dnsmasq.  In my opinion, if you’re going to be running a Cisco IP phone on your network you’d be better off moving all DHCP to dnsmasq.

The full configuration of dnsmasq it’s out of scope for this doc, but in a nutshell you need these in your dnsmasq config:

• Set up a DHCP range

dhcp-range=192.168.1.1,192.168.1.100,24h

• Enable the TFTP server

enable-tftp

• Set the TFTP path

tftp-root=/home/<your user>/tftp  (or whatever works for you)

Download the SIP Firmware from Cisco

Usually Cisco require a valid support contract before you can download anything useful from their website, but it seems that since these phones are now out of support they have offered up the firmware free of charge.  You do still need to register an account to download the files.  At the time of writing the latest version is 9.4.2 SR 3 dated 14th February 2017 – so bang up to date, even though these phones are end-of-life.  Bizarre, but good for us.  Thanks Cisco!

Go here: https://software.cisco.com/download/type.html?mdfid=280083379&catid=280789323

Follow the link to the SIP software.

You want to download the “SIP firmware files only”

Unzip that file into the root of your TFTP server (the location you set in the previous step).  You should have 8 files in there:

apps41.9-4-2ES26.sbn
dsp41.9-4-2ES26.sbn
term41.default.loads
cnu41.9-4-2ES26.sbn
jar41sip.9-4-2ES26.sbn
term61.default.loads
cvm41sip.9-4-2ES26.sbn
SIP41.9-4-2SR3-1S.loads

This is everything you need to reflash your phone to the latest SIP firmware.  Now you need to get the phone to reboot in to firmware download mode.

Flash the phone with the firmware via the TFTP server

1. Unplug the phone from the power.  Make sure that the network cable is still connected (unless you’re using using PoE).
2. Plug the power back in and hold down the # key
3. Eventually you will see the “line” lights start to flash orange.  It might take a couple of minutes to get to this stage, don’t give up, just keep holding down #
4. When the line lights are flashing type 123456789*0#  This will start firmware download mode.
5. The screen will go black for a moment and then go through the process of getting an IP address and connecting to the TFTP server
6. Once connected to the TFTP server the software download will start
7. The phone will reboot once download is complete and present you with an “Unprovisioned” message on the screen.  This is good news!  The phone firmware has now been updated.

I put together a video showing this process.  It’s not very interesting but it will give you an idea of what to expect.  The actual downloading of the firmware section has been sped up 3X.

Configure the SIP extension in Asterisk

Now you need to configure the SIP extension in Asterisk.  Do this as per any other SIP extension, but bear this important piece of information in mind:  The Cisco 7941 can only deal with 8 character passwords, so keep your SIP authentication secret to 8 characters.

While you’re in Asterisk configuration mode, take a moment to note down these bits of information as well (in Advanced SIP settings in FreePBX):

• RTP Port range, start and end.
• Bind Port (probably 5060)

Write the config files for the phone and upload them via the TFTP server

Please take the time to read this section fully,  this is the part that is most troublesome.  The Cisco 7941 is very picky about it’s config file and even a small mistake will stop the phone from working.  These settings are specific to the 79×1 series of phones running at least version 8.x of the firmware.  If your phone is not a 79×1 and/or is not running v9.x.x of the firmware then these settings are not for you.

Once the phone has loaded it’s firmware and booted, it will go looking for a file called SEP<PHONE MAC ADDRESS>.cnf.xml.  So if the MAC address of your phone is 11:22:33:44:55:66 then the config file needs to be named SEP112233445566.cnf.xml.  This file needs to be in the root of your TFTP server.

You will see mention of a file called XMLDefault.cnf.xml.  If you’ve only got a few phones, don’t worry about this, you don’t need it.

So here is a config file which is about as minimal as I can make it:

<device>
    <deviceProtocol>SIP</deviceProtocol>
    <sshUserId>cisco</sshUserId>
    <sshPassword>cisco</sshPassword>
    <ipAddressMode>0</ipAddressMode>

    <devicePool>
        <dateTimeSetting>
            <dateTemplate>D/M/Ya</dateTemplate>
            <timeZone>GMT Standard/Daylight Time</timeZone>
            <ntps>
                <ntp>
                    <name>#IP ADDRESS OF AN NTP SERVER#</name>
                    <ntpMode>Unicast</ntpMode>
                </ntp>
            </ntps>
        </dateTimeSetting>

        <callManagerGroup>
            <members>
                <member priority="0">
                    <callManager>
                        <ports>
                            <ethernetPhonePort>2000</ethernetPhonePort>
                            <sipPort>#SIP PORT NUMBER FROM YOUR ASTERISK SERVER#</sipPort>
                        </ports>
                        <processNodeName>#IP ADDRESS OF YOUR ASTERISK SERVER#</processNodeName>
                    </callManager>
                </member>
            </members>
        </callManagerGroup>
    </devicePool>

    <sipProfile>
        <sipProxies>
            <registerWithProxy>true</registerWithProxy>
        </sipProxies>
        <sipCallFeatures>
            <cnfJoinEnabled>true</cnfJoinEnabled>
            <rfc2543Hold>false</rfc2543Hold>
            <callHoldRingback>2</callHoldRingback>
            <localCfwdEnable>true</localCfwdEnable>
            <semiAttendedTransfer>true</semiAttendedTransfer>
            <anonymousCallBlock>2</anonymousCallBlock>
            <callerIdBlocking>2</callerIdBlocking>
            <dndControl>0</dndControl>
            <remoteCcEnable>true</remoteCcEnable>
        </sipCallFeatures>

        <sipStack>
            <sipInviteRetx>6</sipInviteRetx>
            <sipRetx>10</sipRetx>
            <timerInviteExpires>180</timerInviteExpires>
            <timerRegisterExpires>3600</timerRegisterExpires>
            <timerRegisterDelta>5</timerRegisterDelta>
            <timerKeepAliveExpires>120</timerKeepAliveExpires>
            <timerSubscribeExpires>120</timerSubscribeExpires>
            <timerSubscribeDelta>5</timerSubscribeDelta>
            <timerT1>500</timerT1>
            <timerT2>4000</timerT2>
            <maxRedirects>70</maxRedirects>
            <remotePartyID>true</remotePartyID>
            <userInfo>None</userInfo>
        </sipStack>

        <autoAnswerTimer>1</autoAnswerTimer>
        <autoAnswerAltBehavior>false</autoAnswerAltBehavior>
        <autoAnswerOverride>true</autoAnswerOverride>
        <transferOnhookEnabled>false</transferOnhookEnabled>
        <enableVad>false</enableVad>
        <preferredCodec>g711ulaw</preferredCodec>
        <dtmfAvtPayload>101</dtmfAvtPayload>
        <dtmfDbLevel>3</dtmfDbLevel>
        <dtmfOutofBand>avt</dtmfOutofBand>
        <alwaysUsePrimeLine>false</alwaysUsePrimeLine>
        <alwaysUsePrimeLineVoiceMail>false</alwaysUsePrimeLineVoiceMail>
        <kpml>3</kpml>
        <natEnabled>false</natEnabled>
        <phoneLabel>#PHONE NAME#</phoneLabel>
        <stutterMsgWaiting>0</stutterMsgWaiting>
        <callStats>false</callStats>
        <silentPeriodBetweenCallWaitingBursts>10</silentPeriodBetweenCallWaitingBursts>
        <disableLocalSpeedDialConfig>false</disableLocalSpeedDialConfig>
        <startMediaPort>#RTP START PORT#</startMediaPort>
        <stopMediaPort>#RTP END PORT#</stopMediaPort>

        <sipLines>
            <line button="1">
                <featureID>9</featureID>
                <featureLabel>#EXT NUM#</featureLabel>
                <proxy>USECALLMANAGER</proxy>
                <port>#SIP PORT#</port>
                <name>#EXT NUM#</name>
                <displayName>#EXT NAME#</displayName>
                <autoAnswer>
                    <autoAnswerEnabled>2</autoAnswerEnabled>
                </autoAnswer>
                <callWaiting>3</callWaiting>
                <authName>#SIP AUTH NAME#</authName>
                <authPassword>#8 CHAR PASSWORD#</authPassword>
                <sharedLine>false</sharedLine>
                <messageWaitingLampPolicy>1</messageWaitingLampPolicy>
                <messagesNumber>#VM NUM#</messagesNumber>
                <ringSettingIdle>4</ringSettingIdle>
                <ringSettingActive>5</ringSettingActive>
                <contact>#EXT NUM#</contact>
                <forwardCallInfoDisplay>
                    <callerName>true</callerName>
                    <callerNumber>true</callerNumber>
                    <redirectedNumber>false</redirectedNumber>
                    <dialedNumber>true</dialedNumber>
                </forwardCallInfoDisplay>
            </line>

            <line button="2">
                <featureID>9</featureID>
                <featureLabel>#EXT NUM#</featureLabel>
                <proxy>USECALLMANAGER</proxy>
                <port>#SIP PORT#</port>
                <name>#EXT NUM#</name>
                <displayName>#EXT NUM#</displayName>
                <autoAnswer>
                    <autoAnswerEnabled>2</autoAnswerEnabled>
                </autoAnswer>
                <callWaiting>3</callWaiting>
                <authName>#SIP AUTH NAME#</authName>
                <authPassword>#8 CHAR PASSWORD#</authPassword>
                <sharedLine>false</sharedLine>
                <messageWaitingLampPolicy>1</messageWaitingLampPolicy>
                <messagesNumber>#VM NUM#</messagesNumber>
                <ringSettingIdle>4</ringSettingIdle>
                <ringSettingActive>5</ringSettingActive>
                <contact>#EXT NUM#</contact>
                <forwardCallInfoDisplay>
                    <callerName>true</callerName>
                    <callerNumber>true</callerNumber>
                    <redirectedNumber>false</redirectedNumber>
                    <dialedNumber>true</dialedNumber>
                </forwardCallInfoDisplay>
            </line>
        </sipLines>

        <voipControlPort>#SIP PORT#</voipControlPort>
        <dscpForAudio>184</dscpForAudio>
        <ringSettingBusyStationPolicy>0</ringSettingBusyStationPolicy>
        <dialTemplate>dialplan.xml</dialTemplate>
    </sipProfile>

    <commonProfile>
        <phonePassword></phonePassword>
        <backgroundImageAccess>true</backgroundImageAccess>
        <callLogBlfEnabled>1</callLogBlfEnabled>
    </commonProfile>

    <loadInformation>SIP41.9-4-2SR3-1S</loadInformation>
    <vendorConfig>
        <disableSpeaker>false</disableSpeaker>
        <disableSpeakerAndHeadset>false</disableSpeakerAndHeadset>
        <pcPort>0</pcPort>
        <settingsAccess>1</settingsAccess>
        <garp>0</garp>
        <voiceVlanAccess>0</voiceVlanAccess>
        <videoCapability>0</videoCapability>
        <autoSelectLineEnable>0</autoSelectLineEnable>
        <webAccess>0</webAccess>
        <spanToPCPort>1</spanToPCPort>
        <loggingDisplay>1</loggingDisplay>
        <loadServer></loadServer>
        <sshAccess>0</sshAccess>
    </vendorConfig>

    <versionStamp>001</versionStamp>
    <networkLocale>United_Kingdom</networkLocale>
    <networkLocaleInfo>
        <name>United_Kingdom</name>
        <uid>64</uid>
        <version>1.0.0.0-4</version> 
    </networkLocaleInfo>

    <deviceSecurityMode>1</deviceSecurityMode>
    <authenticationURL></authenticationURL>
    <servicesURL></servicesURL>
    <transportLayerProtocol>2</transportLayerProtocol>
    <certHash></certHash>
    <encrConfig>false</encrConfig>
    <dialToneSetting>2</dialToneSetting>
</device>

Copy and paste this into a text editor and search and replace the following:

• #IP ADDRESS OF AN NTP SERVER#  –  with  –  the IP address of an NTP server
• #SIP PORT FROM YOUR ASTERISK SERVER#  –  with  –  the SIP port of your asterisk server is listening on.  Probably 5060
• #IP ADDRESS OF YOUR ASTERISK SERVER#  –  with  –  the IP address of your Asterisk server
• #PHONE NAME#  –  with  –  the text you want to appear at the top right of the phone screen
• #RTP START PORT#  –  with  –  the RTP port range start from the previous stage
• #RTP END PORT#’  –  with  –  the RTP port range end from the the previous stage
• #EXT NUM#  –  with  –  the Asterisk extension number as configured in the previous stage
• #SIP PORT#  –  with  –  the SIP port of your Asterisk server.  Probably 5060
• #EXT NAME#  –  with  –  the name you want to give this extension
• #SIP AUTH NAME#  –  with  –  the username for the SIP extension as configured in Asterisk
• #8 CHAR PASSWORD#  –  with  –  the password for the SIP extension as configured in Asterisk
• #VM NUM#  –  with  –  the number you dial for Voicemail.  Probably *98

Note that this config file has two lines configured.  If you just blindly search and replace you’ll end up with two extensions configured the same.

Some comments on what some of the XML tags do:

• ipAddressMode – 0 is IP v4 only. But this seems to have little effect.
• registerWithProxy – true – Registers the device with Asterisk, this allows incoming calls to be sent to the phone.  If you’re getting “Unregistered” message on the screen, check you have this set.
• featureId – 9 is SIP
• autoAnswerEnabled – 2 – 2 seems to be “off”
• webAccess – 0 – 0 is on (?!)
• sshAccess -0 – ditto
• versionStamp – bump this up every time you make a change.  Something like YYYMMDD001..2..3 etc
• networkLocale – United_Kingdom – sets the tones to UK, see the optional extras section for more info.
• transportLayerProtocol – 2 is UDP, 1 is TCP
• dialToneSettings – 2 is “always use internal dialtone”.  See option extras for more info.

Edit this file as necessary and then save it to the root of your TFTP server with the filename: SEP<MAC>.cnf.xml.  If your phone MAC address was aa:bb:33:44:55:66 then the filename would be: SEPAABB33445566.cnf.xml  Note that it’s case sensitive, letters in the MAC address should be in upper case the extensions should be in lowercase.  You can get the MAC address for the phone from the syslog on your dnsmasq server.

If your phone is still in “Unprovisioned” mode it will have been asking for this config file repeatedly.  Once you save the file you should see the phone reboot shortly afterwards.  It may download the firmware again for some reason, just leave it to get on with it.

Make a call!

If everything has worked you should see your extension listed on the right hand side of the screen near the buttons, and the name of the phone should appear at the top of the screen.  If the icon next to the line buttons is that of a phone without an x through it, then you’re probably good to go!  Press the line button and see if you get a dial tone.  If not, then check the phone logs:

• Press Settings
• Press 6
• Press 1

From these logs you should be able to tell if the phone has loaded your config correctly.  Errors about “updating locale” or “no trust list installed” can be ignored.  If there is a problem with the config file itself a generic error will be listed here.  If the phone won’t load the config file the most likely reason is that there is a typo in your XML file.  Good luck finding it.  You can SSH in to the phone to get more detailed logs and debugging information, but I haven’t tried this yet.  Google is your friend.

Optional Extras

Dial plan

The dial plan tells the phone how to process the digits you type and when to start sending the call.  Without a dial plan the phone simply waits a period of time for you to stop typing numbers before it decides you’re done and starts the call.  By using a dial plan you can reduce the amount of time spent waiting after you’ve finished keying in the number.  Here’s an example plan I’ve edited based on this post on Phil Lavin’s blog (Thanks Phil!) http://phil.lavin.me.uk/2012/11/united-kingdom-dial-plan-xml-for-cisco-phones/

<DIALTEMPLATE>
    <TEMPLATE MATCH="999" Timeout="0"/> <!-- Emergency -->
    <TEMPLATE MATCH="112" Timeout="0"/> <!-- Emergency -->
    <TEMPLATE MATCH="0500......" Timeout="0"/> <!-- Apparently 0500 is always 10 digits -->
    <TEMPLATE MATCH="0800......" Timeout="0"/> <!-- Apparently 0800 is always 10 digits -->
    <TEMPLATE MATCH="00*" Timeout="5"/> <!-- International, 00 prefixed. No fixed length -->
    <TEMPLATE MATCH="0.........." Timeout="0"/> <!-- UK 11 digit, 0 prefixed -->
    <TEMPLATE MATCH="26...." Timeout="0"/> <!-- My local STD numbers start 26 -->
    <TEMPLATE MATCH="\*.." Timeout="0"/> <!-- Asterisk *.. codes -->
    <TEMPLATE MATCH="\*98...." Timeout="0"/> <!-- Asterisk direct VM access *981234-->
    <TEMPLATE MATCH="1..." Timeout="0"/> <!-- Internal numbers -->
    <TEMPLATE MATCH="2..." Timeout="0"/>  <!-- Internal numbers -->
    <TEMPLATE MATCH="*" Timeout="5"/> <!-- Anything else -->
</DIALTEMPLATE>

Save this to the root of your TFTP server, named “dialplan.xml” (lowercase).

Ring tones

Everyone likes novelty ringtones.  You can find plenty of ringtones in a format which is compatible with your phone (raw format, 8000 Hz sample rate, 8 bit, ulaw, max 2 seconds).  These files need to be placed in to the root of your TFTP server.  I tried putting them in a sub-directory but it didn’t work.  Then you need to create a file called “ringlist.xml” also in the root of the server.  The format of this file is:

<CiscoIPPhoneRingList>
    <Ring>
        <DisplayName>#DISPLAY TEXT#</DisplayName>
        <FileName>#FILENAME#</FileName>
    </Ring>
    <Ring>
        <DisplayName>#DISPLAY TEXT#</DisplayName>
        <FileName>#FILENAME#</FileName>
    </Ring>
</CiscoIPPhoneRingList>

Filenames are case sensitive.  Once you’ve save this file, copy it to “distinctiveringlist.xml” as well.  This will allow you to set ring tones for the default ringer and different rings for each line.

Dial tones

By default the 7941 will have a psuedo North American dial tone.  This is annoyingly shrill (yes, it is).  By specifying a NetworkLocale in the phone config we can get it to load a different set of informational tones from a file stored in (per the example XML above) United_Kingdom.  In the root of the TFTP server create a directory called United_Kingdom.  In this directory you need to create a file called g3-tones.xml.  Bizarrely Cisco require you to have a support contract in order to download the correct tones settings for your country, despite giving the phone firmware away for free.  Go figure.  So this means I’m not going to paste the XML here.  If you search hard enough you’ll find an example g3-tones.xml file you can use as a base.  In our phone configuration above we told the phone to always use the internal dialing tone, so this means we only need to change the idial section of the tones file.  The magic numbers are:

• 31538
• -780
• 30831
• -973

Wallpaper

The phone comes with a single default wallpaper with horizontal lines on it.  This is easily replaced by your own designs with a simple PNG.   Create a directory in the root of the TFTP server called Desktops.  In here create another directory called 320x196x4.

In to this directory you need to place a “List.xml” file:

<CiscoIPPhoneImageList>
    <ImageItem Image="TFTP:Desktops/320x196x4/ubuntu-tn.png"
       URL="TFTP:Desktops/320x196x4/ubuntu.png"/>
</CiscoIPPhoneImageList>

The “-tn” in the file is a smaller thumbnail version of the larger image.  The PNGs need to be sized exactly 320×196 for the large and 80×49 for the thumbnail.  Here’s something to get you started:

Telephone Directory

You will have noticed that the phone has a “Directories” button and a “Services” button.  I haven’t managed to add an extra phone book to the Directories button yet although I think it’s certainly possible, just that the XML file refuses to do anything.  However, I have got a phone directory working on the Services button.

In the main phone config file there is a tag for “servicesURL”.  Point this to a web server on your local network which will serve up an XML file.  For example:

 <servicesURL>http://192.168.1.1/phone/directory.xml</servicesURL>

Assuming you are using Apache 2 to serve that XML file (or it could equally be a CGI script which generates the XML dynamically from a database such as the FreePBX phone book) the format looks like this:

<CiscoIPPhoneDirectory>
   <Title>Whizzy Towers</Title>
   <DirectoryEntry>
       <Telephone>1500</Telephone>
       <Name>Lenny</Name>
   </DirectoryEntry>
   <DirectoryEntry>
       <Telephone>1234</Telephone>
       <Name>Speaking Clock</Name>
   </DirectoryEntry>
</CiscoIPPhoneDirectory>

Important note:  You must tell Apache to serve those files as type “text/xml“.  “application/xml” will not work.

You can do this via your CGI script, or if you want to serve a static file add something like this to your Apache config:

 <Location /phone/>
     ForceType text/xml
 </Location>

Inside your VirtualHost section.

Final Tip

Watch /var/log/syslog on the machine running the TFTP server.  You’ll be able to see exactly what files the phone is asking for.  Bear in mind that it does ask for files it doesn’t strictly need, so don’t worry too much about file not found errors unless it’s one of the above.

Here’s a final video showing the boot up for a fully configured phone

The SSL certificate is easy enough to implement, but it’s a bit of a faff (renewing Let’s Encrypt certs, or paying for cert which needs you to deal with the certificate authorities, send in scans of your passport and other tedious red tape) but – in my opinion anyway – setting up an OAUTH server is even more of a faff.  If only there was some way to avoid having to do either of these things….

Using “Login With Amazon” as your OAUTH provider

Since you already have an Amazon account you can use “Login With Amazon” as your skill’s OAUTH server and your normal everyday Amazon account as your credentials.  You’re only sharing your Amazon account data with yourself, and even then we can restrict it to just your login ID.  You don’t actually need to do anything with the OAUTH token once it’s returned since you’re the only user.  I mean, you could if you wanted to, but this HOWTO assumes that you’re the only user and that you don’t care about that sort of thing.  We are also going to assume that you have already created the Lambda function and the smart home skill or are familiar with how to do that.  This is a bit tricky because you can’t test your smart home skill on a real device until you’ve implemented OAUTH, and you can’t complete the OAUTH set-up until you’ve got the IDs from your Lambda function and skill.  If you haven’t written your skill yet, just create a placeholder Lambda function and smart home skill to be going on with.

Much of this information is available from the official Amazon instructions available here: https://developer.amazon.com/public/community/post/Tx3CX1ETRZZ2NPC/Alexa-Account-Linking-5-Steps-to-Seamlessly-Link-Your-Alexa-Skill-with-Login-wit. What follows is a rehash and slight reorganisation of that doc which is hopefully a bit easier to follow.

1. Create a new Login With Amazon Security Profile

From the Amazon Developer Site, go to Apps & Services -> Login With Amazon.  Or click https://developer.amazon.com/lwa/sp/overview.html

Click “Create a New Security Profile”.  Fill out the form along these lines:

and hit Save.

You should see a message along the lines of “Login with Amazon successfully enabled for Security Profile.”

Hover the mouse over the cog icon to the right of your new security profile and choose “Security Profile”.

Copy your “Client ID”  and “Client Secret” and paste it in to a notepad.  You’ll need this again shortly.

2. Configure your skill to use Login With Amazon

Back in the Developer Console, navigate to the Configuration page for your skill.  (Click on your skill, then click on Configuration).  You need to enable “Account Linking” and this will then show the extra boxes discussed below.

In to the “Authorization URL” box you should put:

https://www.amazon.com/ap/oa/?redirect_url=

and then copy the Redirect URL from further down the page and append it to the end of the Authorization URL.  For example:

https://www.amazon.com/ap/oa/?redirect_url=https://layla.amazon.com/api/skill/link/1234ABCD1234AB

As far as I can tell Layla is for UK/Europe and Pitangui is for the US.  Use the appropriate one for you.  Also, keep a note of the redirect URL in your notepad, you will need this again later.

In to the “Client Id” box paste your client id from step 1.

You can leave “Domain List” blank for now.

For “Scope” I suggest you use:

profile:user_id

This will give your Alexa Skill access to a minimal amount of information about you from Amazon, in this case just a user_id.  That user ID is unique to your app so can’t be used by other apps or to identify that user elsewhere.  Since you don’t really have any customers for your skill, only you, there is no reason to provide access to any other information.

Further down the page you need to configure the Grant Type:

Select an “Auth Code Grant”

Set the “Access Token URI” to:

https://api.amazon.com/auth/o2/token

and in to “Client Secret” paste your secret from step 1.

You must include a link to your “Privacy Policy URL“.  Since you are the only person who cares you could host a blank file somewhere, or maybe link to a Rick Astley video on YouTube?

Finally hit Save.

3. Link Login With Amazon back to your Skill

Head back to the Login With Amazon page: https://developer.amazon.com/lwa/sp/overview.html

Hover over the cog of your Security Profile and choose Web Settings:

In to the “Allowed Return URLs” box paste your Redirect URL from step 2 and hit save.

4.  Login to Amazon from your skill and do the OAUTH dance

From the Alexa app on your phone navigate to your new Smart Home Skill and you see that it says “Account Linking Required“.

Click “Enable Skill” and you’ll be asked to login with your Amazon credentials:

Once you log in you should see a success message:

And you’re done.

Additional:  Here’s a post on how to read the users details from Amazon once they are linked:  /2017/03/retrieving-user-profile-data-from-login-with-amazon-on-alexa/

https://github.com/8none1/bravialib

bravialib

A Python library for talking to some Sony Bravia TVs, and an accompanying Alexa Skill to let you control the TV by voice. If you like that sort of thing.

These scripts make use of the excellent Requests module. You’ll need to install that first.

bravialib itself

This is a fairly simple library which allows you to “pair” the script with the TV and will then send cookie-authenticated requests to the TVs own web API to control pretty much everything. You can:

• Set up the initial pairing between the script and the TV
• Retrieve system information from the TV (serial, model, mac addr etc)
• Enumerate the available TV inputs (HDMI1,2,3,4 etc)
• Switch to a given input
• Enumerate the remote control buttons
• Virtually press those buttons
• Find out what Smart TV apps are available
• Start those apps
• Enumerate the available DVB-T channels
• Switch to those channels
• Send Wake On Lan packets to switch the TV on from cold (assuming you’ve enabled that)
• A couple of convenience functions

The library tries to hide the complexity and pre-requisites and give you an easy to use API.

I built this for a couple of reasons: 1. Because the TV had an undocumented API, and that tickles me 2. I quite fancied hooking it up to Alexa for lols

Most of the information about how to talk to the TV’s API came from looking at packet captures from the iPhone app “TV Sideview”.

There is a script called testit.py that will give you a few clues about how to use it, but it’s a bit of a mess. I’ve left a lot of comments in the code for the library which should help you.

Really, I think that this library should be imported in to a long-running process rather than be called every time you want to press a remote control button. On a Raspberry Pi, Requests can take a while (a couple of seconds) to import, and then bravialib pre-populates a few data sources, and all of that takes time, like about 20 seconds – so you really don’t want to use this library if you just want to fire a few remote control commands. Also – be aware that the TV takes a long time to boot and accept commands. From cold you’re talking about a minute maybe two, it’s really annoying.

The aforementioned long running process – bravia_rest.py

As the main library takes a while to start and needs a certain amount of data from the TV to work properly it really makes sense to start it up once and then leave it running as long as you can. The bravia_rest.py script does exactly that, and also exposes some of the functionality as a very crude REST interface that you can easily hook it in to various home automation systems.

First you need to add the IP address and MAC address (needed to turn on the TV the first time the script is run, it can be discovered automatically if you just power the TV on for a few minutes before you run the script).

Then run bravia_rest.py.

If this is the first time you have run it you will need to pair with the TV. You will be told to point your browser at the IP address of the machine where the script is running on port 8090 (by default). Doing this will make the script attempt to pair with the TV. If it works you will see a PIN number on the TV screen, you will need to enter this in to the box in your browser. After a few seconds, and with a bit of luck, pairing will now complete. This shouldn’t take too long.

If you are now paired, in your browser go to /dumpinfo for a view in to what the script knows about the TV.

Once everything is running you can POST to these URLs for things to happen (no body is required):

• /set/power/[on|off] – turns the telly on and off
• /set/send/<button> – e.g. mute, play, pause, up, down. See dumpinfo for all the key names.
• /set/volumeup/3 – turn the volume up 3 notches. You MUST pass a number, even it it’s just 1.
• /set/volumedown/1 – as above.
• /set/loadapp/<app name> – e.g. Netflix, iplayer. Again /dumpinfo will show you what apps are available.
• /set/channel/<channel> – e.g. BBC ONE, BBC TWO
• /set/input/<input label> – You need to have given your inputs labels on the TV, then pass the label here.

Hooking it up to Alexa

Now we can poke the TV through a simplified REST interface, it’s much easier to hook in to other things, like Alexa for example. Setting up a custom skill in AWS/Lambda is beyond the scope of what I can write up at lunchtime, I’m sure there are lots of other people who have done it better than I could. You’ll need to create a custom app and upload a Python Deployment Package to Lambda including my lambda_function.py script (see inside the Alexa directory), a secrets.py file with your info in it, a copy of the Requests library (you need to create a Python Virtual Environment – it’s quite easy) and possibly a copy of your PEM for a self signed HTTPS certificate. I’ve also included the skills data such as the utterances that I’m using. These will need to be adjusted for your locale.

You can read more about Python deployment packages and AWS here:http://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.htmlhttp://docs.aws.amazon.com/lambda/latest/dg/with-s3-example-deployment-pkg.html#with-s3-example-deployment-pkg-python

Here’s how it works:

1. You issue the command to Alexa: Alexa tell The TV to change to channel BBC ONE.
2. Your voice is sent to AWS (the green lines) decoded and the utterances and intents etc are sent to the Lambda script.
3. The Lambda script works out what the requested actions are and sends them back out (the orange lines) to a web server running in your home (in my case a Raspberry Pi running Apache and the bravia_proxy.py script). You need to make that Apache server accessible to the outside world so that AWS can POST data to it. I would recommend that you configure Apache to use SSL and you put at least BASIC Auth in front of the proxy script.
4. The bravia_proxy.py script receives the POSTed form from AWS and in turn POSTs to the bravia_rest.py script having done a quick bit of sanity checking and normalisation. The proxy and the rest scripts could live on different hosts (and probably should, there are no security considerations in either script – so ya know, don’t use them.)
5. bravia_rest.py uses bravialib to poke the TV in the right way and returns back a yes or a no which then flows back (the blue lines) to AWS and your Lambda function.
6. If everything worked you should hear “OK” from Alexa and your TV should do what you told it.

I could have put bravia_rest.py straight on the web an implemented some basic auth and SSL there – but I think this is something better handled by Apache (or whichever server you prefer), not some hacked up script that I wrote.

Caveats:

• It doesn’t deal with the TV being off at all well at the moment.
• I don’t know what happens when the cookies expire.
• I haven’t done much testing.
• I have no idea what I’m doing.

As a analytically minded person, is it worth getting Amazon Prime?

I signed up yesterday and spent the evening setting up the various services.  It was on offer with £20 off, it was the day before The Grand Tour came out, and I recently bought an Amazon Dot – well played Amazon, well played.

As I’m sure you’re aware Amazon Prime comes with a few bundled goodies to sweeten the deal. But are they actually useful? I think the real value is the next day delivery (if you order lots of things from Amazon), the online photo storage and perhaps the video. Everything else falls short of being good enough to be considered a real product in it’s own right.  Yes, I know that’s on purpose – but the marketing material would have you believe otherwise.  How about that.

The loan of a book a month from the Kindle Lending Library

Rating: 4/10 (based on going to an actual library being 8/10)
You’re not going to find many books you actually want to read in there. Browsing the catalogue from your desktop is virtually impossible and it’s been made that way deliberately. This is frustrating and annoying. Also annoying is that if you use the Kindle app on your phone you get nagged to upgrade to Kindle Unlimited all the bloody time.  You can share this with someone else on your household account, but they won’t thank you for it.

Prime Music

Rating: 6.5/10 (based on Spotify Premium being 10/10)
Better than I expected, but still full of annoyances. And again with the constant nagging to upgrade to Music Unlimited.
The selection is ok. It feels like they’ve taken the time to make it just-good-enough that you will use it but not good enough that you won’t consider upgrading.

You can’t share this perk with anyone else in your household, and since there’s no point in two people in your household having a Prime account each, you end up setting up another Chrome profile just to access the music service for managing play lists etc. Also annoying is that you can’t play music on an Echo or Dot and play music in the browser at the same time. This raises the question of what will happen if you have two Amazon Alexa devices on the same account and you try and play music on both. If you can’t (and I don’t know yet, perhaps someone can comment if they do) then one of the good features of Alexa is somewhat spoiled.  (Edit: some searching suggests that indeed you cannot listen to Prime Music on more than one device at at time.)

This service is a very good replacement for the kitchen radio but not a replacement for even a free Spotify subscription.  If you already have a premium Spotify account then this will be of no interest to you at all.

Prime Video

Rating: 7.5/10 (based on Netflix being 10/10)

Not so many nag screens here, but guess what, you still need to spend more money for the full experience.  It’s a bit galling to discover that some of the headline shows need to be paid for (Game Of Thrones for example).  But, there is some genuinely good exclusive content here, The Grand Tour for example.  I don’t see myself watching it more than Netflix but it’s worth, say, 3 quid a month of the £6.50 a month Prime subscription.

Installing on an iPhone was easy enough, but on Android it’s outrageously bad.  You have to install Amazon’s “Underground” app store, and to do that you have to enable “Unknown Sources”.  Unlol.  The app works fine, and you can uninstall Underground once it’s on there.  The app on my Sony TV is fine.  Prime Video doesn’t support Chromecast, which isn’t a surprise but is, you guessed it, annoying.

Next Day Delivery

Rating: 10/10 (based on going to the shops being 1/10)

I don’t order that much from Amazon, which is why I’m wondering if this is all worth it, but when I do having it turn up the next day for “free” is nice.

Unlimited Online Photo Storage

Rating: 9/10 (based on Google Drive being 7/10)

Google also offer unlimited photo storage, but critically they compress your photos.  The Amazon offering does not (based on MD5 sums for a picture selected at random).  There is also an excellent Linux cli util called acd_cli (https://github.com/yadayada/acd_cli).  You’ll want to exclude videos and other files from being uploaded because they will count towards your 5GB limit for other stuff.  Something like this:

acd_cli --verbose ul -xe mov -xe mp4 -xe ini . /Pictures

Don’t be surprised if this has a storage limit applied in the future.

Prime Early Access

Rating: 0/10 (based on normal Amazon shopping being 10/10)

Early access to the electronic jumble sale that is Black Friday.  Pointless.

Twitch Prime

Rating: 5/10 (based on not having it being 0/10)

Skip this if you don’t know what Twitch is.  With Prime you can subscribe to a channel for a month for free.  It’s a free way to support Twitch streamers you like, and you get some crappy game add-ons that you won’t ever use.

Summary

Amazon Prime is basically shareware from the late 90s.  It does do what they claim, but there are constant nags to spend money on the basis that all the good stuff is just out of reach.

The new Amazon Dot in the kitchen was not warmly welcomed by everyone at Whizzy Towers but having it play music on demand has changed that perception quite a lot in the last day.  I have saved the first episode of The Grand Tour to watch tonight, which I am looking forward to since the reviews have been good, and all my photos which had previously been backed up on to a USB HDD are now slowly making their way to the cloud.  I also ordered a £5 book yesterday which arrived today as promised.

However, the shortcomings have the feeling of being deliberate.  Which is more annoying than if the services were just a bit crap.

So all in all you’re getting what you pay for.  It’s not amazing value, but neither is it a total rip off.  Someone at Amazon has done their job well.  I would recommend you get it.

You might have already logged in and kicked the proverbial tyres.  If not I would urge you to do so.  Please take the time to install a couple of apps as laid out here:

http://insights.ubuntu.com/2016/10/13/unity-8-preview-session-in-ubuntu-16-10-yakkety-yak/

The main driver for getting Unity 8 in to 16.10 was the chance to get it in the hands of users so we can get feedback and bug reports.  If you find something doesn’t work, please, log a bug.  We don’t monitor every forum or comments section on the web so the absolute best way to provide your feedback to people who can act on it is a bug report with clear steps on how to reproduce the issue (in the case of crashes) or an explanation of why you think a particular behaviour is wrong.  This is how you get things changed or fixed.

You can contribute to Ubuntu by simply playing with it.

Read about logging bugs in Ubuntu here: https://help.ubuntu.com/community/ReportingBugs

And when you are ready to log a bug, log it against Unity 8 here: https://bugs.launchpad.net/ubuntu/+source/unity8

I was experiencing a very annoying problem trying to serve static HTML pages and CGI scripts from Apache 2 recently.  The problem manifested itself like this:

• Running the scripts on the server hosting Apache shows they ran in well under a second
• Connecting to the Apache server from the LAN, everything was fine and ran in under a second
• Connecting to the Apache server from the Internet, but from a machine known to my network, ran fine
• Connecting from an AWS Lambda script, suddenly there is a 20 second or more delay before getting data back
• Connecting from Digital Ocean, there is a 20 second delay
• Connecting from another computer on the internet, there is a 20 second delay

What the heck is going on here?

I spent time trying to debug my CGI scripts and adding lots more logging and finally convinced myself that it was a problem with the Apache config and not something like MTUs or routing problems.

But what was causing it?  It started to feel like like a DNS related issue since the machines where it ran fine where all known to me, and so had corresponding entries in my local DNS server.  But but but… I clearly had “HostnameLookups Off” in my apache2.conf file.  When I looked at the logs again, I noticed that indeed hostnames were being looked up, even though I told it not to.

Why?  Because I don’t know how to configure Apache servers properly.  At some point in time I thought this was a good idea:

Order deny, allow
Deny from none
Allow from all

But, there is no such thing as a “none” directive.  Apache interprets “none” as a host name and so has to look it up to see if it’s supposed to be blocking it or not, which causes a DNS lookup delays and hostnames to appear in your Apache logs.

Enlightenment came from here: http://kb.simplywebhosting.com/idx/6/213/article/

There is also a suggestion that inline comments can do the same thing here:  https://www.drovemebatty.com/wp/entries/11

Eleni has been making improvements to reduce the amount of visual effects that are seen while running in low graphics mode.  At a high level this includes things like:

• Reducing the amount of animation in elements such as the window switcher, launcher and menus (in some cases down to zero)
• Removing blur and fade in/out
• Reducing shadows

The result of these changes will be beneficial to people running Ubuntu in a virtual machine (where hardware 3D acceleration is not available) and for remote-control of desktops with VNC, RDP etc.

Low graphics mode should enable itself when it detects certain GL features are not available (e.g. in a virtualised environment) but there are times when you might want to force it on.  Here’s how you can force low graphics mode on 16.04 LTS (Xenial) :

1. nano ~/.config/upstart/lowgfx.conf
2. Paste this into it:

start on starting unity7
pre-start script
    initctl set-env -g UNITY_LOW_GFX_MODE=1
end script

3. Log out and back in

If you want to stop using low graphics comment out the initctl line by placing a ‘#’ at the start of the line.

This hack won’t work in 16.10 Yakkety because we’re moving to systemd for the user session.  I’ll write up some instructions for 16.10 once it’s available.

Here’s a quick video of some of the effects in low graphics mode:

I run a dnsmasq server on my router (which is a Raspberry Pi 2) to handle local DNS, DNS proxying and DHCP. For some reason one of the hosts stopped registering its hostname with the DHCP server, and so I couldn’t resolve its name to an IP address from other clients on my network.

I’m pretty sure it used to work, and I’m also pretty sure I didn’t change anything – so why did it suddenly stop? My theory is that the disk on the client became corrupt and a fsck fix removed some files.

Anyway, the cause is that the DHCP client didn’t know to send it’s hostname along with the DHCP request.

This is fixed by creating (or editing) /etc/dhcp/dhclient.conf and adding this line:

send host-name = gethostname();

Well, we’re making some changes to the Unity 7 Dash searches in 16.04 LTS.  On Unity 8 the Scopes concept has evolved into something which gives the user finer control over what is searched and provides more targeted results.  This functionality cannot be added into Unity 7 and so we’ve taken the decision to gracefully retire some aspects of the Unity 7 online search features.

What is changing?

First of all online search will be off by default.  This means that out-of-the-box none of your search terms will leave your computer.  You can toggle this back on through the Security & Privacy option in System Settings.  Additionally, if you do toggle this back on then results from Amazon & Skimlinks will remain off by default.  You can toggle them back on if you wish.  Further, the following scopes will be retired from the default install and moved to the Universe repository for 16.04 LTS onwards:

1. Audacious
2. Clementine
3. gmusicbrowser
4. Gourmet
5. Guayadeque
6. Musique

The Music Store will be removed completely for 16.04 LTS onwards.

Why now?

By making these changes now we can better manage our development priorities, servers, network bandwidth etc throughout the LTS period. We allow ourselves more freedom to make changes without further affecting the LTS release (e.g SRUs), specifically we can better manage the eventual transition to Unity 8 and not have to maintain two sets of scope infrastructure for the duration of the LTS support period of five years.

What about previous supported releases?

Search results being off by default will not affect previous releases or upgrades, only new installs (i.e. we will not touch your existing settings).  Changes to search results from Amazon & Skimlinks will also only affect 16.04 and beyond.  The removal of the Music Store will be SRU’d back to older supported releases and the option will be removed from the Dash.

When will this happen?

We’re preparing the make the changes in the archive, to Unity 7 and to the Online Search servers right now.  This will take a little while to test and roll out.  We’ll let you know once all the changes are in Xenial.

My home automation systems use two mechanisms for communication:  Ethernet (both wired and wireless) and 433MHz OOK radio.

433MHz transmitters are readily available and are cheap but unreliable.  Wifi enabled MCUs such as the ESP8266 are also cheap (coming in at around the same cost as an Arduino clone, a 433MHz transmitter and a bag of bits to connect them together), they are reliable enough but extremely power hungry.  If I can plug a project into the mains then I’ll use an ESP8266 and a mobile phone charger for power, if the project needs to run off batteries then a 433MHz equipped Arduino is the way I’ve gone.

Like most people playing with 433MHz radio I found reliability and range of the radio link to be super flaky.  I’ve finally got a more-or-less reliable set-up:

• A full wave dipole antenna at the receiver
• A high quality receiver from RF Solutions in place of the cheap ones which are bundled with transmitters. A decent receiver on eBay
• A big capacitor on the transmitter.  I saw the frequency and amplitude drifting massively during transmission.  Adding a 470µF cap helps.  Allow time for the cap to charge and the oscillator to stabilise, a few seconds delay seemed to do the trick.
• Using the RCSwitch library on the transmitter:

  • RCSwitch mySwitch = RCSwitch();

  • mySwitch.setProtocol(2); // Much longer pulse lengths = much better range?

  • mySwitch.setRepeatTransmit(20); // Just brute-force it!

With this setup I can get receive a 24bit number from an Arduino running off 2 AA batteries and a coiled 1/2 wave antenna from about 5 meters indoors through walls.  That’s still poor, but it does the job.  Increasing the voltage to the transmitter would probably help.

Once you have a reliable 433MHz receiver setup then you can also buy off the shelf 433MHz enabled home automation gizmos like this smoke alarm or these door sensors.  They have a set of jumpers inside where you can set an ID, which is essentially the same 24bit number that RCSwitch lets you transmit.  For what it’s worth I also have kite-marked smoke detectors in my house, but from the testing I’ve done with a bit of smoldering paper the cheap imports work just fine.

I couldn’t find a cheap Carbon Monoxide which also has 433MHz support so I thought I’d quickly hack one together out of this Carbon Monoxide detector and an Arduino clone and 433MHz radio:

You can barely notice it!

It’s certainly untidy, but it does the job.  If I had PCB facilities at home I’m fairly sure it could be made to fit inside the alarm, along with some more holes in the case for ventilation.

The premise is simple enough.  The Arduino is powered by the 3v3 regulator on the CO alarm PCB.  The cathode of the red alarm LED is connected to pin 2 of the Arduino as an external interrupt.  When the pin goes low the Arduino wakes up and sends it’s 24bit ID number over the radio which is picked up by the receiver which sends an SMS alert, switches the boiler off, etc.  I’ve connected the radio transmitter to directly to the 3 x AA batteries (4.5 volts) via a transistor which is switched by a pin on the Arduino.  In standy-by mode the additional equipment draws a fraction of a milliamp and so I’m not worried about draining the batteries faster.

As with the smoke alarms, this is not my only source of Carbon Monoxide detection.  I’ve yet to test it’s sensitivity.  This is considered to be a “well, if it works, and it turns the boiler off automatically then it’s certainly worth a go, but I’m not relying on it” project.

Today I have had a Launchpad account for ten years!

I got started out on this road around 1992.  I remember the day Stuart got a PC and installed Minix on it.  That box was biege, naturally, was about 3 feet square and constructed from inch thick iron plate.  Minix was totally alien when compared to the Acorn MOS and RISCOS powered machines I’d used until then, and absolutely intriguing.

A few years later at university I encountered VAX/VMS and Sun SPARCstations and The Internet and Surfers and Mozilla and a Gopher connected Coke machine.

Then out into the big wide world of work and run-ins with AS400 and RS/6000s running AIX.  During this time I started seeing more and more Red Hat in places where there once would have been the more established players, providing email and web servers.  The fascination with *nix was always there and I started using Red Hat at home for fun.

I quickly ran into frustrations with RPMs and Stuart, always a source of wisdom, suggested I try Debian.

Dpkg made my life a whole lot easier and I started using Debian as my default OS for everything. Pretty soon after that I found myself compiling kernels, modules and software packages because I needed or wanted something in a newer version.  Coupled with the availability of cheap unbranded webcams, sound cards, network cards, TV cards etc and a strong desire to make these things work with Linux meant that I had found a wonderful way to stay up until 4 in the morning getting more and more frustrated.  The phrase “I’m going home to play with the kernel” was frequently questioned by my boss Jeremy.  I wanted these things to work but was endlessly faffing about trying to make it happen.

Better call Stuart.

“You should try this new Debian based distribution called Ubuntu” he said.

So I did, and it just worked.  A box fresh kernel with all the goodies I needed already compiled in and an up-to-date GNOME desktop (I’d set my allegiances before trying Ubuntu so this was another tick in the box), not forgetting one of the brownest themes known to man.

And that was that.  Ubuntu worked for me and I was immediately a fan.

And here I am today, 10 years later, still running Ubuntu.  My servers run Ubuntu, all the desktops in my house run Ubuntu, I have an Ubuntu powered phone and soon I’ll have an Ubuntu powered Mycroft with which I’ll be able to control my Ubuntu powered things while wearing my Ubuntu T shirt and drinking tea (should that be kool-aid?) from my Ubuntu mug.

I salute my Ubuntu brothers and sisters.  Thanks for making all of this possible.

Over the years the bug lists for Unity 7 and Compiz have grown to become unmanageable.  To make sure we are focusing on the most important issues we have to do some serious tidying up of the bug lists and we need some help.

At the time of writing there are 2680 open bugs for Unity 7 (https://launchpad.net/ubuntu/+source/unity/+bugs) and 1455 for Compiz (https://launchpad.net/ubuntu/+source/compiz/+bugs) and 322 for nux, our graphical toolkit (https://launchpad.net/ubuntu/+source/nux/+bugs).

We’re proposing to cut this down to size with the following plan:

1. Close all bugs which relate to an unsupported release of Ubuntu.  We will do a manual review of the high heat bugs affecting unsupported releases first, but low heat bugs will most likely be closed by a robot.  The rationale is that the majority of these older bugs will have been fixed and that the original reporter is probably no longer affected by the bug and has forgotten to close it.  Plus manually screening each of these bugs cannot be done at this scale in a reasonable timeframe.  There will be some collateral damage which is an unfortunate but unavoidable side-effect.  Sorry if this affects you, but please do re-open the bug against a supported release.
2. Close all private apport bugs and review public ones with a view to closing them as well.  Apport is the automated error reporting tool which runs when it detects a crash.  It can open a private bug in Launchpad, private because stack traces might contain sensitive information which shouldn’t be public.  We have errors.ubuntu.com which can monitor crashes and provide a much clearer picture of which crashers are affecting numerous people and which are one-offs.  We will use errors.ubuntu.com instead of trying to triage the 250 or so bugs which fall in to this category.
3. Manually try to reproduce bugs and flag those which are still a problem.  This is where we need the most help.  We will create a list of bugs which need to be checked and then ask people to spend a few minutes trying to reproduce a chosen bug on 15.10.  If it’s still a problem then the tester would mark the bug as triaged or add a specific tag, or if it cannot be reproduced then they would mark the bug as Invalid.  This will give us a curated list of real bugs which we can then triage further to assess the impact and priority.  We will work through the triaged list in an agile manner and have regular meetings to review what has been fixed and decide on which bugs to focus on next.  By distributing this problem across many people we can get the job done in a reasonable time scale.

How you can help

First of all we need help in triaging the bug list.  You don’t need to be a superstar software developer to do this, everyone can help and contribute to Ubuntu.  You will need a Launchpad account though.  We will publish a link to a list of bugs in Launchpad for Unity 7 (and in time Compiz & Nux) which we think need manual checking.  The links are available at this wiki page:  https://wiki.ubuntu.com/BigDesktopBugScrub

Please choose a bug from this list and try to recreate it in 15.10.  If your main machine isn’t running 15.10 you could set up a virtual machine using VirtualBox.

1. Choose a bug from the list.  The heat metric is a good indication of which bugs are more important to a lot of people.  The list is sorted by heat so selecting one from somewhere near the top is a good starting point.  It’s possible that someone else will be working on the same bug as you so check the comments to see if anyone has added anything recently.
2. Can you recreate the bug?  There are a number of possible outcomes when you attempt to recreate the bug.  Listed below are the most common ones.  If you can’t match one of these categories directly, or don’t know what to do just leave the bug where it is and try a different one.
   1. No – I can’t understand from the report what the problem is:
      1. Add a comment along the lines of:  “Thank you for taking the time to report this bug.  Unfortunately we can’t work out how to recreate this bug from your description.  Please describe the process you go through to trigger this bug and then change the bug status to NEW.  See this page for more information. https://wiki.ubuntu.com/BigDesktopBugScrub”
      2. Set the bug status to Incomplete
   2. No – I’ve tried to but it doesn’t seem to be a problem any more:
      1. Add a comment along the lines of: “Thank you for taking the time to report this bug.  We have tried to recreate this on the latest release of Ubuntu and cannot reproduce it.  This bug is being marked as Invalid.  If you believe the problem to still exist in the latest version of Ubuntu please comment on why that is the case and change the bug status to NEW.”
      2. Set the bug status to Invalid
   3. Yes – it’s still a problem in 15.10:
      1. Add a comment along the lines of: “As part of the big bug review for 16.04 LTS I have tested this on 15.10 and the bug is still there.”
      2. Mark the bug as Triaged or, if you don’t have permission to do that, add the tag “desktop-bugscrub-triaged”
   4. Yes – but I don’t think it’s really a bug (perhaps a feature request):
      1. Add a comment along the lines of: “As part of the big bug review for 16.04 LTS I have tested this on 15.10 and the bug is still there.  I think this is a feature request rather than a bug.”
      2. Mark the bug as “Opinion”, or if you don’t have permission to do that, add the tag “desktop-bugscrub-opinion”
3. Thank you!  We’re one bug closer to perfection!
4. Lather, Rinse, Repeat

What happens next

Once we have a list of high quality, reproducible bug reports which are affecting many people we can start to chip away at them in a logical manner.  We will be using an Agile-like workflow:

1. Meet at the start of a “sprint” to discuss which of the most important bugs (importance will be decided on a mixture of bug heat and expert knowledge) will be working on during the sprint duration.  We will decide how many of the bugs we think are fixable in that sprint and take them into our backlog.  The backlog will be managed using Trello (https://trello.com/b/9YvUSYqq/unity-7).
2. The sprint will start and developers will take bugs (Cards) from the backlog to work on.
3. The card will move to the In Progress colum
4. If there is a problem the bug will move to the blocked column and these cards will be discussed at regular intervals during the sprint.
5. Once a bug is fixed it will move to the Review column.  A code review will be done and if everything is OK then the fix will be merged and automatically tested.  If there are problems it will move back to the In Progress column.
6. At the end of the sprint the fixes will be demonstrated and everyone will have a chance to spot any problems with the fix.  If there is a problem the card will go back into the Backlog for more work next sprint.  If everything is OK then the card is moved to Done and that bug is now fixed.
7. The next sprint will start and we will go back to step 1.

We will endeavour to do our reviews in a Hangout On Air so that everyone can join to see what progress is being made.  We will also use our IRC channel on Freenode #ubuntu-desktop.

Software developers who want to help

If you are a developer who wants to help fix the code as well as triage bugs please join us on IRC (#ubuntu-desktop on Freenode) and introduce yourself.  We can get you write access to the Trello board and invite you along to the Sprint planning and review meetings.  We’d love you to get involved.

Bug Squash Hours

In order to kick start the process we will be setting aside a few hours a week where a core Unity 7 developer will be available on IRC to help answer questions about bugs and we’ll be working through the list as well.  Feel free to ask for help or come and join us while we work through the bug list.  Exact schedule will be announced as soon as we know what it is.

Ubuntu Online Summit

We will have a session at UOS to review how the bug triage is going, discuss our tooling and policy on which bugs to auto-close etc.

Introduction

The Pro Mini V2 is an Arduino Pro Mini clone available on eBay for, typically, £1.50.  The version I buy is adjustable between 5v and 3.3v and has an ATmega 328 clocked at 8 MHz.  It’s an ideal board for development of IoT remote sensors and great for playing with and learning about the Arduino development environment.

Here’s a link to the version I buy and know works: 3V Pro Mini 2 Arduino Clone

When you want to put a sensor in a remote location the last thing you want to do is have to run a power cable to it. I’ve experimented with solar with generally poor results so battery operation is the obvious solution.  While Li-ion batteries offer higher energy density the sweet spot still seems to be the good ol’ alkaline battery.  They’re cheap, safe, recyclable and readily available.

For what it’s worth the Ikea alkaline batteries offer good value:  http://www.batteryshowdown.com/results-lo.html (I suggest buying as many packets as you can carry, so that you never ever have to go back there.  Unless you like arguing with your wife of course.)

Power Usage

The ATmega 328 has various power saving functions which involve putting it to sleep when not doing anything.  I use the Rocket Scream Low Power library to take care of putting the processor into a low power state, but I wasn’t seeing anything like the low power savings they detail on their site.

Some quick calculations:  Let’s assume a AA battery provides 2000 mAh.  I measured my Pro Mini V2 as drawing 6.7mA when powered up and doing things and 2.8mA when in sleep mode.  As a conservative estimate, let’s say it’s running for 1 hour in every 24 hour period and asleep the rest of the time.  That averages out to about 3mAh of draw.

For a 2000 mAh battery, that would give about 667 hours of runtime, or 28 days. So a standard Pro Mini V2 could run for about a month on a pair of AA batteries.  Not bad, but I think changing the batteries every month is still going to be a bit of a drag. Besides, Rocket Scream are seeing power usages in the micro-amps range when asleep.  There is clearly work to do.

How to dramatically reduce the power consumption

In this photo you can see I’ve identified some sections which are related to the power usage of the Pro Mini.

The red section is the power LED.  This is always on when power is applied and sucks about 0.2mA when lit. If you don’t need this to be lit all the time then you can easily remove it to save some juice.  I found the easiest way was to use a pair of cutters to snip/crush the middle of the LED and then use a soldering iron to remove the bits left over.

The green section is the on-board regulator.  If you are going to be supplying power to the board via a couple of AA batteries (each battery being 1.5v, so two is 3v) then you don’t need the regulator.  You can cut this off too if you like, but.. keep reading, there’s no need to hack it off.

Saving the best until last, the yellow section is the power-selection jumper to switch between 3.3v and 5v.  It passes the power supplied by the RAW pin through the regulator and on to the board.  The regulator is inherently inefficient.  You might think that you could bypass the on-board regulator by powering the board by apply power to the Vcc pin instead, but it still seems to power the regulator.  By simply unsoldering this jumper you can disable the on-board regulator and save loads of power.  Once removed you will need to apply power to the Vcc pin at ~ 3.3V.  I used some solder-wick to clean up but you could just scrape it off with a soldering iron if you need to.

Here’s one I prepared earlier.

With the jumper and LED removed. (Red and yellow boxes from previous image)

With the LED and solder jumper removed I measured the power usage again.  Running current is now down to 3.8mA, pretty much half of what it was.  But, most impressively the power used when asleep is down to 0.004mA.  4 microamps! Yay!

Some more quick calculations based on the same usage as before:  average power consumption drops to 0.17mA.  That gives us 490 days, 1.3 years run time off a pair of AA batteries.  That should allow for 2.5 minutes “work” an hour.  Waking up, taking some readings and sending them off via a radio should take well under 1 minute, which should allow for more power usage by a radio.

Conclusion

Unless you’re going to run your Arduino off a permanently attached serial connector, then just do this.  Get yourself a couple of AA batteries & a battery holder.  Apply the +ve side of the batteries to Vcc and the -ve to ground.  Stick your multi-meter in between the battery and Vcc pin to measure the lovely low current usage.  You can read the battery power being provided with the Secret Arduino Volt-meter trick.

I’ve got quite a few sensors around the house running with this set-up so I will monitor battery usage over the next few weeks or months and report back.

Coming Soon…

A write up of my cheapo IoT sensor network, including smoke detectors, door contact sensors, movement sensors, house-plant watering monitors, room temperature sensors and a weather station.  Plus, build an IoT sensor and buy a sausage roll for less than a fiver.

A few years ago, when I started working at home, I had a second ADSL line installed so that I could still get online if my ISP had an outage.  As well as fault tolerance I wanted to try and use all the available bandwidth rather than just have it sitting there “just in case”.  I achieved this using multi path routing and documented the solution here:  Over Engineering FTW.

This has been running really well on a Raspberry Pi for about 3 years (with an older kernel, see later in this post for why) but recently the SD card has started to fail.  Although this would be easy to fix; simply replace the SD card and copy my scripts over, the rural town I live in has just been upgraded to FTTC and so my connection speed has gone from about 8 Mbps to about 70 Mbps on each line.  The first generation Pi doesn’t have enough horsepower to cope with 70 Mbps let alone 140Mbps, and indeed the ethernet interface is only 100Mbps.  I had a Raspberry Pi 2 spare anyway so I figured I would use that and add a second gigabit NIC so I could cope with the theoretical 140 Mbps connection to the internet, and since I had two NICs I might as well use both of them.

Physical layout

This is what I came up with:

• Two lines coming from the cabinet to my house, one with Plusnet and one with TalkTalk
• The Plusnet line:
  • It came with an OpenReach vDSL bridge and a crappy locked down router, so I chucked the router away and used PPPoE tools to bring up the PPP connection
  • The vDSL bridge talks to the Raspberry Pi over a VLAN to keep it separated from the other noise on the switch
  • Interface eth1.1000 is an unnumbered interface and ppoeconf uses a layer 2 discovery protocol to find the bridge
  • Once the PPP connection is established ppp1 can be used to route traffic to the internet
• The TalkTalk line:
  • It too came with a crappy router, but no OpenReach bridge.  So I had to use it.
  • The TalkTalk router talks to the Raspberry Pi over VLAN 10.  Those ports are untagged on the switch, so as far as everyone on that network knows its just a self contained LAN.
  • Interface eth0 on the Raspberry Pi has an address on that LAN and uses the TalkTalk router to talk to the internet
• The main LAN:
  • Interface eth1 is used to connect to the main LAN
  • Clients on the LAN use the Raspberry Pi as their default gateway

With me so far?  Essentially we have the normal eth0 interface of the Pi connected to one LAN with its own router and eth1 (a USB gigabit ethernet adapter) has a tagged VLAN for connection to the OpenReach bridge (eth1.1000) and an untagged default network for connecting the the main LAN.  Once the layer 2 connection with the bridge is established a PPP connection becomes the second route to the internet.

The death of route caching

Around version 3.6 of the Linux kernel “route caching” was removed.  With route caching in place you could set up a default route with multiple hops, something along the lines of:

ip route add default nexthop via 192.168.1.254 dev eth0 nexthop via 192.168.2.254 dev eth1

When a packet needed routing to the internet the kernel would do a round-robin selection of which route to use and then remember that route for a period of time.  The upshot of this was, for example, that if you connected to www.bbc.co.uk and got routed first via 192.168.1.254 and so SNATed to 212.159.20.70 then all subsequent traffic for that destination also got routed via the same route and had the same source IP address.  Without route caching the next packet to that same destination would (probably) use the other route, and in the case of my home user scenario would arrive from a different source IP address – my two internet connections having different IP addresses.  Although HTTP is a connectionless protocol this change of IP address did seem to freak some services out.  For protocols with connections the story is worse, e.g. packets of an SSH connection would arrive at the far end from from two different IP addresses and probably get dropped.  Route caching was a simple fix for this issue and worked well, as far as I was concerned anyway.

Im sure the reasons to remove it are valid, but for my simple use case it worked very well and the alternative, and now only option is to use connection marking to simulate the route caching.  When I first looked at it I was baffled and thought I would just go back to a pre 3.6 kernel and use route caching again.  But, in the standard Raspbian distro there isn’t a kernel old enough for the Raspberry Pi 2 to make use of it.

So I was stuck…  I had to use a Raspberry Pi 2 to get enough packet throughput to max out my internet connections, and I couldn’t use route caching because there wasn’t a kernel old enough.  This meant I was going to have to either compile my own kernel or learn to use connection marking.  Joy.

Alternative projects

The documentation for Netfilter is extensive but I found a lot of it to be out of date and very hard to grok.  I found a few projects who had already implemented connection tracking/marking namely FWGuardian and Fault Tolerant Router.

FWGuardian is, as far as I can tell, designed for something orthogonal to my set up.  Where you might have lots of connections coming in to a server, or a number of offices which need to connect to other offices via pre-defined routes.  I played around with it for a while, and Humberto very kindly offered me support over email, but ultimately it was too involved and complex for my needs.  You should check out the project though if you have advanced requirements.  It’s got some brilliant features for a more enterprise oriented setup.

Fault Tolerant Router is a much simpler setup and matched my requirements very closely.  At it’s core it’s a Ruby script which can write your iptables rules and routing tables and constantly monitor the links.  If one goes down it can dynamically rewrite your rules and direct all traffic down the working connection.  However, it’s not expecting to use a PPP connection where gateways can change and it’s not really been tested with VLANs, although in practice it handled VLANs just fine.

But, at the end of the day, I wanted to learn how to do this myself and so I used the rules generated by Fault Tolerant Router to understand how connection marking was supposed to work and then started to implement my own home-grown solution for teh lolz.

Multi-path routing and connection marking

As I understand it, the idea with connection marking, or connection tracking – I’m not sure what the difference is, is that when a new conversation starts the packets are marked with an identifier.  You can then set ip rules to dictate which route packets with a particular mark take.  In essence once a new connection is established and a route selected, all other packets in that conversation take on the same mark and so the same route.  This emulates the route caching of the past.  I don’t really get how, in the case of an HTTP conversation (or flow) which is connectionless, all the packets in the conversation get marked the same.  This page has some more details, but I haven’t read it properly yet.  Anyway, we don’t know HOW it works, but it does.  Good enough.

IPtables

First of all we need to create the iptables configuration to set up connection marking.  Here’s the relevant extract from the iptables.save file:

*mangle
 :PREROUTING ACCEPT [0:0]
 :POSTROUTING ACCEPT [0:0]
 :OUTPUT ACCEPT [0:0]
 :INPUT ACCEPT [0:0]
 [0:0] -A PREROUTING -i eth1 -j CONNMARK --restore-mark
 [0:0] -A PREROUTING -i ppp1 -m conntrack --ctstate NEW -j CONNMARK --set-mark 1
 [0:0] -A PREROUTING -i eth0 -m conntrack --ctstate NEW -j CONNMARK --set-mark 2
 [0:0] -A POSTROUTING -o ppp1 -m conntrack --ctstate NEW -j CONNMARK --set-mark 1
 [0:0] -A POSTROUTING -o eth0 -m conntrack --ctstate NEW -j CONNMARK --set-mark 2

-i = –in-interface and -0 = –out-interface

These rules set a mark depending on which interface is used.  These changes happen in the mangle table.

Packets going in or out the WAN via ppp1 or eth0 which are a new connection are marked with a 1 or a 2 depending on which interface they use.  The decision about which route to use is done in the rules which we will see later.  Any packets coming in to eth1, so from the LAN, have their marks restored on the way in so they can be dealt with accordingly.

Now let’s have a look at the filter table:

*filter
 :INPUT DROP [0:0]
 :FORWARD DROP [0:0]
 :OUTPUT ACCEPT [0:0]
 :LAN_WAN - [0:0]
 :WAN_LAN - [0:0]

[0:0] -A INPUT -i lo -j ACCEPT
 [0:0] -A INPUT -i eth1 -j ACCEPT
 [0:0] -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

[0:0] -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
 [0:0] -A FORWARD -i eth1 -o ppp1 -j LAN_WAN
 [0:0] -A FORWARD -i eth1 -o eth0 -j LAN_WAN
 [0:0] -A FORWARD -i ppp1 -o eth1 -j WAN_LAN
 [0:0] -A FORWARD -i eth0 -o eth1 -j WAN_LAN

## Clamp MSS (ideal for PPPoE connections)
 [0:0] -I FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
 [0:0] -A LAN_WAN -j ACCEPT
 [0:0] -A WAN_LAN -j REJECT

The default policy is set to DROP, so any packet not matching one of the rules are dropped.

INPUT applies to packets which are bound for the router itself.  Packets from the local interface are allowed, and packets from eth1 (the main LAN) are also allowed.

FORWARD applies to packets which are passing through the router on their way somewhere else.  Packets which are known to be part of an already in-progress session are allowed.  Packets are then categorised as LAN to WAN or WAN to LAN and dealt with by the rules LAN_WAN or WAN_LAN, getting accepted and rejected respectively.  All this boils down to LAN clients using the Raspberry Pi as a router and so having their packets forwarded are allowed out and packets coming in from the internet are rejected, the exception being if they are part of an on-going connection.

Clamping MSS to MTU deals with a particular issue with using PPPoE connections where the MTU can’t be the usual 1500 bytes.  Because a lot of ISPs block the ICMP messages that would normally deal with asking the client to send smaller packet sizes we use this handy trick to make sure that packets can go out unfragmented.  If you find that some web pages are slow to load and others are not, then try switching this on.  If you’re only using upstream ISP provided routers you probably don’t need this.

Lastly in iptables we enable SNAT or masquerading so that connections out to the internet appear to come from a valid internet routable IP address not our LAN IP address:

#SNAT: LAN --> WAN
 [0:0] -A POSTROUTING -o ppp1 -j SNAT --to-source 212.159.20.70
 [0:0] -A POSTROUTING -o eth0 -j SNAT --to-source 192.168.1.253

Routing tables

We’ve configured iptables to add a mark to traffic depending on which WAN interface it is going in or out of.  But this is only marking the packets, there is no logic to make sure that packets of the same mark use the same route.  To make this happen we use ip rules.

First create three new routing tables by editing /etc/iproute2/rt_tables.  I’ve added this to the bottom:

1 plusnet
 2 talktalk
 3 loadbal

Now we add a default route to the first two of those tables:

ip route add default via $PPP_GATEWAY_ADDRESS dev ppp1 src 212.159.20.70 table plusnet
ip route add default via 192.168.1.254 dev eth0 src 192.168.1.253 table talktalk

$PPP_GATEWAY_ADDRESS is set when the PPP session is established and changes.  We can look at ways to find that address later, but for now just substitute the “P-t-P” IP address from “ifconfig ppp1” or whatever your ppp interface number is, or in the case of an ISP-provided router, the LAN side IP of that router.

This is simply creating a routing table with the name of the ISP that will be used and a default route which can find its way to the internet for that ISP.

Next we create the loadbal routing table which is a combination of the previous two:

ip route add default table loadbal nexthop via $PPP_GATEWAY_ADDRESS dev ppp1 nexthop via 192.168.1.254 dev eth0

which is the same idea as we used in the old route caching days, a round-robin route which flicks between the two available routes to the internet.

ip rules

We’ve now created the iptables entries to track and mark traffic from each of the two ISPs and add some basic firewalling and IP masquerading.  We’ve also created a routing table for each ISP and a load-balancing table which splits the traffic between the two ISPs.

Now we need to create some rules to govern which of the routing tables is used for a particular connection.  The commands to do this are:

ip rule add from $PPP_IPADDR table plusnet pref 40000
ip rule add from 192.168.1.253 table talktalk pref 40100
ip rule add fwmark 0x1 table plusnet pref 40200
ip rule add fwmark 0x2 table talktalk pref 40300
ip rule add from 0/0 table loadbal pref 40400

The rules are matched in numerical order based on preference and once a rule matches that’s it.  The first two rules make sure that traffic from the routers uses the correct table.

The important rules are the last three.  Traffic which has been marked “1” will always use the plusnet routing table, traffic marked as “2” will always use the talktalk routing table.  This ensures that all traffic which is part of an on-going conversation will always use the same router out to the internet, and so always come from the same IP address.

The last rule only matches traffic which is not already marked i.e. new conversations.  This routing table, as can be seen in the previous section, has a multi-path route to balance traffic between the two routes out.  Once a conversation is established the IPtables conntrack rules will mark the traffic and so one of the two fwmark rules will match.

Now delete the main default route so that the above rules don’t get bypassed with a route in the “main” table:

ip route del default

And that’s it.  You should now have a router which splits the traffic fairly evenly across two internet connections and keeps tabs on which packets should go out of which routers.  I’ve had this running for a month or so now, and it seems to be working fine.  I’ve had the Pi lock up a couple of times, but I think that’s related to the USB gigabit ethernet adapter.

Smart Netflix hacks

Services such as unblock-us allow you to work around some geographic content blocks by acting as your DNS server and replying with the IP address of, say, the US based Netflix server instead of the UK ones.  I’ve installed dnsmasq on my Pi as well and configured it to use the Unblock DNS servers instead of my ISP or Google servers.  The clients on the LAN get their network configuration over DHCP from the Pi which sets the DNS server address for the clients to the Pi itself which then handles DNS lookups using the Unblock servers upstream.  This works really well for most Netflix clients but I was having a lot of problems getting the Chromecast to work with Netflix and Unblock US.

It turns out that Google have hard-coded it’s own DNS servers into the Chromecast and so your local DNS settings are ignored.  Nice one Google.

Because we’re using a Linux box as our router we can do this:

iptables -t nat -A PREROUTING -s <Netflix Client IP>/32 -d 8.8.8.8 -p udp --dport 53 -j DNAT --to <Alternative DNS Server IP Address>
 iptables -t nat -A PREROUTING -s <Netflix Client IP>/32 -d 8.8.4.4 -p udp --dport 53 -j DNAT --to <Alternative DNS Server IP Address>

Using the NAT table we rewrite the DNS lookup bound for Google’s DNS servers to send it to our dnsmasq server instead. lol.

Spreading interrupts across cores

Network cards have queues for tx and rx.  Higher end cards will typically have more queues, but on the Pi the on-board NIC (which is actually connected via USB) has one for tx and one for rx, as do the VLAN interfaces and the PPP interfaces.  Each of these queues has a CPU affinity and it seems that by default the queues all use the same CPU core.

When downloading an ISO with BitTorrent and the load-balancing set up I was able to achieve just over 10 MBytes a second.  But the Pi became really unresponsive.  Looking at top showed one CPU core maxed out in soft interrupts:

By adjusting the CPU affinity to spread these IRQs across multiple CPUs I squeeze out a tiny bit more network throughput, but more usefully the Pi remained responsive under heavy load:

The commands I used to do this are:

echo 1 > /sys/class/net/eth0/queues/rx-0/rps_cpus
echo 1 > /sys/class/net/eth0/queues/tx-0/xps_cpus
echo 2 > /sys/class/net/eth1/queues/tx-0/xps_cpus
echo 2 > /sys/class/net/eth1/queues/rx-0/rps_cpus
echo 4 > /sys/class/net/eth1.1000/queues/tx-0/xps_cpus
echo 4 > /sys/class/net/eth1.1000/queues/rx-0/rps_cpus
echo 8 > /sys/class/net/ppp1/queues/tx-0/xps_cpus
echo 8 > /sys/class/net/ppp1/queues/rx-0/rps_cpus

Source

Here’s a tgz file containing my iptables rules and a script to set up the above: routing

Update:  I’ve put the files in this Github repo:  https://github.com/8none1/multipathrouting

If you’re interested in helping to make the scripts a bit more generic and adding fault-tolerance let me know.

Start by reading this: https://developer.ubuntu.com/en/snappy/

In summary; a Snappy application is secure because it’s wrapped with AppArmor. It’s easier to install and upgrade because everything is packaged in a single file and installed to a single location. That location is backed-up before you install a new version, and so if the installation goes wrong you can revert to the previous version easily by copying the original files back (or rather, Snappy will do all of that for you). Simplifying things slightly there are two types of Snappy “application”: Apps and Frameworks. Frameworks can extend the OS and provide a mediation layer to access shared resources. Apps are your more traditional top-level items which can use the provided frameworks, or bundle everything they need in to their Snap. This makes things much easier for app providers because they are now in charge – they can be assured that no library will change underneath them.  This is a huge benefit!

Let’s get Mosquitto snapped.

1. Install QEMU to run an Ubuntu Core machine

http://www.ubuntu.com/cloud/tools/snappy#snappy-local

First we install the KVM hypervisor:

sudo apt-get install qemu-kvm

Then check everything is as it should be with:

kvm-ok

Now download the latest Ubuntu Core image from here: http://cdimage.ubuntu.com/ubuntu-core/preview/  At the time of writing this is the newest x86-64 image: http://cdimage.ubuntu.com/ubuntu-core/preview/ubuntu-core-alpha-02_amd64-virt.img

Then launch the virtual machine. This command port forwards 8022 on your local machine to 22 on the virtual machine, so you can SSH to port 8022 on localhost and actually connect to the Ubuntu Core machine. It gives the Core machine 512MB of RAM, nicely achievable on a modest budget (The Pi2 has 1 GB).  We also forward port 1883 from to the VM, which will allow us to connect to the Mosquitto server on our VM once it’s all installed.

kvm -m 512 -redir :8022::22 -redir :1883::1883 <vm image file>

Once it’s booted you can connect to it with SSH. The username and password are “ubuntu”.

ssh -p 8022 ubuntu@localhost

To make things a bit easier, why not use key authentication? On your host machine:

ssh-copy-id -p 8022 ubuntu@localhost

We should also upgrade our Ubuntu Core VM before we start.  SSH in to your box and run:

sudo snappy update
sudo reboot

2. Build Mosquitto in the right way

Back on your host (not the virtual machine you just created above) create some directories to hold the code and download the latest stable source and the build dependencies for Mosquitto:

sudo apt-get install build-essential cmake
sudo apt-get build-dep mosquitto

mkdir -p mosquitto/install mosquitto/build

cd mosquitto

wget http://mosquitto.org/files/source/mosquitto-1.3.5.tar.gz

tar xvzf mosquitto-1.3.5.tar.gz

cd build

Time to build Mosquitto.  Before you run the commands below, a bit of background information.  The cmake line will force cmake to install the binaries to the location specified with INSTALL_PREFIX, rather than /usr/local.  This is required to bundle all of the binaries and other files to the “install” directory we created above, making it possible to package as a Snappy.

cmake -DCMAKE_INSTALL_PREFIX=`readlink -f ../install/` ../mosquitto-1.3.5

make -j`nproc`

make install

nproc spits out the number of processor cores you have, so the make line above will use as many processor cores as you have available.  It’s not required, and for Mosquitto which is fairly small it’s not worth worrying about, but for a bigger job this is quite handy.

If you look in the “../install” directory you’ll see a familiar structure containing all the goodies needed by Mosquitto.

3. Find the libraries needed and copy them in to your Snappy project

Change in to the install/lib directory and use ldd to display the linked libraries for the two main .so files:

ldd lib/libmosquitto.so.1.3.5 lib/libmosquittopp.so.1.3.5 | grep '=>' | awk '{ print $1 }' | sort | uniq

This uses ldd to show the libraries required by Mosquitto, and then sorts them in to a nice list. You’ll see something like this:

libcares.so.2
libcrypto.so.1.0.0
libc.so.6
libdl.so.2
libgcc_s.so.1
libmosquitto.so.1
libm.so.6
libpthread.so.0
librt.so.1
libssl.so.1.0.0
libstdc++.so.6
linux-vdso.so.1

Now, on the Ubuntu Core machine we can run this little script:

for i in `cat`; do find /lib /usr/lib -name $i; done

Copy the list from the previous command to the clipboard and then paste it in to terminal where this command is running and hit Ctrl-D to submit the list.  The script will then search Ubuntu Core for the libraries required.  If it finds them they will be displayed, if it doesn’t then they are not available in Ubuntu Core by default and will need to be included in your Snappy package.

linux-vdso is the Linux kernel and is available on every Linux system by default, so we don’t need to provide that specifically.

libssl, libcrypto, libpthread, librt, libc and libdl are all available in Ubuntu Core by default – so we don’t need those either.

That leaves just libcares to be copied in to our package.

 cp /usr/lib/x86_64-linux-gnu/libcares.so.2.1.0 .

We should already be in the ‘lib’ directory, hence the ‘.’ above.  We are copying libcares in to the lib directory of our Snap, and when we run the Snap we will pass in the library path to make sure Mosquitto can find it.  More on this later.

4. Add the meta data required for the Snappy package

Reference: https://developer.ubuntu.com/en/snappy/guides/packaging-format-apps/

Create the meta data directory inside the install directory (change to the install directory, it should just be cd ..):

mkdir meta

Create the package.yaml file:

nano meta/package.yaml

And this is what we’re putting in it:

name: mosquitto.willcooke
architecture: amd64
version: 1.3.5
icon:
services:
 - name: mosquitto
 start: ./sbin/mosquitto.sh
ports:
 required: 1883

Information about these fields and what they mean is available in the reference linked to above, but they are easily understandable.  A comment on the name though, you need to append .<yournamespace> where your namespace is as you select in your Ubuntu myapps account.  One thing to mention, you can see that to start our Snap we are calling a shell script.  This allows us to pass in extra options to Mosquitto when it runs.

Next we need to create a readme file:

nano meta/readme.md

This file needs to contain at least a couple of non-blank lines.  Here’s what we put in it:

This is a Snappy package for Mosquitto MQTT broker.

Information about Mosquitto is available here:  http://mosquitto.org/

Information about MQTT is available here: http://mqtt.org/

We also need to configure our Mosquitto server, by editing the conf file.  Most of the settings can be left as default, so we will create a new conf file with only the bits in we need.

mv etc/mosquitto/mosquitto.conf etc/mosquitto/mosquitto.conf.ori

nano etc/mosquitto/mosquitto.conf

Add these two lines:

user root
persistence_location /var/apps/mosquitto/current/

We need to change this to run as root.  Since our Snap will be confined there is no risk here.  I expect the ability to run as non-root users when using Snappy will be improved, but really it’s not necessary.

We also need to add a small shell script to start Mosquitto with the right options.  Create a file in install/sbin called mosquitto.sh:

nano sbin/mosquitto.sh

And add this:

#!/bin/sh
LD_LIBRARY_PATH=./lib:$LD_LIBRARY_PATH exec ./sbin/mosquitto -c etc/mosquitto/mosquitto.conf

We are specifying where to find the extra libraries we require and where to find the conf file.  Make that file executable:

chmod +x sbin/mosquitto.sh

5. Build your Snappy package

Add the Snappy PPA to get the build tools, and then install them:

sudo add-apt-repository ppa:snappy-dev/beta
sudo apt-get update
sudo apt-get dist-upgrade
sudo apt install snappy-tools

In your install directory run:

snappy build .

If you see an error about ImportError: No module named ‘click.repository’ then you likely have a clash between the Click library version in the SDK team PPA and the version in the Snappy PPA.  This will be fixed soon, but in the meantime I would suggest installing ppa-purge via apt-get and then running sudo ppa-purge ppa:ubuntu-sdk-team/ppa.

If you see an error about “expected <block end>” in the package.yaml check the whitespace in the file.  It’s likely a copy and paste error.

6. Install your Snappy package

Once you have your .snap file you can install it to your virtual machine like this:

snappy-remote --url=ssh://localhost:8022 install ./mosquitto_1.3.5_amd64.snap

7. Test your Snappy package

If everything has gone to plan Mosquitto should now be running on your virtual machine.  In order to test you’ll need to write a test Publisher and Subscriber.  I used the Python Paho library.

Here’s an example Publisher:

#!/usr/bin/python

import paho.mqtt.client as mqtt
from datetime import datetime
from time import sleep

def send_mqtt(topic, message):
 log("Sending MQTT")
 log("Topic: "+topic)
 log("Message: "+message)
 mqttc.reconnect()
 mqttc.publish(topic, message)
 mqttc.loop() 
 mqttc.disconnect()

print "Time server starting up...."
mqttc = mqtt.Client("python_pub")
mqttc.connect("localhost", 1883)

while True:
 tstr = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
 send_mqtt("/information/time",tstr)
 sleep(10)

And here’s an example Subscriber:

#!/usr/bin/python

import paho.mqtt.client as mqtt
import datetime

def on_connect(client, userdata, rc):
 print "Connected with result code "+str(rc)
 client.subscribe("#")
 
def on_message(client, userdata, msg):
 print "Topic: ", msg.topic+'\nMessage: '+str(msg.payload)
 
 
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message

client.connect("localhost", 1883, 60)

client.loop_forever()

What’s next?

We’ve built a Snappy package for amd64 (or whatever your native architecture is), but we really need to be cross-architecture to give people the best choice of platform on which to use the package.  This involves cross compiling, which can be tricky to put it mildly.

I spoke to Alexander Sack, the Director of Ubuntu Core, and asked what was coming next for Snappy and I was very excited to hear about easier cross-compilation methods as well as a cool script to help automate gathering the libraries in to your package.  I’ll find out more about these and follow up with another post about

Special Thanks

A huge “Thank You!” to Saviq and Didrocks for doing the actual work and letting me watch.

Where to get Snappy Mosquitto

amd64 version:  https://myapps.developer.ubuntu.com/dev/click-apps/ubuntu/1500/
armhf version: https://myapps.developer.ubuntu.com/dev/click-apps/ubuntu/1502/ (please note, I haven’t been able to test the ARM version because of a lack of hardware.  If it doesn’t work let me know and I can fix it.)

If you like playing with new toys you might have already downloaded and tried the Unity 8 Desktop Preview (available here:  http://cdimage.ubuntu.com/ubuntu-desktop-next/daily-live/current/)

If you haven’t, you should take it for a spin.  If you have an Intel graphics everything should be fine and dandy, if not YMMV at the moment.

1. Download the ISO
2. Find a spare >1GB USB thumb drive
3. Run “disks” via the dash
4. Highlight your USB drive and from the cog icon on the right choose “Restore Disk Image…”
5. Select your ISO and “Start Restoring”  – this will of course erase everything else on your USB stick
6. Done

You can now boot from your USB stick and have a play with Unity 8.  Right now you’ll be seeing the Phone view of Unity 8, but that will all be changing in time.

Capturing a screencast

Once you’ve got everything up and running you might like to make a few screencasts, so how do you do it?  Well, the Mir developers have provided us with the mirscreencast tool so let’s use that:

Switch to tty1 (ctrl+alt+f1) and log in and run:

mirscreencast --file <output file> -m /run/lightdm-mir-0

Then switch back to tty8 (ctrl-alt-f8) and use Unity.

Your file will now be filling up FAST.  Mirscreencast will be trying to write every raw frame to that file, probably at a rate of 60 frames a second.  To kill mirscreencast I first hit ctrl-z and then:

 pkill -9 mirscreencast

but there is probably a better way.

Capturing a better screencast

There are a few command line options for mirscreencast which will can help us shrink the file size a bit:

mirscreencast --file <output file> -m /run/lightdm-mir-0 -s 683 384 -n 3600

The option “-s” will resize the captured frames.  Note that 683 384 is exactly half my native resolution, so you will need to adjust this to your display.

The option “-n” will capture n frames and then stop.  At 60 frames a second, 3600 frames is one minute.  If you use -n then mirscreencast will exit gracefully at the end.

Playing back your screencast

I am lucky enough to have a spare machine with a touch screen just for running Unity 8 on (http://www.dell.com/uk/dfh/p/inspiron-11-3137/pd) so I SCP the raw video file on to my main machine for playback and editing.

I use mplayer for most of my video playback and encoding needs and it will happily play the raw video file, but it needs a few pointers:

mplayer -demuxer rawvideo -rawvideo fps=60:w=683:h=384:format=bgra <filename>

or, to convert the raw file into something which you can edit in OpenShot try this:

mencoder -demuxer rawvideo -rawvideo fps=60:w=683:h=384:format=bgra -ovc x264 -o <output filename> <filename>

And then you can edit and upload the processed file.  When I export from OpenShot I use the “Web” profile, then target “YouTube-HD”, “HD 720p 29.97 fps” “Med” – it’s a bit overly compressed, but it looks OK.

It’s really happening!  The breadboard prototype has been running the heating and hot water for a week or so now, and so far nothing has caught on fire.  The relays are happy, they’re not getting warm or anything, the Pi is turning things on and off when it’s supposed to, the REST API is working, and I’ve knocked up a quick Web interface to switch things remotely.  Last night for the first time I switched the heating on from the sofa, because I was a little bit cold.

Great success!

I’ve turned the breadboard layout in to a PCB design and today at 09:11 UTC it was sent off for manufacture, I should have it back in a week.  All the other components are on their way and so next weekend I should be able to put everything together and solder it to the board.  With a bit of luck it will work first time.

I’m quite please with the board layout, it’s a fairly useful breakout board for the Raspberry Pi.  In the future I would like to re-work it with smaller traces so that it can fit directly on top of the Pi.

The software to drive everything is rather basic, but I will make it available via Github anyway, it might come in handy.  I will be working on this over the coming months, so it should improve soon.  You’ll need a MySQL server set up.  The schema for the database is included in the repo.  The whole thing has grown organically and so the naming and structure is poor, but it works.

Code:  https://github.com/8none1/heating

I’ve starting making some 1wire temperature sensors which I will place in various rooms and hook up back to the Pi via cat5.

I won’t be able to control individual radiators at the moment, but I can set a target temperature for a given room and rely on the TRVs to control the temperature in other rooms.

I’ll also be adding outside temperature sensors and looking to replace the main room stat with another Pi in the future.

In summary then, I should have the final thing built in the next couple of weeks.  More to follow when that happens.  I’m considering looking at moving the whole thing to a micro controller rather than a Pi, that would reduce the BOM quite considerably, and might even warrant a commercial product in the long run – a hackable heating controller.  Is this something people might be interested in?

Further Updates

• Part 1
• Part 2
• Part 3

The final circuit design

In the meantime I’ve been thinking about the software to drive the thing.

My programming skills are pretty basic so I have been trying to keep everything as simple as possible.   It became apparent that I was going to need some inter-process communication so that I could handle switching things on and off from outside the controller program (for web control etc).  This presented me with a problem as all the basic reading from files/FIFOs were blocking – in that they would sit and wait for a “command” to be received before continuing – and that was too limiting for my needs.  I was going to have to deal with some kind of threading.  This troubled me deeply.  The good people of Google+ suggested a few options and in the end I decided a RESTful interface was the way to go as it should be accessible from the widest range of other languages I might have to use, and especially easy from a web browser.

Originally I had imagined that I would have a single Python script to handle pretty much everything, including:

• Switching the relays on/off
• Responding to button presses
• Proving a scheduling engine for standard “on at this time, off at this time” behaviours
• Providing a way for other things to control the system
• Monitoring temperatures around the house
• Intelligent switching (e.g. it’s extra cold this morning, turn on early)

That way, I figured, I could just write one script and each function could interact with each other function very easily.  But following a conversation with Mark S. a few years ago, and then more recently with Stuart L. I was starting to think that the monolithic architecture was not the way to go and instead I should push the intelligence out to the edges and the main loop should just deal with the basic on/off functionality.  I was worried that if something went wrong with the electronics, for example the heating turned on but never off, then it would be difficult to spot from outside the main loop.  But, given that the heating system has a lot of built-in safety features like room stats, boiler stats and hot water stats which are all wired in series with the mains that won’t be a problem, and besides the current controller doesn’t do anything more intelligent than ON or OFF.

To that end I decided that I’d go with this architecture:

The Relay Controller would handle switching the power on and off in the right order to the right outputs, it would handle the switches for manual override, provide feedback via LEDs, and it would provide a REST interface for anything else to control the system or find out what the current state is.  This way I can write a much more simple script to manage the scheduler for example.  It can run independently and then simply poke the relay controller at the right time.  I also decided that the relay controller would only have the ability to switch off after a certain period of time.  The default will be 60 mins.  This further simplifies things and puts just the right amount of intelligence in the core.  The scheduler just says “switch on now, and off again in 90 minutes”.  If the scheduler crashes the correct state is maintained by the controller and things just pick up from where they were when the scheduler starts again.  If the controller crashes then the power is removed from the system and it “fails safe”.

The REST API is simple:

• /get/ch or hw/ – gets the current state of the system as a JSON object. e.g. “http://10.0.0.1/get/ch”
• /set/ch or hw/on or off – switch the system on or off.  e.g. “http://10.0.0.1/set/ch/on” switches the central eating on for the default time (60 minutes)
• /set/ch or hw/on/n – switch the system on for n minutes.  e.g. “http://10.0.0.1/set/hw/on/90” switches hot water on for 90 minutes

I used the Python SimpleHTTPServer, the ThreadedHTTPServer and threading.Thread.

I’m certain that it could be rewritten using proper OO methods and whatnot, and there is quite a lot of code which started out as an idea which then became redundant, but if you’re interested it’s available here:  piheat

Next on the agenda is to draw a PCB and get it made (once I’ve tested the circuit properly) and work on the scheduling engine.  At this rate I should be finished right around the time when we don’t need to use the heating any more.

Further Updates

• Part 1
• Part 2
• Part 4

In part one of this series I explained how the central heating system was wired up and what electrical connections you needed to make to switch your heating and hot water on and off.

I ordered all the parts and now I’ve plugged them all together.  Here’s a quick video demonstrating how it will all work.

Further Updates

• Part 1
• Part 3
• Part 4

TL;DR:  It should be fairly straight forward to add a Raspberry Pi controlled heating and hot water system to a standard UK domestic set up and, more importantly, remove it again without messing with the existing set up.  As a minimum you’ll need a Raspberry Pi and 4 relays.  A few other bits and bobs wouldn’t go a miss though.  The theory checks out, I’ve ordered the bits, come back next time to see what it looks like.

It occurs to me that for a long time we’ve had a thermostat in our homes which switches the heating off when it gets warm enough, but wouldn’t it be just as useful to have something which turns the heating on when it gets too cold?

This thought, together with a Raspberry Pi that wasn’t doing much and a strong desire to make my home more connected, led me to think about how I might control my heating system from, say, a smart phone.  I’m far from being the first person to think of this idea, and there are loads of really good examples out there, but none of them did quite what I wanted in the way I wanted to do it.  So I’m going to start from first principals and walk through this project design to try and build a removable & non-destructive add-on to an existing system.  I’m writing this at the very start of the project, so I’ve no idea if it will work, if I will break some expensive components on way to getting it working, or if I will just give up before I get to the end.  Let’s see.

Typical domestic hot water and heating systems

This may be UK specific.  Here is a very very crude diagram of a typical home set up:

A crude diagram of how the central heating system works in a typical UK home.

The boiler burns gas and heats water.  That hot water is circulated around the system (called the primary circuit) by a pump and can do three jobs.  It can circulate through a heating element in a hot water cylinder and heat more water which is stored in the cylinder.  Note that the water which circulates through the element does not come in to contact with the actual water it is heating, the two are kept separate for water quality reasons.  The second job it can do is circulate through radiators in the home and hear the air.  The third job is to do both.  The hot water from the boiler moves around the primary circuit losing it’s heat to either the hot water in the cylinder or the air and eventually passes through the boiler again, heats up, and goes round and round again.

There are two “header” tanks of cold water in the loft.  One is for the cold water to the bathroom for flushing the loo, filling the bath, brushing your teeth, that kind of thing.  This tank also fills the hot water cylinder.  The other is the header tank for the primary system and ensures that it can’t boil dry.  Both use gravity and water pressure to make sure the water flows to where it is needed.

The system in the diagram is an “open” system.  If the hot water in the cylinder gets too hot it can expand up the vent pipe and dump itself in to the cold water tank.  The cold water tank can over flow to outside.  If the hot water in the primary gets too hot it can expand up in to the header tank ready to be reused to fill the primary when the water cools.

There is such a thing as a sealed pressurised system which doesn’t have these vents.  These are more complex and if you have one please be very careful in tinkering with the control mechanisms.  In an open system, if you get things wrong and the boiler runs and runs you would end up with a lot of steam in the loft.   In a pressurised system things can go pop and blast you with boiling water.  That said, in an open system you could still end up dumping a header tank full of boiling water down on to the bedrooms below.  People have died from this happening, so tinkering with the heating system is not something to be taken lightly.

In summary then; we have three things we can ask a system for:

1. Make hot water
2. Heat the house
3. Make hot water and heat the house

And we have a number of key elements:

1. Boiler
2. Hot water cylinder
3. Primary header tank
4. Cold water tank

Typical electrical system to control hot water and heating

Once we understand how the wet bits fit together we can take a look at the electrical components:

Y Plan electrical wiring plan for central heating and hot water.

There are multiple “standards” for wiring up a heating system.  You can find heaps of information on the excellent DIY FAQ wiki.

My system has been wired in the “Y Plan” configuration and if you have a single 3-port valve in your airing cupboard and a couple of tanks in your loft – then there is a good chance you have too.  I will run through the wiring, and some of the inherent safety systems built in (which is why I’m keen to make sure my controller is a simple replacement for the existing controller, and is not a complete re-wire).  Before we start though, a further word of caution.  Mains electricity is lethal.  You need be comfortable playing with this stuff to consider attempting anything to do with the heating system.  It’s also probably illegal in UK due to some draconian restrictions on what a home owner can and can not do to the wiring in their own home.  Don’t try this at home kids.  A competent tradesman might be able to help you hook it all together.

The incoming mains supply goes through a double pole switch which will disconnect live and neutral when switched off.  In this diagram, the live feed provides power to only the controller (sometimes you might see a parallel (switched and fused) connection to the boiler from that live).  So first and foremost, all power to the components comes through the controller. Neutral is common to pump, boiler and valve and so is earth.

Thermostats are placed in series for both the hot water circuits and the heating circuits.  These will physically break the circuit when a specific temperature is reached.

Let’s consider this example:  I tell the controller to heat the water.  It connects the live feed to the “HW ON” cable via point 6 on the diagram. The current flows to the cylinder stat, which allows the current through since the temperature is lower than the trigger point it is set to.  The pump and the boiler are connected in parallel so you can’t run the boiler with out the pump running too (at least that’s the plan), and they are provided power via the room stat to point 8 on the diagram.  The boiler is told to turn on, and the pump moves that heated water around the system.  The hot water reaches the three port valve.  The valve has an electrical actuator on which moves to set position depending on what electrical connections are made to it.  In our case, no INPUT power is being applied to the valve, so it sits in it’s default position – which just happens to be “Hot water mode”, and so the heated water from the boiler passes through the hot water cylinder only.  When the hot water cylinder get’s to the right temperature the thermostats clicks over to the other contactor and now no power is applied to the pump and boiler via the HW ON output on the controller.  Instead, the grey wire, point 7 on the connector, is energised.  This tells the valve that HW is no longer required.  This system is pretty safe, since as soon as the cylinder stat is triggered power is removed from the boiler and so it would shut down.  Now, thermostats do fail, but they usually “fail safe”, but sometimes they don’t.

What if I want just the heating to run?  The controller connects to the live input to the CH cable via point 4 on the connector.  This passes through the room stat which will allow the current to flow if it’s below the temperature set.  The current ends up at the valve via point 5 on the connector.  In this case, where we only want heating, the “white” wire is live (it’s black on the diagram) and the valve connects the “white” wire to the “orange” wire which goes back to point 8 on the connector, and in turn provides power to the boiler and pump.  At this point the “grey” wire is also energised, as the controller makes it’s “HW OFF” output live when you ask for only heating. The room stat is able to cut power to the circuit when it reaches the set temperature.

If we want both hot water and heating, the controller energises the “CH ON” and “HW ON” outputs.  Here current is provided to the pump and boiler when any of the thermostats indicates that more heating is required.  If the HW reaches it’s temperature first, then the stat energises the grey wire, which tells the valve that no more hot water is required, and so it will move to the CH ONLY position, and current will continue to be provided by the orange wire when the valve reaches the correct position.  The heated water from the boiler will stop circulating through the hot water cylinder and go only through the radiators – concentrating the heating to where it is needed.  Pretty neat!

This system strikes me as being both simple and brilliant at the same time.  It’s also pretty safe, as long as the stats are working as they should do.

In summary then, the controller is able to indicate a requirement for hot water, central heating, or both by linking three outputs to live in the right sequence.  The four states are therefore:

1. HW OFF, CH OFF (0,0)
2. HW OFF, CH ON (0,1)
3. HW ON, CH OFF (1,0)
4. HW ON, CH ON (1,1)

Confirming my deductions

I’ve looked at the plumbing, and I’ve looked at the wiring, and I’m pretty sure that I know what’s going on.  Next thing to do is apply the scientific method and gather the evidence to back up my assumptions.

The first thing I did was to turn off the power to the heating system.  I’m paranoid, so I turned it off at the fused connection to the left of the controller and also at the fuse box.  I also wore rubber boots, and jumped in the air every time I touched a wire.  Better safe than sorry, eh?  And, rightly so it turns out.  The fused connection unit did actually cut all the power to the heating system, but look carefully at the third connection from the left and you’ll see an earth wire being used to carry live current.  This is against all the regulations.  Whoever installed this system originally was clearly a free spirit.  I was also quite impressed that they’d managed to squeeze all the connections in to a double gang back box.  What a mess.  Remember kids, only a competent person is allowed to fiddle with these things – they do a better quality job you see.

Looking at the zoomed in image you can see 6 terminals:  N, L 1, 2, 3, 4.

N & L are self explanatory.  2 is not connected to anything, and so I don’t need to worry about it.  So that leaves three connections that do something (1, 3 and 4).  Exactly what I was expecting.  One will be CH ON, one HW ON, and one HW off.  Which is which?

Looking at the back of the controller unit it’s self:

My theory is sound!  1 is HW OFF, 3 is HW ON, 4 is CH ON.

It looks like everything is connected as I had expected, but better safe than sorry.  Let’s do a bit more testing:

I wired in a few bits of cable and then (not shown) removed the connections to the rest of the system (labelling where they came from when I removed them!).  I left the L & N connected.  To recap, I removed the existing wires from 1, 3 and 4 and replaced them with my cables which came down to some screw down connector blocks.  The reason I put connector blocks on the end was two fold.  Firstly, to make it easier to probe with my multimeter and secondly to stop me accidentally brushing against one of the cables and giving myself a shock.  I also labelled the permanent live with a bit of red heat-shrink, just so I don’t get confused.

Putting the controlled back on, I hooked up my multimeter and switched through the options to see what happens when.  My findings are below:

.

.

.

.

.

Exactly what I expected.  Point 1 must, therefore be “HW OFF”, point 2 “HW ON” and point 3 “CH ON” – which they are, as we saw from the back of the controller.  I’m now confident enough with the set up to proceed with roughing out a block diagram for the controller and ordering the parts.

The plan

This rather unclear breadboard layout (with the awesome http://fritzing.org/home/) logically lays out what I intend to do.  I’ve also added a crude block diagram for good measure.

First, I will add a real-time clock module.  They’re cheap and easy to fit.  This will provide the Pi with a source of time when it can’t talk to NTP servers, and so it will be able to turn things on and off at the right times, even when the network connection is down.

Next I will add four relays.  I will take the main 240V incoming supply out of the existing controller and put it through relay 1.  This relay will pass the supply on to the existing controller via the “Normally Closed” relay output.  When I switch this relay, the supply to the existing controller will be dropped, and instead routed to the other three relays which will then be able to switch this current.  These three relays will be wired in parallel with the existing controller connections, much like in the image above showing the test harness connected in to the controller connections.  That is to say: one relay will go to point 1, one to point 3 and one to point 4.  The existing safety features (thermostats in series in the circuits) are un-changed and so still offer the same protection.  In order to activate heating or hot water we switch the relays as per the table above.  By adding my new system in parallel and being able to easily switch between the two I can bring the RasPi powered one online gradually.  A few hours here, a few hours there.  And once I’m happy that it’s not going to go crazy I can leave it unsupervised for longer and longer periods.  It also means that if I update the software and break something, we can still wash.

I will also add a number of 1wire temperature sensors.  I will have three on the hot water cylinder: 1 at each of the top, middle and bottom.  This will give me insight in to how much hot water is in the cylinder, and the temperature thereof.  This is not intended to be a safety system.  The temperature readings from these sensors will not be relied upon to switch things off in an emergency, that will be left to the original thermostats, but – we could use these readings as well to help make decisions.  I will also fit a temperature sensors in the cold water tank, the primary header tank and somewhere outside.  This will give me insight in to a couple of things:  Firstly, how cold is the water in the CW tank, and what is the temperature outside?  This has a direct effect on the number of showers that can be had from a given amount of hot water at a known temperature. Useful for trending too.  Secondly, fitting a sensor in the primary header tank can report when the header tank is getting hot.  Really, the header shouldn’t heat up too much.  If it does then either the system is “pumping over” – where the pump is forcing water up the vent pipe pipe OR the water is so hot it has expanded enough to push water out of the vent, or I expect some combination of the two.  Either situation is sub-optimal, and with a sensor in the header tank I get some visibility of what’s going on.  I might also add a sensor to the boiler input and output, so get an idea of how much work the boiler is doing.  Adding a sensor to each room would be a nice addition at some point too.

A couple of switches will be added for manually switching the hot water or heating on/off from the airing cupboard, where the current controller is situated and where the more senior visitors to Whizzy Towers will expect the heating buttons to be.

Shopping list:

• 1 x Real Time Clock.  http://www.ebay.co.uk/itm/200929798800?ssPageName=STRK:MEWNX:IT&_trksid=p3984.m1439.l2649
• 4 mains rated relays.  http://www.ebay.co.uk/itm/190950013824?var=490205033810&ssPageName=STRK:MEWNX:IT&_trksid=p3984.m1439.l2649
  
• Some 1wire temperature sensors.  http://www.ebay.co.uk/itm/DS18S20-Temperature-Sensor-1-Wire-Dallas-Maxim-/130621920626
• A few meters of 0.75mm 5 core heat resistant flex (type 3095Y).  http://www.ebay.co.uk/itm/5-Core-Heat-Resistant-Flex-Electrical-Cable-3095Y-0-75mm-/310663083027?pt=UK_BOI_Electrical_Components_Supplies_ET&var=&hash=item4854f67413
• A short bit of twin brown and earth for the connections between the 1st relay and the original controller live input.
• A 4k7 resistor

Couple that with a few odd bits of wire, some LEDs a bit of Python and we should have ourselves a Raspberry Pi powered heating and hot water controller which is relatively safe, easy to remove and cheap to build.  Let’s see what happens when all the bits turn up.  Should be here in a week or so.

Stay tuned.

Further Updates

1. Part 2
2. Part 3
3. Part 4

The premise is simple; you’ve got a requirement to put a map on your site.  You don’t have the first clue how to do it.  You follow these instructions and you get you map.  Hopefully you’ll pick up enough to tweak the map to your requirements, if not – please ask in the comments and I’ll try and help you.  I’m assuming that you’ve got a bit of HTML experience, and that you’ve got a JavaScript debugger available to you (either Firebug for Firefox or Developer Tools built in to Chrome).

I’ve based a lot of this on the official Google tutorial here:  https://developers.google.com/maps/documentation/javascript/tutorial.  (I’m pretty sure they borrowed that from me in the first place, so it’s only fair ).

Part 1.  Getting the map on the page

Get yourself an API key for Google Maps from here:  https://developers.google.com/maps/documentation/javascript/tutorial#api_key

Let’s start from the end and work back to the beginning.  Save this as an HTML document, change the API key, open it in your browser and you’ll have a map.  I’ve saved you a bit of time by already centring it on Barrow-in-Furness bus depot.

[js]
<!DOCTYPE html>
<html>
<head>
<title>My first Google Map</title>
<meta name="viewport" content="initial-scale=1.0, user-scalable=no" />
<style type="text/css">
html { height: 100% }
body { height: 100%; margin: 0; padding: 0 }
#map-canvas { height: 100% }
</style>
<script type="text/javascript"
src="https://maps.googleapis.com/maps/api/js?key=XXXXXXXXXXXXXXXX&sensor=false">
</script>
<script type="text/javascript">
function initialize() {
var myLatLng = new google.maps.LatLng(54.124634, -3.237029)
var mapOptions = {
center: myLatLng,
zoom: 17,
mapTypeId: google.maps.MapTypeId.SATELLITE
};

var map = new google.maps.Map(document.getElementById("map-canvas"),
mapOptions);

var myMarker = new google.maps.Marker({
position: myLatLng,
map: map,
title: "A place on Earth",
draggable: true,
})
}

google.maps.event.addDomListener(window, ‘load’, initialize);
</script>
</head>
<body>
<div id="map-canvas"/>
</body>
</html>
[/js]

Copy and paste that in to a text editor, replace XXXXXXXXXXXXXXXX with your own API key, save it somewhere and the load it up in your browser.  You should see a satellite style map with a marker in the middle. You’re done.  Simple eh?  Read on learn a bit about how it works, and what you can do to change the appearance.

Part 1.1 Understanding the basic HTML

In order to try and guarantee a consistent layout across browsers you need to make sure that your page is rendered in ‘Standards Compliant’ mode as opposed to ‘Quirks’ mode. To do this you need to specify a DOCTYPE at the top of your HTML file.  We’re using a very simple “html” DOCTYPE which tells the browser that we’re HTML5.  In HTML4.x there were a plethora of variations – thankfully we don’t need to care about them anymore.  HTML5 is the way to go, and so the only DOCTYPE we care about is “html”.

We set the title of the page, and then we set some initial viewport settings to help mobile browsers render the page correctly.  This is widely regarded as a good thing, and you can learn more about it from here: http://webdesign.tutsplus.com/tutorials/htmlcss-tutorials/quick-tip-dont-forget-the-viewport-meta-tag/

Skipping over the style and scripts for a moment, we create the main body of the page with a single div element in it.  We give it the id “map-canvas”, and then we close off the bottom of the body and mark the end of the HTML.  You won’t be surprised to read that the div we’ve just created will be where the map will soon appear.

Part 1.2 All about style

In standard HTML5 (remember the DOCTYPE from above) there are a few specifics you need to know about CSS.  If an element specifies a size as a percentage, then that percentage is calculated from the parent objects size.  Imagine you have a nested DIV, called DIV2.  It lives inside DIV1.  Where DIV1 has a fixed size of 500px by 500px, then DIV2 knows that 100% high equals 500px, but what if DIV1 didn’t have a size specified?  In that case, in standards mode, DIV2 would decide that 100% high equals zero px – because it doesn’t know any better.  This has caught people out a few times.  In order to make sure that all our DIVs can inherit a size correctly we set the height of the entire HTML page and the BODY to be 100%.  This is calculated by the browser when the page loads, and then can flow down to the elements within the page correctly.

Once we have the parent elements size set correctly (the page, and the body) we can style our map DIV to be 100% high safe in the knowledge that it has enough information to render and the correct size and not 0 px high.

Part 1.3 The Meat Section

Now we’re going to look at the actual JavaScript and understand what it’s doing and in which order.

First of all, we have the scripts in the HEAD tag.  The browser will load the head part of the HTML first and your scripts will be loaded before the page is fully rendered.  Any functionality that you make available in your scripts should be available to the rest of the page when it comes to need it.  This is generally the right way to do it.

The first SCRIPT tag takes care of loading the Google Maps code.  We tell the browser that the content of the script is text/javascript and then where to find it.  There are a couple of parameters we pass  in to the script through the URL.  The first one is “key” – this is your simple API key for access Google Maps (see above for details of where to get this key).  The second parameter is “sensor”.  This is required and must be “false” if you’re not using a GPS (or similar) to work out where you are.  In our case, we’re just picking a point on and saying “centre the map here” – so we use false.  If were using a GPS to centre the map on our current location, then this would be “true”.

This script gets loaded by the browser and now we can start to make use of the Google Maps JavaScript APIs in the rest of our page.

The second script is a bit more complex, but should be easy to understand:

We create a new function called “initialize”.
Inside that function we create an object called myLatLng.  We use the Google provided API google.maps.LatLng() to create an object which can be understood by the rest of the Google Maps API and pass in the co-ordinates of Barrow-in-Furness bus depot.  We use “var” to limit that object’s “scope” to within the “initialize” function – that is to say, we won’t be able to get access to that particular myLatLng from other functions on the page.
Next we create another thing called mapOptions.  The format of this is as per the spec here: https://developers.google.com/maps/documentation/javascript/reference?hl=en#MapOptions

There are loads of options, most of which we don’t need to worry about, so we’re just setting a few key options:  where the map is centered, how much it is zoomed in, and the type of map we see.  The map types are provided by Google as a set of constants, which are identifiable by being all in upper case.  In our case we’re using SATELLITE, but we could also use HYBRID, ROADMAP or TERRAIN.

Once we’ve set up the various mapOptions we create a new var called “map” which is an instance of a Map object as provided by the Google APIs.  We pass in to it the id of the HTML object where we want the map to appear, as we created in section 1.2, and we pass in the options var which we just created.  This is enough information for the Google APIs to set up the map as we want it and put it on the page.

The last thing we do in our example is add a marker.  A marker is the indicator which you use to highlight a point on the map.  Google provide a lot of icons and colours for us to use, but the default is the red tear-drop one, so we’ll stick with that for now.

To create a marker we create a new instance of google.Maps.Marker and set up some of the options as we did for the map itself.  We tell it the position for the marker to appear.  We use the “myLatLng” object we created earlier.  You might notice that we are using the myLatLng object twice in our example.  Once as the centre point for the map, and once for the position of the marker.  You can probably deduce from this that the marker will appear in the centre of the map.  We also tell the marker which map it should be added to.  We only have one map on our page, but if we had many this is how you’d add a marker to the correct map.  We give it a title, which is simply a string and we make it draggable by setting the draggable option to “true”.  You can read more about the marker options here: https://developers.google.com/maps/documentation/javascript/reference?hl=en#MarkerOptions

That’s very nearly it for our first simple map.  The last thing to do is use a DOM listener to trigger the above JavaScript when the page loads, and so make our map and marker appear.

google.maps.event.addDomListener is provided via the Google APIs and we pass in three pieces of information.  window is the object provided by the browser.  The ‘load’ event is actually is separate from the “onload” event you might have read about.  The load event signifies that the page is fully rendered and any JavaScript which wants to manipulate the DOM can begin work, and that’s what we want to do.  We want to swap the empty div with the id of “map” with the actual map.  So once the page is indeed loaded the command will execute the “initialize” function and all the magic will happen.

And that’s it.  We’re done.  We’ve got a map on the page centred at our chosen location, and there is a little marker to show a specific part of the map.

If you compare this to the original Beginners Guide I think you’ll say that this new version of the API is even easier to use.  I will try and find time of the next few months to jot down a few notes on doing more interesting things with the map but I hope that this will get you started.

You can see the results in this YouTube video:

Over the next few days I’ll tidy up the code and write up a blog post about how to do it.  It’s pretty straight forward though, using APIs provided by Google, MythTV and Asterisk and then just glueing them together.

This is the frequency for the PSB3 mux on Sandy Heath, I expect the encoding scheme will work pretty much where ever. At the very least it should get you started:

Frequency:  474200000
Bandwidth:  8 MHz
Inversion:  Auto
Constellation:  QAM 256
LP Coderate: 2/3
HO Coderate: Auto
Trans. Mode: 8K
Guard Interval: 1/32
Hierarchy:  None

If that doesn’t work, and you know it should (as in you are on the same transmitter) try setting the tuning timeouts higher. Also, just blindly re-trying because “WHY DOESN’T IT WORK” seemed to help.

Once I’ve got all the kinks worked out, I’ll tell you how I fixed it.  For now, this is just a test…

tl;dr: Scroll down to “How to fix Caller ID in the UK on Asterisk 11 and Dahdi 2.6.1″

I’ve been having some problems with Asterisk 1.6 for a while now, in that when I’m dialled in to conferences via SIP, or when I call someone via the work Asterisk server from my Asterisk server at home I get cut off every 15 minutes. My colleagues have all kind of accepted this now, it was starting to get annoying. It was a known problem in Asterisk 1.6, and so I was going to have to upgrade.

The Ubuntu Asterisk packages are fairly up-to-date but FreePBX isn’t packaged, and it’s a pain in the backside to get set up with the correct permissions, especially if you install Asterisk from packages. I had a spare machine anyway so I decided that I’d give FreePBX Distro a go and once I’d burnt the ISO to a CD (ya rly) I found the set up process to be simple and quick.

FreePBX Distro is a stripped down CentOS (2.6.32 kernel) which boots quick, has minimal footprint and bundles in loads and loads of FreePBX “apps” like conferences, voicemail, blacklisting and makes setting up extensions a breeze. It also includes things like kernel source for I sell settlement so building extra modules from source is pretty easy. Which is a good job, because it was about to get messy.

If you’ve found this page via Google then this will probably sound pretty familiar:

• You got a TDM400p four port FXO/FXS interface card, or a clone from someone like OpenVox (which work perfectly by the way – as do those cheap daughter cards off eBay).
• You’ve upgraded to Asterisk 11, or Asterisk 1.8 perhaps, or changed to the Dahdi 2.6.1 drivers and all of a sudden your ”UK Caller ID” has “stopped working” or at the very least become “intermittent“.
• It was working fine before, so it’s not a hardware issue and your phone provided have NOT suddenly stopped sending caller ID information, regardless of what people of forums tell you. No, in fact this is clearly a bug which has crept in somewhere along the line.
• You’ve searched and searched and tried things like setting rxgain=x.x and txgain=x.x or whatever. (Side note: setting txgain and rxgain in chan_dahdi_channels_custom.conf has no effect, the only place is seems to work is in chan_dahdi_groups.conf)
• You might have even stumbled across a post hinting that cid_rxgain=x.x will solve your problems, only it didn’t

If that sounds like you, dear reader, then read on. I am your salvation.

1. This page: http://downloads.openvox.cn/pub/drivers/callerid_patches/ (I can only attribute this discovery to divine intervention)
2. Find a new wctdm.c driver here: http://downloads.openvox.cn/pub/drivers/callerid_patches/2.6.1-wctdm.c
3. Download the 2.6.1 Dahdi source from here: http://downloads.asterisk.org/pub/telephony/dahdi-linux-complete/dahdi-linux-complete-2.6.1+2.6.1.tar.gz
4. Replace dahdi-linux-complete-2.6.1+2.6.1/linux/drivers/dahdi/wctdm.c with the version downloaded from openvox
5. Note it doesn’t compile, add the missing semicolon to line 333 (or thereabouts), successfully build and install the new driver
6. You might need to change /etc/modprobe.d/dahdi.conf and add:

options wctdm opermode=UK fwringdetect=1 battthresh=4

1. restart Dahdi and Asterisk and caller ID should now be fixed! Yay!

How to fix Caller ID in the UK on Asterisk 11 and Dahdi 2.6.1

Here is a patch against wctdm.c from stock Dahdi 2.6.1 with the fixes in: wctdm.c.patch

OR: Replace dahdi-linux-complete-2.6.1+2.6.1/linux/drivers/dahdi/wctdm.c with this one: wctdm.c

OR: If you’re running FreePBX distro 64 bit “BETA-3.211.63-5 Release Date-01-24-13″ (or close) here is a binary driver: wctdm.ko  Move it to /lib/modules/2.6.32-279.11.1.el6.x86_64/dahdi/wctdm.ko

It works for me. I have not had to increase the rxgain, or txgain or anything else. It just works.

Once that’s working, we can move on to…

How to send & receive SMS in the UK on FreePBX Distro Beta 3.211

I’ve talked about receiving and sending SMSes before, but that was quite a long time ago, and things have moved on a bit since then. I’ve also learnt a bit more about the Asterisk dial plan, and have a slightly cleaner way of doing it now.

You need to have working caller ID. Hint: see above.
You need to install smsq.
smsq doesn’t come built in FreePBX Distro so you have to build it yourself if you want to send SMS. This is pretty easy:

1. Download the Asterisk 11.2.1 source from here: http://downloads.asterisk.org/pub/telephony/asterisk/asterisk-11.2.1.tar.gz
2. Install popt: yum install popt-devel.i686 popt.i686 popt.x86_64 popt-devel.x86_64 (prerequisite for smsq)
3. In the Asterisk source directory do “./configure”
4. run “make menuselect”
5. Scroll down to “Utilities”
6. In the right hand pane, check “smsq”
7. Save & Exit
8. run “make”
9. DO NOT run “make install” as this will trash your FreePBX config
10. Once that’s done you should find an executable smsq in the utils directory

Just to be sure, create /var/spool/asterisk/sms/motx and /var/spool/asterisk/sms/mtrx
You can find more details on building Asterisk here: http://blogs.digium.com/2012/11/05/how-to-install-asterisk-11-on-centos-6/

You need to be able to send SMS in order to register with the BT SMS system, otherwise when you receive an SMS you will get a phone call from the BT robots who will read the text to you, badly.

To register with BT try this:

<path to smsq>/smsq –motx-channel=DAHDI/2/17094009 -d 00000 -m “test”

where DAHDI/2 is the channel number of your outgoing line.

If you look in the Asterisk logs you should see the message going out with some TX and RX hex dumps. If it works, a short while later you will receive a phone call from either the SMS system trying to talk modem at you, or the robot. Either way, this means your text went out. Great success!

Now we need to set up receive, which is pretty straight forward.

Edit /etc/asterisk/extensions_custom.conf and add the below. Note – I’ve added a lot of comments to explain what’s going on, it might improve readability if you take them out.

[from-internal-additional-custom]
;; This makes sure our new 'app' gets included in the FreePBX dial plan
include => app-rx-sms
[app-rx-sms]
;; This is the app to receive an SMS which will be called when we match
;; the caller ID of an incoming call to the BT computer
exten => s,1,Answer()
;; This wait seems to help
exten => s,n,Wait(0.5)
;; We simply use the built in SMS function
exten => s,n,SMS(default,a)
;; Hangup with code 16, normal termination.
exten => s,n,Hangup(16)
;; Done
;; We also extend the from-pstn context with a branch to the SMS app
;; if the caller ID matches, hence why it's necessary to get caller
;; id working first
[from-pstn-custom]
;; we add ourselves in to the end of the current dialplan
;; priority 7 gets us there...
exten => s,7,GotoIf($["${CALLERID(num)}" = "08005875290"]?app-rx-sms,s,1)
;; pretty simple - if the caller ID matches, go to the sms app
exten => s,n(dest-ext),Goto(ext-group,1100,1)
;; Note: 1100 is my "ring all" group. You will need to change 1100
;; to what ever you use. You should see this at the end of the
;; from-pstn context in extensions_additional.conf

Save that, and in asterisk do a “dialplan reload”. From smsq send an SMS to 00000 saying “register”· If everything works you should get a text file in /var/spool/asterisk/sms/mtrx.

Anyway – I finally got round to buying a Pulse-Eight CEC to USB adapter:  http://www.pulse-eight.com/store/products/104-usb-hdmi-cec-adapter.aspx

This awesome little box of tricks makes up for the lack of CEC control in the vast majority of HDMI-Out equipped graphics cards.  It’s the final piece in the jigsaw of a Linux based home entertainment device.  It allows you to talk to the other devices in your HDMI network; your surround sound amplifier and your big screen TV being the best examples (assuming they support CEC of course).

CEC has been around for a long time but for some reason it doesn’t seem to be widely used, or very well implemented, in most consumer electronics devices.  It’s a published standard but with OEM manufacturers wanting to differentiate their products and introduce a bit of vendor lock-in, they all call it something different.  The fact is that your Toshiba Regza Link TV will talk to your Sony Bravia Link Amp just fine, for the most part.  There might be a couple of proprietary things which don’t work, but on, off, volume up, volume down etc will all just work.

Pulse Eight’s USB to CEC adapter lets your computer get in on the act too, and opens up a whole realm of automatic switching, which really cuts down on the number of remote controls you need and the number of buttons you have to remember the purpose of.

The good folk at Pulse Eight have also made libCEC, an open source library to allow pretty much any software to take advantage of the USB adapter.  http://libcec.pulse-eight.com/

It comes with C++, C and .NET interfaces, and a CLI utility called cec-client.  XBMC & MythTV already support libCEC and have some neat features baked right in.  It’s good, but it’s not exactly what I was looking for – I want a bit more control.

Now, I don’t know anything about C++ or C or .NET, so until someone writes some Python bindings, my ticket to this party lies solely with the CLI utility cec-client.  It can do most things on the “transmit” side, so you can send commands to your other CEC devices fairly easily.  Acting on a request, or listening to the CEC traffic is a bit more complex – but not beyond the realms of possibility.

This weekend I wrote (and rewrote and rewrote) a couple of Bash scripts to:

• Let me control the system volume (i.e. the real hardware, not the mixer on the computer) on the TV & Amp from the PCs remote control
• Let me switch the TV & Amp on and off
• Activate proper muting, again not the mixer on the computer – the hardware itself
• Switch the amp & TV to my MythTV PC
• Power off all the hardware when the PC suspends, and then switching it all back on again
• Shut the whole lot down when the screensaver on the PC kicks in.

I make these scripts available for your amusement:

cecsimple is a client of the “server” which itself is a client of cec-client.  Fire up the server and then issue it commands down the FIFO either directly or via the abstraction layer which is cecsimple.sh.

I hooked up the volume control and amp power via “irexec” and lirc.  I tell irexec to execute, for example, “cecsimple.sh volup” or “cecsimple.sh ampon”.  If the server component is already running then these commands are sent very quickly and you don’t really notice the lag.

To switch the TV off when the PC goes in to suspend mode I added a script in /etc/pm/sleep.d which calls “cecsimple.sh tvoff” and then “cecsimple.sh tvon” when it resumes.  In theory if the TV is using the Amp to output surround sound audio then the TV will tell the Amp to turn off, it it’s not – it wont.

To switch things off when the screensaver kicks in, I simply “sudo pm-suspend” from an “xscreensaver-command -watch” script.

The practical upshot is that I can now control 99% of my media centre from a single remote control.  I’ve opted to use the remote connected to the PC as I found it to be the least laggy – using the TV remote to send up/down/left/right etc to the PC was sluggish.

I think it should be possible to parse the log output from cec-client and write a “listener” component too, but it’s probably a better idea to learn some rudimentary C and do it properly.  Or some Python bindings.  Oh yeah, and you know what would be really cool, a hook in to MythTV so that when I’m watching something in surround sound the amp turns on automatically. That would be cool.

UPDATE 3 Dec 2013:  When someone leaves the amp’s HDMI switch set on the PS3 and you switch the MythTV box on from the remote, the amp doesn’t automatically switch to MythTV.  This has been annoying me for a while now, so I fixed it.

In the scripts linked to above the “active source” command does this:

send_command "tx 45 82 11 00"

4 (the MythTV device) to 5 (the amp) – 82 (switch active source) to 1.1.0.0 – but my Sony amp just ignores this request.

I spent some time trying out a few alternatives with cec-client and I’ve found one which works, and it kinda makes sense why:

send_command "tx 45 70 11 00"

4 (MythTV) to 5 (Sony amp) 70 (System Audio Mode) 1.1.0.0 (the input where MythTV is connected)

My assumption is that amp only speaks “system audio” – what with it being an amp.  I’ve changed the “activesrc” with the 45:70:11:00 code and now it works!  (It also switches the amp on, whether I like it or not – so it’s not perfect).

Useful links:

• http://www.cec-o-matic.com/
• http://www.jwz.org/xscreensaver/man3.html
• https://github.com/Pulse-Eight/libcec

I was getting a lot of locale errors, things like this:

perl: warning: Setting locale failed.
 perl: warning: Please check that your locale settings:
 LANGUAGE = (unset),
 LC_ALL = (unset),
 LC_MESSAGES = "en_GB.UTF-8",
 LC_COLLATE = "en_GB.UTF-8",
 LC_CTYPE = "en_GB.UTF-8",
 LANG = "en_US.UTF-8"
 are supported and installed on your system.

I thought this would just go away by itself, but it didn’t – so I had to fix it. Note: I’m in the UK, so I’m using en_GB as my locale, change yours to en_US or whatever.

Type:

export LANGUAGE=en_GB.UTF-8
sudo locale-gen en_GB.UTF-8
sudo dpkg-reconfigure locales

And you should be all set.

AdSense lets you easily monetize a feed by sucking in an RSS feed, and this is how to increase your Adsense earnings, parsing it and adding the advert code and then spitting it back out again via FeedBurner.  You can do this by logging in to your AdSense account, click “My Ads” and then expand the “Feeds” section on the left. Once you’ve filled in the boxes, it’ll provide you with a new URL for your feed complete with adverts.

All this is pretty easy, but then comes the question of how you get people to see the new monetized feed?

First things first, FeedBurner can generate a little bit of HTML for you to add to your WordPress Sidebar. It’s described here http://support.google.com/feedburner/answer/78487/?hl=en& and for the sake of completeness also described here:

1. Log in to FeedBurner using the same credentials as you log in to AdSense
2. Click on Publicize from the menus near the top of the page
3. Click on Chicklet Chooser
4. Choose the type of button you’d like to appear on you blog. Personally I think the smaller “Subscribe in a reader” is the best choice
5. Then the Javascript at the button of the page is automatically updated, so copy it to the clipboard (ctrl-c)
6. Go to the Admin page of your WordPress blog
7. Choose Appearance
8. Choose Widgets
9. Drag “Text: Arbitrary text for HTML” to your sidebar
10. Drop down the Sidebar widget and paste the text from above in to the big box
11. Click Save and click close
12. You’re done!

That was the easy bit. Now new subscribers to your blog feed can simply click the button and will be taken to the new monetized feed.

But what about existing subscribers? Or 3rd party links which point direct at your “/feed/” URL? There’s the rub.

What you need to do is redirect them to the new URL, but then if you put a global redirect from the old URL to the new URL, how is FeedBurner going to keep itself updated? (It’ll get redirected back to itself and so never see any of your new posts)

The trick is to use mod_rewrite and check for the HTTP_USER_AGENT of “FeedBurner”.

I’m going to assume that you’ve already got mod_rewrite enabled in your Apache config. If you’re running WordPress on your own server then you most likely have it enabled without even realising.

In your blog’s web directory you will find a “.htaccess” file, open this up and you’ll probably see something like this:

# BEGIN WordPress
 RewriteEngine On
 RewriteBase /
 RewriteRule ^index.php$ - [L]
 RewriteCond %{REQUEST_FILENAME} !-f
 RewriteCond %{REQUEST_FILENAME} !-d
 RewriteRule . /index.php [L]

What you need to do is add these lines between “RewriteBase /” and “RewriteRule ^index.php$ – [L]”

RewriteCond %{REQUEST_URI} ^/feed/* [NC]
 RewriteCond %{HTTP_USER_AGENT} !FeedBurner
 RewriteRule ^feed/?.*$ http://feeds.feedburner.com/Whizzyorg [R,L]
 #Next rule

How does it work?

Mod_rewrite matches all the conditions in order, when it hits a Rule it decides if all the preceding conditions are met, and if so acts on the rule. If not, if moves on to the next line. The rules are the the separators between the conditions.

So the new logic says:

• IF you’re trying to get to “/feed/” (the [NC] makes it case insensitive)
• AND your USER_AGENT is NOT “FeedBurner” (the ! inverts the logic)
• THEN re-direct people to the new URL

So, if you are FeedBurner you get access to the old standard WordPress RSS feed and keep yourself updated, if you’re anyone else you get redirected and see the adverts.

The adverts are fairly inconspicuous and are easily blocked by those in the know, so I don’t have a problem with it.

Neat!

One of the drawbacks however, is that you are entirely reliant on your broadband connection.  You could, as many do, have a 3G modem/phone as a backup.  Fine for a few hours, but you’re soon going to run in to data caps, invisible proxies and traffic shapers (more so than a fixed line telco).

So what to do?  My solution has been to get a second line installed.  The big UK telcos are running a lot of offers at the moment to try and secure as many customers as they can in readiness for their Triple & Quadruple Play strategies.  This means you can get a reasonable ADSL line and line rental for about 15 to 20 quid a month.

When you’re as reliant on a connection as I am, that’s money well spent.  It also gives you a perfect opportunity to play with some new toys!

Now, you could simply plug in another wifi router to the new line, and when the primary connection fails move your connections over to the secondary one.  But where’s the fun in that?  What you really want to do is some load-balancing and multi-path routing over both the lines.  You won’t see a sudden doubling in download speeds in normal browsing, but certain activities which open a lot of connections to different IP address should benefit, and you get to play with routing tables.

Here’s what it looks like:

We don’t really get much diversity until we get south of the exchange, and even then we’re probably going to live in the same fibre all the way to The Internet, but we do work round the much more common problem of ISP configuration errors.

Here’s how it works:

Router 1 (R1) and Router 2 (R2) are two independent ADSL modem/routers  connected to a phone line each and the ethernet port connected to a NIC in a server.  These modem/routers also take care of NAT, so that any traffic coming from inside your network will be translated to appear as if it had come from the appropriate router on it’s way to the rest of the internet.  The server has three NICs in total, one each for the two routers and one back to the LAN for everyone else on the network to use as a gateway.

$IF0 = eth0
$IF1 = eth1
$IF2 = eth2

The IP addresses associated with those interfaces are:

$IP0 = 192.168.3.100
$IP1 = 192.168.1.1
$IP2 = 192.168.2.1

The routers are configured with these LAN IPs:

$P1 = 192.168.1.254
$P2 = 192.168.2.254

As you can see on the diagram, the networks are all /24 (255.255.255.0) which gives us network numbers of:

$P0_NET = 192.168.3.0
$P1_NET = 192.168.1.0
$P2_NET = 192.168.2.0

Then we configure the Ubuntu box as a router & NATing gateway:

1.  Enable IP forwarding by editing /etc/sysctl.conf and uncommenting or adding a line like this:

net.ipv4.ip_forward=1

2.  Create two new routing tables.  Edit /etc/iproute2/rt_tables and add a line for each of your providers.  For example:

1 my_isp_name
2 my_other_isp_name

3. Create some routes in each of the routing tables so the server knows where to send data.  They take the general format of:

ip route add $P1_NET dev $IF1 src $IP1 table my_isp_name
ip route add default via $P1 table my_isp_name
ip route add $P2_NET dev $IF2 src $IP2 table my_other_isp_name
ip route add default via $P2 table my_other_isp_name

Using the real numbers and the template above gives us this actual set of commands:

ip route add 192.168.1.0 dev eth1 src 192.168.1.1 table my_isp_name
ip route add default via 192.168.1.254 via eth1 table my_isp_name
ip route add 192.168.2.0 dev eth2 src 192.168.2.1 table my_other_isp_name
ip route add default via 192.168.2.254 via eth1 table my_other_isp_name

In english, that means:  In order to send traffic to the 192.168.1.0 network or the 192.168.2.0 network send it out of eth1 or eth2 from IP address 192.168.1.1 or 192.168.2.1.  These routes are then stored in the relevent routing table; either one called my_isp_name or one called my_other_isp_name.

4. Create some additional routes for communicating between the routing tables and the local interfaces.  This is as much of an efficiency saving as anything, telling each routing table the quickest way to the other interfaces.

ip route add $P0_NET dev $IF0 table my_isp_name
ip route add $P2_NET dev $IF2 table my_isp_name
ip route add 127.0.0.0/8 dev lo table my_isp_name
ip route add $P0_NET dev $IF0 table my_other_isp_name
ip route add $P1_NET dev $IF1 table my_other_isp_name
ip route add 127.0.0.0/8 dev lo table my_other_isp_name

5.  Add the routes to the main routing table, so that traffic knows how to find the networks which have the gateways to the internet on:

ip route add $P1_NET dev $IF1 src $IP1
ip route add $P2_NET dev $IF2 src $IP2

6.  Apply the correct routing rules depending on where the traffic is coming from:

ip rule add from $IP1 table my_isp_name
ip rule add from $IP2 table my_other_isp_name

7.  The science bit.  Create a route which will send traffic down each of the internet connections in turn.  This is a global default route, so any traffic which isn’t bound for the local $P0_NET network will use this rule to get to the internet.

ip route add default scope global nexthop via $P1 dev $IF1 weight 2 nexthop via $P2 dev $IF2 weight 1

As you can see, this isn’t a 50/50 split.  We’ve applied some weighting, in this case because I get more GBs with my_isp_name.  We’re actually going to send twice as much traffic down $IF1 as we are down $IF2.

8.  By default, routes to other networks are cached for 10 mins.  This means that once a route is decided on, traffic will always follow the same route for at least the next 10 minutes.  The counter will be reset every time that particular route is used.  This is great for large downloads because it means that your IP address isn’t suddenly going to change half way through getting the file.  But, for something like IRC where it’s possible that no traffic will be sent for 10 minutes, then the IRC server could well see your IP address changing as traffic flips between the two routes.  IRC servers don’t like that.  So you need to fix some routes so that traffic will always follow the same path.

ip route add 174.143.119.91/32 via $P2

This sets the route to a specific Freenode IRC server to always go via my_other_isp_name.

9.  Last of all, we need to add an extra layer of NAT so that traffic can find its way back to the host on the LAN side.  This is necessary because our internet modem/routers are not really routers at all.  They simply take traffic from one side, rewrite the headers and push it out the other side, while maintaining a look up table of what came from where and doing the reverse when a reply arrives. If you can configure the routes on to your modem, then you don’t need to be reading this in the first place, so let’s assume you can’t.

iptables --table nat --append POSTROUTING --out-interface $IF1 -j MASQUERADE
iptables --table nat --append POSTROUTING --out-interface $IF2 -j MASQUERADE
iptables --append FORWARD --in-interface $IF0 -j ACCEPT

For things like port forwarding you do now have to deal with a bit of an odd NAT situation.  For example, let’s assume I want to run a web server on the same machine that runs the routing.  I decide to only make the server available on the WAN IP address of $R1.  I set up Router 1 to NAT port 80 on the WAN to port 80 on $IP1.  Then I need to configure the web server on the routing server to listen on that interface.  Not too bad really.

And there you have it.  Two internet connections, both available to all the clients on your network.  Make sure that your clients use the IP address of the server as their default gateway and everything should just work.  The additional load on the server is negligible.

If you reboot the server you’ll need to add all the routing information again, so create yourself a Bash and have it run from /etc/rc.local or something.

/2011/02/more-asterisk-hints/

using the L(nnn,mmm,yyy) options for DIAL_TRUNK_OPTIONS. But, what if you don’t want to limit the length of calls for a specific trunk?

Well, FreePBX has a context called [macro-dialout-trunk-predial-hook] which lets you jump in at the very last moment and override any settings you like, which is perfect for this sort of thing.

I got the idea from here:

http://www.freepbx.org/book/export/html/5893

I’ve added this to extensions_custom.conf:

[macro-dialout-trunk-predial-hook]
 exten => s,1,NoOp(Trunk ${OUT_${DIAL_TRUNK}} selected)
 exten => s,n,Gotoif($["${OUT_${DIAL_TRUNK}}" != "SIP/TRUNK_NAME"]?skip)
 exten => s,n,NoOp(Setting DIAL_TRUNK_OPTIONS to Ww)
 exten => s,n,Set(DIAL_TRUNK_OPTIONS="Ww")
 exten => s,n(skip),MacroExit()

Which tests the name of the trunk which is to be used and explicitly sets DIAL_TRUNK_OPTIONS if it matches, otherwise nothing happens. This is a very powerful feature and my extremely crude hack doesn’t do it justice. I wanted to make a note of this before I forgot.

A better way might be to look at the dial prefix? I’ll investigate that option, but for now that will have to do.

Reading around the internet I found a lot of stuff about squeezing a bit more throughput from your drive.

I did a couple of benchmarks in Ubuntu 11.04:

• Adding noatime,discard to /etc/fstab

Result:  No change.  Not better or worse, but in theory the “discard” will help in the long term

Decision:  Switch this on.

• Setting the scheduler to noop in /etc/rc.local

Result: Average read rate increased by 2MB/s but Average access time went from 0.2ms to 0.3 ms.  In itself that’s not a real problem, but the graphs show a different picture.

Without NOOP

Note the scale on the right hand side.

With NOOP

The access speed is all over the place.

Decision:  Switch this off

I’d be interested in hearing if your experience differs, but it seems to me that “Doing nothing” is a valid choice.  Ubuntu, out of the box, doesn’t really require any fettling in order to get the best from your SSD.

I don’t suck at MW2 and I’m not too bad at Black Ops either, but on the PS3 Black Ops has been very unstable to the point of hanging the console.

Anyway, what really gets me cross is Lag. You can tell a match is going to be laggy the second the game opens. It feels as if you’re running through treacle. “GET A MOVE ON” I scream. And then it starts; you get shot while behind brick walls, your shots fail to connect even though “IT WAS CLEARLY A HEADSHOT WHY WONT YOU JUST DIE?”. And so on. You know what I’m talking about.

Originally I had a written a script to ping the PS3 and when it came online and tell Transmission to pause all the Torrents. When the PS3 went offline again it would start them. I thought this would be enough, but it isn’t. Even with Transmission paused and the bandwidth limited to zero kBps it still maintains a connection to other peers.

While this isn’t a significant amount of traffic over the WAN – it really does cause lag. Presumably having something like 1000 NAT sessions in the memory of my router is a bit of a stretch, so now I simply kill Transmission when the PS3 comes online. It takes a few minutes for the sessions to get pruned but it’s made a noticeable difference to the quality of my matches.

Of course, you’re still at the mercy of the internet connection of whoever hosts the match – but that’s beyond your control.

So to make things a bit easier for me after a full re-scan, I’ve put together a list of the channels I don’t watch and with a tiny bit of SQL I can trim them from my channel list.

To make things a bit easier for you here is a SQL dump of my “unwatched channels” list:

unwatched_channels.sql

And here is the SQL to trim these from your channel list:

update channel set visible=0,useonairguide=0 where name in (select name from unwatched_channels)

You’ll probably want to edit that list yourself to remove and add the channels as you prefer. Generally speaking, my list trims:

• Regional variations
• Specialist interest
• Shopping
• Games and other text based services

I’ll update this list occasionally, this page will always have my most up to date information.

• UPDATE: 6 Sept 11.  Refreshed channel list
• UPDATE: 8 Oct 11. Refreshed channel list
• UPDATE: 14 Dec 2011.  Refreshed channel list
• UPDATE: 4 Aug 2012.  Refreshed channel list
• UPDATE: 29 Dec 2013.  New list of channel IDs available here: unwatched_by_chanid.  Add a new table and import that CSV file.  Then do a “update channel set visible=0 where chanid in (select chanid from <your new table>”

I was fortunate enough to be introduced to this book by my wife reading it to our son.

• Max duration

My calling plan gives me unlimited free calls as long as those calls are under an hour in duration. Pretty standard BT stuff. If you do make a call over an hour you don’t just get charged for that bit of the call over the hour, oh no, you get charged for the whole call.
Anyway – we have the technology to defeat them!

In FreePBX under General Settings change your Asterisk Outbound Dial command options to include:

L(3360000,3240000,10000)

which will drop the call after 3360000ms (56 minutes) and should alert you at 3240000ms.

• Courtesy Tone

The above works very well and drops the calls, but without a bit of extra magic you don’t get the warning in your ear – it just drops the call. To enable the warning tones etc edit this:

/etc/asterisk/features_general_custom.conf

and add

courtesytone=beep

substituting “beep” for what ever noise you want to hear.

• Call Recording

By also adding:

Ww

to the Dial command options in #1 you can press *1 when you’re in a call to start recording.  It plays the courtesy tone to both parties though.

I also couldn’t access voicemail from the SIPGate trunk either, but I could get it to work with a phone connected to an ATA.

I had a look in the logs:

channel.c: Unable to find a codec translation path from 0x100 (g729) to 0x40 (slin)

Since slin is, I think, the Asterisk native format this is quite bad.  My assumption is that SIPGate sends through G.729 encoded audio regardless of whether you want it to or not (as configured in your allow/disallow lines for the SIP trunk PEER).

G729 from SIPGate works to the ATA because the ATA supports G729 and G729 to G729 pass-through works without any extra requirements from Asterisk.

So – in order to get SIPGate to Asterisk apps working again I installed the G729 codec binaries from here:

http://asterisk.hosting.lv/

and restarted Asterisk.  Now a:

         core show translations

shows I can convert between G729 and loads of other codecs.

I’ve been having this odd problem since I got my Asterisk box back up and running in that whenever a call came in and hung up I’d lose internet connectivity for a few seconds.

I tracked it down to the router dropping the PPP connection, which initially made me think that the polarity reversal indicating the call had hung up was causing the modem to b0rk, perhaps due to the distance between the phone socket and the modem, or my dodgy cat5 cabling, or something.

Turns out it was none of the above.  EDIT:  Bit hasty there.  Hasn’t fixed it al all.

This post:

http://forums.contribs.org/index.php/topic,43733.msg209081.html#msg209081

points to exactly my problem. I disabled the DDoS protection and no more dropped internet connections. I don’t worry about DDoS attacks too much, so for now I’m going to leave it off.

UPDATE:  None of the above actually worked.  In the end it turned out the modem had simply broken.  The NV RAM wasn’t remembering the config after a proper reboot and it was generally broken.  So I replaced the whole thing with a new DrayTek modem.  The old one did pretty well, it worked perfectly for about 3 years non-stop.

Search hints:

DrayTek VoIP Asterisk dropping ADSL PPP drop DDoS SIP

I kept getting this error when trying to send an SMS;

No call scheduled as already sending

It seems SMSq or Asterisk is quite picky about who sends the SMS. If I tired to send an SMS as myself it would fail, and then clog up /var/spool/asterisk/sms/motx with undead SMSes.

If I run smsq as the same user as Asterisk is running as it works (once you’ve deleted all the crap out of motx directory.

Just need to work out why now.

They don’t appear to have advanced functions like switching off the load independently of the supply which means you can’t power down the servers and then have the UPS switch them back on again once the power is restored and the battery has had a chance to charge, but I can live without that sort of thing.

I assumed that something somewhere would support these, and for the most part NUT does a bang up job.  It’s a bit advanced for my needs but most of the work has already been done.  In NUT the LiebertPSP is sort-of-supported by the Belkin USB driver with a few notable exceptions:

• The Online/On Battery indication doesn’t work
• The numeric values don’t work for things like Output Voltage

The numbers I can live without but the Online / On Battery is really rather important.

So I’ve cobbled together a new sub-driver using the tools supplied with Nut.  It’s far from complete but it does fix most of the annoying problems.

/wp-content/uploads/2010/10/liebertpsp.tar.gz

You’ll need to download the Nut source and put the .c and .h files in the above tarball in to the drivers directory, then you’ll need to apply this patch in the drivers directory to get the new driver included in the build and to stop the native Belkin driver from claiming the ID of the LiebertPSP:

/wp-content/uploads/2010/10/liebertpsp.diff

Then recompile Nut.

If everything works you should see a bit more info from your UPS:

battery.type: PbAc

device.mfr: Emerson Network Power

device.model: LiebertPSP

device.serial:

device.type: ups

driver.name: usbhid-ups

driver.parameter.bus: 004

driver.parameter.pollfreq: 30

driver.parameter.pollinterval: 2

driver.parameter.port: auto

driver.version: 2.4.3-2519M

driver.version.data: LiebertPSP HID 0.1

driver.version.internal: 0.35

ups.input.frequency: 50.0

ups.input.voltage: 243

ups.mfr: Emerson Network Power

ups.model: LiebertPSP

ups.output.percentload: 54

ups.output.voltage: 240

ups.powersummary.capacitygranularity1: 1

ups.powersummary.capacitygranularity2: 1

ups.powersummary.capacitymode: 2

ups.powersummary.configvoltage: 12.0

ups.powersummary.designcapacity: 100

ups.powersummary.fullchargecapacity: 100

ups.powersummary.imanufacturer: 19

ups.powersummary.ioeminformation: 19

ups.powersummary.iproduct: 1

ups.powersummary.presentstatus.batterypresent: 1

ups.powersummary.presentstatus.belowremainingcapacitylimit: 0

ups.powersummary.presentstatus.charging: 1

ups.powersummary.presentstatus.discharging: 0

ups.powersummary.presentstatus.needreplacement: 0

ups.powersummary.presentstatus.shutdownimminent: 0

ups.powersummary.rechargeable: 1

ups.powersummary.remainingcapacity: 100

ups.powersummary.remainingcapacitylimit: 38

ups.powersummary.voltage: 14

ups.powersummary.warningcapacitylimit: 38

ups.productid: 0001

ups.serial:

ups.status: OL

ups.vendorid: 10af

Search hints:

Vendor ID: 0x10af

Product ID: 0x0001

I’d installed smartmontools but it just doesn’t seem to be working.

Finally sussed it. Have a look at /etc/default/smartmontools and enable deamon

But, it would appear there is a defect in these boards, or at least a strange design, which means that they won’t lock on to some frequencies which require the 22kHz tone sending to the LNB with new drivers because there’s no link between the flange and dolphin-points.  There’s plenty to read about here:

https://bugzilla.kernel.org/show_bug.cgi?id=9476

And there’s a patch which fixes the problem by controlling the tone generator directly but it’ll never get in to the main kernel.  For your convenience here is a link to a binary driver built for Ubuntu Lucid kernel version 2.6.32-23-generic:

/wp-content/uploads/2010/08/isl6421.ko

Replace the current isl6421.ko from /lib/modules/2.6.32-23-generic/kernel/drivers/media/dvb/frontends/isl6421.ko with this one.  It might also work for newer kernel versions, or not.  Who knows?  Not me.

I’ve also got a Hauppauge S2 HD and this patched driver doesn’t seem to effect it.

Search hints:

Hauppauge Nova S plus linux won’t lock horizontal 22khz tone can’t pick up some channels. You can also check out my guide on Everything You Need to Set Up a Backyard Cinema.