Diogo Gomes Blog

How I Automate my house

2017-12-04T12:00:00+00:00

Introduction

I’ve always been interested in making all things smart through the use of cheap electronics and computers, moving all the “brains” from expensive hardware to hopefully Open Source Software. Today you can buy really cheap sensors on the internet and modify them to fit your purposes thanks to the numerous tutorials you can find online in great websites such as Instructables (here is one of my favorites). Since 2012 I’ve been building up stuff for my home, first very basic, today much more complex. Since people keep asking me why and how I do this stuff, I figured I should just write about it in this extended blog post.

TLDR: I have a huge number of contraptions in my house to make it “smart”.

Version 1.0

Making my house smart was obviously one of the first objectives. I first started with a simple RF plug socket to turn on the kitchen lights at dusk and off when everyone would have gone to bed. You can get one of these online or at any local DIY store. To control it, I built a RF gateway using an Arduino and very few electronic components (RF gateway alone, deserves a post of it’s own). Connected to an SBC that consumes less than 7W I could control any plug in the house either through a webpage or a command-line, furthermore I could use the internet to add knowledge to my system, such as knowing at which time the sunset takes place each day.

Cheap RF devices

Next move, was getting notifications of when someone entered the house. Again, cheap sensor bought online, and added a receiver to my existing RF gateway. These sensors are lousy for security purposes, but to know at which time everyone leaves/enters it is perfect. Having an RF receiver, it was easy to read information from yet another inexpensive devices: a weather station. The weather station could now help me decide when to sprinkle my balcony garden. About here I decided to complicate a bit more and decided I wanted a fully automated garden (better read about it in another post).

MQTTwarn

I did not built everything over the weekend, this was about the long run and having an hobby. I was making all things by my self and writing all the code, when suddenly I realized my wife was also enjoying the smarter house and friends were asking me about how to do the same. At this time I figured my system should become more professional and started evaluating existing projects. My first approach was to use MQTTwarn. All my sensors were already publishing to MQTT so it made a lot of sense to port my automation rules to MQTTwarn. The project is great with lots of plugins for various services, and in small use case scenarios it completely fits the purpose.

Nonetheless I was at that time limited to wireless devices, and to the lack of control points in the house (for lights, window covers, heating).

Version 2.0

When I bought my current house, bricks were still being laid. That gave me the opportunity to do some changes to the house project and have the house domotics done “my way”.

From the previous house and from the previous DIY system I learned what I really wanted to have and more importantly what I found most relevant. This was important to keep costs down, as I planned for an incremental upgrade of house controls.

Wireless vs Wired

First decision was to whether have a wired or wireless solution. Z-wave has become quite popular but wireless sensors require power, batteries are a bothersome, so cables became inevitable. The other very popular solution is KNX, which pretty much has become the standard in domotics. KNX is great is many aspects except the price and hack-ability (remember this is still my hobby). I really like the ability to control everything and to be able to change everything to my needs, so I opted to have a centralized control using an Arduino based PLC from Industrial Shields. I contracted an extra switch board in the space next to the energy and communications switch boards which would have pipes to everywhere in the house, enabling me in the future to either run extra cables or exchange the existing ones by new ones (future proofing). Also for future proof reasons I ran Cat6E cable everywhere (even if the sensor on the other side uses just 2 wires), having the cable in place saves you lots of work in the future.

In the extra board I also placed a Raspberry Pi 3 to control everything, this was an upgrade from my previous system and got a perfect enclosure to place it in the board hanging from a DIN rail. The Raspberry has a nRF24 HAT for connecting to wireless sensors in the future. Since I was not putting to waste all my previous setup, I connected my existing RF gateway to the Raspberry.

Lights On

All light switches in the house are push buttons that toggle Impulse Relays, this means that pressing the button on the wall or through the PLC will have the same effect. This choice comes with a major disadvantage, I’m not aware if a light is ON or OFF, as I don’t keep track of the wall switch pushes. On the other hand the major advantage is that my PLC can be offline and all the light switches still work (I’m not dependent of the domotics system to work the lights).

For the living room I wanted to have a bit more control, such as the possibility of dimming individual head lights. Here I decided with the industry standard: DALI. I bought a USB based controller that I’ve connected to the Raspberry Pi 3. And the lights are controlled through a OSRAM DALI controller.

Window Covers

The window covers are motorized, again each window has two push buttons (up/down) and the PLC controls two relays (up/down) per window. The same as the lights applies here, I don’t know the position of the covers, but in this case I can send them all up or down (by pressing up/down for a long period of time) to reset their state to a known one.

Irrigation System

My new irrigation system comes with 2 poles to connect a trivial rain sensor (NO/NC), so I hooked those poles to the PLC and can control when the irrigation systems should work. The irrigation system runs a normal schedule and from the PLC, by simulating the rain sensor, I can disable the sprinkles whenever I find the need (already rained, raining, will rain shortly).

Scaling

If you do the math, I’m using too many ports for a single PLC. I splited my system between two PLCs by placing a second smaller and more basic PLC in the second floor switchboard and connected the two using a RS232 interface over Cat6E cable.

CCTV

Having run Cat6E all over the place, I installed an IP based CCTV system with a PoE switch. This system is more expensive that an analog one, but image quality is much better and PoE avoids the need to have several power adapters around. I’m using relative cheap cameras from HIKVISION model DS-2CD2020F-I. Since I distrust these equipments, they have a VLAN of their own and cannot communicate to the internet.

Energy Monitoring

Another important sensor is Energy monitoring. Since I’m all for Open Source and Hardware, I went with the emonTx V3 with a catch: Since I run cables everywhere I do not need the wireless transmitter and have replaced it with a serial interface that connects to my PLC. I have two clips: One measuring the total of my energy consumption (measured in the cable that comes from the power utility meter) and a second one measuring my heat-pump consumption (this equipment accounts for most of the house energy consumption, so I like to keep a close eye on it), in the future I plan to monitor a Photovoltaic installation (production) and the EV charger. You can find my fork of the firmware on GitHub.

Heating Monitoring

In the basement I have two large water tanks, one for the Underfloor heating (Piso Radiante Hidraulico - PRH) and another for sanitary hot waters (Águas Quentes Sanitárias - AQS). PRH tank is heated directly by a Heat-Pump and the AQS by Solar thermal collectors (with the heat-pump supporting when there is no sun). There were empty probe slots in the both deposits so I decided to datalog the temperatures and get a better idea of my energy expenses and savings. I’ve used inexpensive ds18b20 probes connected to an Arduino Yun. I could have gone for a cheaper solution, but since I already owned the Yun it was a no brainer. The Yun also monitors the underfloor heating circulation pump so I can know when the system has been used, using an LDR strapped tight to one the power leds of the pump.

Events/Motion detection

For security purposes I currently have a regular alarm system connected to a dispatch central. This system is closed sourced, so no tinkering for the duration of the contract. In order to detect when people enter/leave the house or the occupation of the rooms, I’ve installed several wireless sensors capable of transmitting on 433Mhz band.

Door sensors
Garage door sensor connected to an Arduino with an RF433 transmitter and used as a switch.
Motion detectors

All these sensors can be easily integrated thanks to the RF gateway.

Home Assistant

Enough talking about hardware, how do I control everything? MQTTwarn was great, but as the new system became more complex a new control software was in need. I decided to stick with Python, so between OpenHAB, Domoticz and Home Assistant (the 3 main contendants) the choice was obvious. Home Assistant is also the newest kid on the block, which translates into a more modern software stack and user interface. I installed Home Assistant (HA) in the raspberry pi 3 that lies in the extra switch board, enabling direct access to the RF gateway (through serial interface) and to the DALI controller (also USB/Serial). Connecting HA to my house is done mostly through MQTT. I kept the mosquitto server from version 1.0 and connected most of the new hardware through MQTT: PLC relays’s, Heating Monitoring and a Sonoff RF Bridge 433. The Sonoff RF bridge is used in addition to the DIY RF Gateway, in order to increase range.

Owntracks

Many of the most interesting automations are related to localization of the home inhabitants (eg: light the porch when someone arrives home). HA fully integrates with Owntracks, a platform much similar to iOS Find my Friends that can track mobile devices and share location information. Owntracks has both an iOS and an Android version and requires a MQTT Broker which in my case is hosted on my VPS for security and reliability reasons.

Database/Graphing

This system generates loads of information which I, as a data-scientist, like to keep around for future analysis. HA by default uses an SQLite database which is quite inappropriate to be run on the Raspberry Pi 3. To store everything I’m using a virtual machine inside my QNAP NAS. In this device I’ve setup MySQL as an alternative backend to HA, which keeps 7 days of history at most. InfluxDB, a time-series database that tracks all my history of only a specific set of sensors (energy, climate) and Grafana a time series analytics platform that can read all the history from influxdb.

Exposing

Exposing HA to the outside is dangerous, if you really need to do it, take has many security measurements as possible. In my current setup I expose only the Web interfaces of Grafana and HA through a NGINX reverse proxy. NGINX serves both websites through an SSL connection (thank you letsencrypt), protected with HTTP Auth and IP filter rules (limiting outside access to my work network). At home HA can be easily accessed through the home network.

Open your garage door using Bluetooth LE (BLE)

2015-10-03T19:00:00+01:00

This was one of those projects I had kept in the backlog for too long, but finally got the time and drive to finish it up.

The problem was very simple and common. In the building I share, we have a common garage door which can be open/closed through a (very expensive) radio remote control. Each apartment has a single remote control and any now or then one of them gets lost.

Since everyone nowadays owns a smartphone and Bluetooth LE (BLE) has become standard, I decided to create an appliance with a relay that can trigger the garage opening. The relay can trigger either the motor directly or (in my case) “press” the button on a radio remote control.

How to build

The system consist of 2 parts: hardware that is placed near the garage door, and an Android application installed on the smartphone. Let’s follow the next steps.

BOM (Bill Of Materials)

BLE Nano with respective MK20 board (for programming purposes only)
3V Relay (bought on eBay)
Power adapter (something on the range of 3v to 12v with 1A) or 3.7V battery (e.g. 18650)
perforated board
female header socket
wires
box

I choose the BLE Nano since it is has a very small package size and punch lots of processing power thanks to the Nordic nRF51822 (an ARM Cortex-M0 SoC which includes the BLE radio).

Build the hardware

Building is very straight forward. Start by soldering the header socket to the perforated board (comes very handy if you need to re-flash the BLE Nano later). Then wire power and the relay.

Power:

BLE Nano	Power Source
VIN	3v - 12v
GND	GND

Relay:

BLE Nano	Relay
VDD	VCC
GND	GND
RTS / P0_8	IN

On your radio remote control, identify on the push button which two points need to be short circuited to trigger the radio. Solder wires to those points and connect the wires to the relay.

Fit everything inside a nice box 😊

Programming

Get the source code from https://developer.mbed.org/users/dgomes/code/Garagem/

Compile and upload it to your BLE nano using the MK20 board.

(don’t forget to fit the BLE nano back in the box 😉)

The mobile app

The source code to the mobile application is available on https://github.com/dgomes/garagemBLE

You will also be able to find the latest version on Google Play.

Administration

All communications between your smartphone and relay hardware are kept safe through an AES128 encrypted message with a shared password. This shared password is configured to the hardware on it’s first use, and will remain so until it loses power (will be reprogrammed next time someone accesses the garage). You can share the password with anyone you wish to grant access to the garage.

That’s it!

Enjoy your safe Bluetooth LE remote control!

Greenhouse

2014-10-04T03:20:51+01:00

This is my Greenhouse, here I grow basil, thyme, coriander, parsley and other aromatic herbs which I enjoy using in my dishes. Having a greenhouse in my balcony is great, I have fresh herbs all year around. Problem being that I need to keep a constant look over it, checking humidity levels and watering.

There are several projects online by fellow makers in how to automate a greenhouse, monitoring humidity levels and triggering watering (either through a water pump or valve attached to a tap). These are all great, but lack an important feature: you can’t monitor the levels remotely, nor can you trigger watering on-demand.

In my research work at Instituto de Telecomunicações, I recently got to do a project with Escola Superior Agronomia de Coimbra (ESAC) in which we setup several sensors in one of their greenhouses and provided monitoring and actuation capabilities to the agronomic engineers there. Project was quite successfully, and immediately decided to build a similar setup at home.

So lets go through the BOM:

Greenhouse (AKI)
Plants (IKEA FRÖER)
Cables and connectors (scavenged IKEA DIODE)
Arduino (Freaduino - operates at 3.3v and its easy to plug sensors)
Valve (Electric Solenoid Valve for Water Air)
Drip Watering Irrigation Kit (Am-tech)
Relay Module (5V Relay Module for Arduino)
Humidity Sensor (Soil Hygrometer)
Temperature Sensor (Keyes DS18B20 Digital Temperature Sensor)
Light Sensor(Photo Resistor)
Radio (NRF24L01+)
Plastic Storage Box (Any will do)
Transistor (BC548 other will do)
1k Ω resistors (Resistors)
assorted cables (Breadboard Jumper Wires)
prototyping PCB (Glass Fiber Prototyping PCB Universal Board - Yellow )
Electric cable (scavenged)
CD-ROM cable (scavenged) - These are just wonderful to connect sensors! :)
Sockets - used to simplify the connection of cables to the transistor (scavenged)
Leftovers from a 3D printed Arduino case :) (you actually don’t need this part!)

Next step, bring everything together. I’ve create a diagram using fritzing, some parts do not match my BOM but you can clearly get a picture of what I have:

End result should look something like this:

Some detail on the humidity sensors power control using a transistor:

Since the solenoid valve works with 12 volts, I used an external 12v power converter and put the relay in between:

Here is my solenoid valve attached to the tap and to the dripping pipe:

Fitting everything inside the plastic box:

Ready to go:

I’m currently powering the Arduino through a 10000mAh battery bank seen on top of the box which lasts for almost a full week. In the future I’ll probably buy a solar panel.

Next step, we need to program the Arduino. I’ve got some dependencies:

Since we need to define a protocol over the radio link, I’ve create a separate file (GreenhouseProtocol.h) which we will use on the other side to monitor/control the greenhouse.

Next step? Remote controlling the Greenhouse from afar. You already have everything you need to do this (GreenhouseProtocol.h), but I’ll be doing another post in the future about my RF Gateway that enables me not only to control the Greenhouse but also several other appliances in my house.

This is a work in progress post!

Drop your questions and comments below and I’ll work towards enhancing this post

Lisbon (Mini) Maker Faire

2014-09-21T21:25:51+01:00

So this just happened :-)

I just spent 2 wonderful days at the “Pavilhão do Conhecimento - Ciência Viva” with my best friend and colleague João Paulo Barraca showcasing a maker’s version of our work in M2M communications, and with my pals from Hack’Aveiro who quickly put together a web-badged created specially for this event.

In order to show case what M2M communications are all about, we brought to the faire our very own motes and sensors that power our greenhouses and hydroponics systems, as well as a plant or two to gather everyones attention ;-)

I’ll create a post in the future with all the instructions on how to assemble a similar setup.

The environment was extraordinary, makers from all over Portugal gathered to show and tell about their projects/ideas/visions. What started as Mini event sure looked like a Mega event!

My compliments and thank you to all the organization at SAPO and “Pavilhão do Conhecimento - Ciência Viva” who where the big responsibles for the success of the event.

MeoCloud Music

2014-07-01T19:25:51+01:00

Desde o lançamento do MeoCloud no Codebits de 2012, que tenho acompanhado as diversas evoluções desde serviço de armazenamento online da PT.

Como “pioneiro” tive o previlégio de uma quota muito generosa, que infelizmente não tenho usado muito devido aos efeitos de rede (poucos utilizadores MeoCloud quando comparados com o Dropbox por exemplo). No entanto, a rapidez do serviço é notória assim como a abertura do serviço a 3rd party developers, através de uma API muito semelhante à da Dropbox mas com mais uns pózinhos.

Quando alguns meses atrás foi anunciado que o trafego para o MeoCloud não contava para os limites mensais de trafego móvel na rede M4O, surgiu-me logo a ideia de criar uma aplicação que me permitisse fazer “streaming” de música da cloud para o meu android.

Tal aplicação resolve vários problemas:

espaço no telemóvel é limitado para ter toda a minha colecção de música.
tenho todos as músicas sempre disponíveis em qualquer dispositivo/lugar.
custo 0 (já que uma alternativa capaz de responder aos pontos anteriores seria o Spotify por exemplo)

É verdade que também existe o serviço Meo Music, mas não são os meus álbuns e há um custo pelo serviço.

Assim comecei a desenvolver pelas noites dentro uma aplicação que capitaliza-se o serviço MeoCloud para poder ouvir música no meu Android.

Acontece que a ideia não é assim tão disparatada e hoje na newsletter do MeoCloud é anunciada a futura disponibilização desta funcionalidade na próxima versão da aplicação MeoCloud para Android.

Assim sendo, resolvi interromper o desenvolvimento (para o qual não tenho dedicado tempo algum nos últimos meses) e publicar o código existente no github por forma a servir de exemplo a quem esteja interessado a utilizar as API’s do MeoCloud em ambiente Android.

Já agora… se ainda não tem uma conta no MeoCloud click aqui ;)

Open Vidonn

2014-06-07T21:30:00+01:00

For some month’s now I’ve been using Moves.app which was recently acquired by Facebook. Moves allows me to track my daily activity through the use of my smartphone accelerometer and GPS. In the end of the day/week, I’m able to analyze how much exercise I did, as well as where I’ve been to… The last feature being the one which I would rather not disclose to Facebook. Moves works great except for a tiny detail, that has become a big one: It eats up all the battery. I was still using my iPhone 4S when I started using moves, battery would drain before the end of the day. My Nexus 5 is holding better, but still it’s taking a big hit.

Moves also got me interested in Quantified Self and lead me to look into alternatives such as the fitbit or jawbone up. These are basically pedometers that you can wear in your wrist next to your watch. They look very nice but are quite expensive for a pedometer.

Then a few weeks ago I found the Vidonn X5 which I purchased from dx.com. With a very attractive price, it features a pedometer and an OLED display which doubles as a watch. The Vidonn X5 has a single button which you can use to check your steps, calories and distance walked, as well as to enter in a special training mode which can track a subset of your day such as a run. The device is very small and light with a battery that can last for almost 4 days, the display is detachable from the bracelet in order to charge and synchronize with your PC (Haven’t found any version for the Mac). There are nonetheless some issues with the Vidonn X5: under direct sun light it is impossible to read the OLED display making it useless, and steps taken? well lets just call it a rough estimate… Even soo it is a great device that can be used to put your week into perspective (when were you more active, are you being last today, etc).

There are companion apps to the Vidonn X5 for both Android and iOS, but unfortunately I consider them being of poor quality.

This pretty much settled the motivation to develop an alternative Android application to read the data from my new device. Since the Vidonn X5 communicates over Bluetooth LE, it was very easy to retrieve information using a Bluetooth developer tool such as LightBlue. Equipped with patience and differential analysis, I was able to decode the Bluetooth Characteristics of the Vidonn X5.

Android 4.3 (API Level 18) introduced support for Bluetooth Low Energy, therefore being my minimum requirement for the application development besides the obvious requirement for a smartphone supporting Bluetooth LE (e.g. Nexus 4, Nexus 5, Samsung S4).

I believe my app can be of use to many people, so I decided to publish it on Google Play for others to use. The app doesn’t feature (yet) any sharing option (export to a spreadsheet, Twitter, Facebook, etc), nor does it track more then a week worth of workouts (amount of data kept in the device). There are certainly bugs around, but if you installed the app I would appreciate to ear from you, be it praises of critics.

I’ll keep a page dedicated to OpenVidonn, so please leave any comments related to the app there.

iOS to Android

2014-04-02T21:30:00+01:00

I’ve been an iOS user (in various incarnations: iphone 3G, iphone 3Gs, iphone 4s, ipad, ipad 3) for more then 4years, I’m what many of you would call an Apple fanboy.

Two years ago, I bought a Nexus 7 to get more involved with Android. At first it all seemed very strange, various applications weren’t available and I just kept getting back to my iOS devices.

But this year, instead of the next logical upgrade to an iphone 5s, I decided to go with a Google Nexus 5.

Truth be told, I was getting a little bit tired of iOS as I could not see any new feature on the iphone 5S besides an improved battery…

So I took the plunge!

Software wise, I can now do everything I used to do with my iphone, all my favorite applications are available at both stores, even though some applications do seem a lit bit sub-par with their iOS counterparts. On the other hand, the android platform does seem more accessible to developers who are not required to pay an upfront fee of 100€. I don’t sell on the respective App Store’s any of my applications, but I do like the fact that I can easily develop and distribute Android applications without any upfront fees (I’m not even considering less legal options

Hardware wise, the Nexus 5 is a great phone. It’s lighter than my iphone 4s and has a bigger screen, which was by the way my biggest concern as I privilege a small form factor. The Nexus 5 is equipped will all sorts of new technologies such as Qi charging, NFC, BLE which is create for a geek that likes to play around with this stuff. Everything is blazing fast thanks to the quad-core CPU and LTE.

In the last iterations of both operating systems there has been a lot of catching up (for both platforms) and Android KitKat in its Nexus 5 incarnation is a really good platform.

Migration was also very smooth. It was just a mather of synchronizing all my stuff with Google servers and I was ready to go. Only complain so far is the battery, it doesn’t last more than a day, but neither did the iphone 4s :)

Jekyll Reboot

2014-02-04T00:14:00+00:00

This blog is now 7 years old and time hasn’t been gentle on it.

When I started this blog it was a Wordpress blog and during the first months I mostly used to post random stuff. Twitter and Facebook where not what you see today, and therefore it made complete sense to post that random stuff here. Looking back into those posts I see lots of stuff I would not say/post today has they are not “blog material”.

Half way I started to hand pick what I posted, posts became more sporadic but also more interesting with most of my referrals pointing to those posts. This is actually the content I’m proud of, but being a personal blog it means that I also post opinions and thoughts such as this post which will definitely will not bring any referrals in.

All said, today I rebooted this blog.

I created a new jekyll theme (I will later post on that), moved to a new VPS (previous one was about to expire) and started blogging again after more than a year time. This year I intend to publish more often, show case a bit of what I’m doing as a researcher, and who knows share some thoughts on random stuff.

So if you haven’t done it yet, I invite you to subscribe my blog or follow me on twitter.

Cocoa Programming

2013-01-10T17:38:00+00:00

At a time everyone is learning Objective-C and Cocoa in order to program for the iPhone, I’ve spent most of my time programming for OSX. After doing some contributions to Sidestep during my stay at CMU, I decided to endeavor deeper and develop some apps of my own, just to scratch my back.

So during the holiday season I came up with two very simple applications:

The first app (MeuComando.app) a simple MS MediaRoom remote control app, that enables me to control my STB without reaching for the command. It follows a trivial protocol which can be easily found online (examples in java, python, etc). Thing is, I really didn’t want a crappy Java app, so I reinvented the wheel and made myself a new app.

My last app (StatusPing.app) was a request by my father. His internet connection through a router supplied by his ISP keeps dropping and unless he periodically “debugs” by opening/trying to open a webpage or firing up Terminal.app and running ping, he never knows for sure if the connection is up or not. Therefore I developed an app that stays in the menu bar of OSX right there in the top, which provides the user with a visual alert if something is wrong.

You can find my apps at github.

Two months at CMU

2012-09-11T11:31:00+01:00

In a time when Portugal is facing a severe recession, with several country men moving abroad to greener pastures link, I decided to endeavor myself on the pros and cons of living abroad even thought for a very brief period of time. Under the scope of the CMU-Portugal program, I spent July and August in Pittsburgh, Pennsylvania, USA. During this time I was a visiting professor at the ECE department at Carnegie Melon University and got the chance to be part of the research activities taking place at the Parallel Data Lab.

Although short, it was a very enriching experience. Living abroad you miss important things such as family but you also get to realize your self professionally in a away you cannot in your home country (at least not at this time).

The economic crisis is driving all the skilled people abroad, family, friends, everyone is leaving for greener pastures in the hope of a future. I didn’t want to make this post too negative, but the truth is that Portugal at this time and date, cannot offer much of the things you want for yourself and your family.

I deeply hope to be part of a solution for the problems we face, by doing my best at teaching to undergraduate and graduate level, shaping good engineers able to do anything in their life’s. But several times I wonder, if I shouldn’t be the one buying the ticket out…