Cambridge Energy Data Analysis

The smart meter rollout: current status

2015-02-18T13:49:00.003+00:00

We could write a very long blog post answering questions like "What are smart meters?", "Why do we need smart meters?", "Who is installing smart meters?", or even "Can smart meters read my mind?"... But instead, we'll stick to the data available out there, plot it, and try to analyse it! If you are interested in knowing more about smart meters and how you can benefit from them, we advise you to read the nice and simple article about smart meters on uSwitch.co.uk, or our own blog post.

A smart meter looks like this:

Figure 1: A smart meter!

The smart meter rollout timeline

In 2007, the UK government started to investigate the possibility of a smart-meters rollout. In 2009, it was agreed to proceed with the rollout with a target of replacing every single traditional meter with its smart version by 2020. An intermediate target is to have 20 million meters fitted between 2016 and 2018. The peak of smart meter installation should happen in 2019... a year before the target. Let's check where we are now.

Current status of the rollout

Thanks to the great data portal of the UK government, we can access some numbers about the smart meter rollout. The number of domestic meters by type and quarter is represented in figure 2.

Figure 2: Number of domestic gas and electricity meters by meter type and quarter. Click on "Traditional Meters" to realise how far we are from the 2020 target. No reason to panic though... 5 years to go.

First of all, some jargon clarification. "Smart Meters" is the official term to design licensed meters as defined by the regulatory organism OFGEM. "Smart-Type meters" corresponds to meters installed by utility companies which have some similarities with smart meters (they can store real-time consumption data, be accessed remotely...) but don't fully comply with the current regulation. Therefore, they will have to be replaced by official smart meters by the end of 2020. We now understand that smart meter is a very precise term and being able to display electricity consumption does not necessarily qualify your device to fit in the "smart meters" category.

By the end of 2014, 500 thousand smart meters had been installed which corresponds to a tiny percent of the totality of gas and electricity meters. The beginning of the massive rollout should however happen in 2015 which should be an exciting year for the smart meter rollout and therefore for electricity data-analysis.

References:

http://www.energy-uk.org.uk/policy/smart-meters.html
http://www.smartenergygb.org/national-rollout/how-its-happening
http://www.uswitch.com/gas-electricity/guides/smart-meters-explained/
http://www.energysavingtrust.org.uk/domestic/content/smart-meters
https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/387725/Smart_Meter_Statistics_Data_Quarter_3_2014.xlsx
http://en.wikipedia.org/wiki/Smart_meter#United_Kingdom

Processing multi-dimensional data visually

2015-02-13T10:53:00.000+00:00

In an earlier post I discussed the challenges we face currently at CEDL when we look at Big Data (http://blog.camenergydatalab.com/2014/10/big-data-crunching.html). This has been complex already, but we love new challenges here at CEDL. So let’s talk about multi-dimensionality.

If we presume that our data is arranged in a table like this:

ID	location	date	temperature	humidity
1	London	2015/02/12	4	68
2	Cambridge	2015/02/12	2.5	55

then aspects of big data refer loosely to the number of rows and multi-dimensionality of the data refers to the number of columns. Basically, we do not only have a lot of data (rows) but it is also complex due to the high number of features (columns).

Understandably, it is very challenging to extract information from such complex data in particular when we do not know what we are looking for. As part of the data exploration a data scientist will look for patterns or clusters that might tell us more about the processes which shape the data.

Of course, a data scientist wants to use the best tools available to find patterns and clusters in the data and as it turns out the most powerful machine for pattern detection is the visual cortex! The brain is your very personal supercomputer. The challenge in utilising the brain for detecting patterns in multidimensional data sets does not, thankfully, come down to brain surgery. Nevertheless, a problem still remains: how to interface the visual cortex with the data set? The only and best working interface are of course the eyes. All what is required is to transform the data set into a representation suitable for the eyes -> visual cortex interface. You might wonder why this sounds rather like an engineering problem than the typical task of a data scientist. Unfortunately, the role of a data scientist is commonly misunderstood. In fact, with today’s challenges the task is not so much about calculating statistics but to engineer a way to access and consume data.

Usually, this happens in the form of charts and plots and it is up to the data scientist to find a suitable data representation for the problem at hand: to explore data, find answers, and to communicate them.

For example, a fantastic way to represent multidimensional data are Parallel Coordinates [1] if you want to utilise the pattern recognition abilities of the brain’s visual cortex.

In a chart with parallel coordinates each column of the table is a vertical axis and each row becomes a line in the chart. Here follows an example:

This type of chart works with both discrete and continuous data. Additionally, colour and line types can be used to add some additional context to the data. Obviously, this chart is very simple but it can help us understand how parallel coordinates work. For example, looking at the location axis we can note that we have three records with location London and three with location Cambridge while looking at the axis data we note that we have two records for each day.

This chart shows you parallel coordinates in full action:

The chart shows the visualization of the mtcars dataset [2]. The data was extracted from the 1974 Motor Trend US magazine, and comprises fuel consumption and 10 aspects of car design and performance for 32 cars (1973–74 models).

A way to explore data in parallel coordinates is called “brushing”: Simply select a range over one or multiple axis and explore how the data segregates.

For example compare the models with better fuel economy versus models with less miles per gallon (mpg):

Whereas the cars with low fuel economy don’t seem to show any specific segregation, the cars with good fuel economy are light cars with 4 cylinders and small displacement.

[1] http://web.cs.ucdavis.edu/~ma/ECS289H/papers/Inselberg1997.pdf

[2] http://stat.ethz.ch/R-manual/R-devel/library/datasets/html/mtcars.html

Renewable energy in Europe: how far are we from the targets?

2015-02-10T15:00:00.000+00:00

In 2009, the European Union set mandatory targets for renewable energy use that every member state has to reach by the year 2020. In this post we will analyse the progress of each member state using the latest estimates released by Eurostat.

Shares of renewable energy in 2012

In the figure below we have the shares of gross final renewable energy consumption for each member state and how far the states are from their target: Here we note that Sweden, Estonia and Bulgaria already reached their targets while Malta Luxembourg and the UK have the lowest shares of renewable energy in gross final energy consumption. Also, Norway is the country with the highest share of renewable energy. Netherland, France and the UK are the countries furthest from their targets.

Increase since 2006

In the following chart we compare the increase of shares from 2006 to 2012 of each country: From this chart we note that all the member states increased their share of renewable energy since 2006. Another interesting fact we note here is that the three states with the highest increases are, in order, Malta, the UK and Belgium, which are also some of the countries furthest from the achievement of their targets.

Evolution of the shares from 2004 to 2012

In this figure we compare the trend of the shares of renewable energy among the biggest European countries excluding the Scandinavian ones: We can observe that Italy and the UK had the fastest growth of renewable energy shares, but while the UK share has never been comparable to the ones of the other countries, Italy was able to overtake France and Germany in 2011. We can also see that the German share had the slowest growth and that Spain is the country with the highest share since 2009.

Feed in tariffs: Small scale solar PV cost

2014-11-05T17:56:00.000+00:00

In our previous posts we focused on the excess energy that can be generated using photovoltaic (PV) panels and on on its trends. In this post we will focus on the price of the solar PV installation through two interactive visualizations based on the latest data provided by the UK government about the cost per kW of PV deployments by month.

Cost per kW installed by size band

In the following chart we can compare the cost per kW of solar deployments on a monthly basis for different size bands: Each point of this chart represents the median of the costs in a particular month while the error bars represent the 95% confidence interval of the cost in a given month. Which means that we can be 95% certain that the mean cost per kW lies within the two values reported by the error bar assuming that the cost data entered is a non-biased sample. We can note that the cost per kW installed has remained almost static over the 12 months. With the median of the cost per kW ranging from £1830 in June 2013 to £2010 in October 2013. The chart also shows that in the higher size bands the cost per kW is lower, but since the confidence intervals are wider, it is also more changeable.

Number of PV installations

In this figure we compare the number of installations over a monthly period. As in the previous chart, we differentiated deployments into three different size bands: Here we note that in correspondence of the decreases of the deployment costs, June 2013 and March 2014, the number of installations substantially increased. The month with the lowest number of installation is July 2013 while the month with the highest is March 2014. We also have an increasing trend of installation from June 2013 to November 2013.

Big Data Crunching

2014-10-31T13:34:00.000+00:00

There has been much talk about Big Data in the last years and the word cloud shows terms commonly related to the definition of Big Data. First and foremost, the most important attribute of Big Data comes as no surprise: its volume! Big Data is, as the name suggests, BIG. What big actually means in regards to bytes or number of records is circumstantial. It becomes big when your traditional way of data processing hits a wall and becomes unfeasible.

The first symptom will be that your data does not fit into memory. In the beginning you might simply beef up your computer with some extra memory. This is commonly called to scale up. A more sophisticated solution would be to load only partial data into memory as disk size is much less of a bottleneck. This is how a database operates. A join operation on two massive tables, e.g. in Postgres, will load and write many chunks of intermediate data but will eventually succeed even though all the data never fit into memory at once. Relational Databases and scaling up to more powerful computers was the gold standard for tackling growing data volumes. Things changed in particular after 2004 with Google's publication of "MapReduce: Simplified Data Processing on Large Clusters"[1].

Instead of running huge databases on expensive supercomputers the trend went to massive parallelisation on clusters of cheap hardware. With this came new challenges which MapReduce successfully addresses:

parallelisation must be easy
automatic distribution of data between the workers of the cluster
fault tolerance

If you process data on a big cluster of cheap hardware the chances are quite high that one of the computers breaks down. In the ACID world of RDBs (all or nothing transactions) this would mean we never get any results.

So what exactly is MapReduce doing differently?

The MapReduce Paradigm

Let's discuss a simple example inspired by a common task in processing genetic data: imagine you have vast amount of strings and you want to trim the last 5 letters of the strings.

In such a task we have to process each single record. This means the task is of linear order:

\[ O(n) \]

where the computational effort grows linearly with the number of records.

However, each record can be processed independently from the other records which allows to scale out the task over multiple processes, cores, or computers.

Let $k$ be the number of processes, cores, or computers available, the order of our task becomes

\[ O \left ( \frac{n}{k} \right ). \]

This is much better for the case of Big Data when $n$ is very large as we can control the computational effort easily by increasing $k$. Additionally, if one subtask fails we only have to rerun that specific subtask. A complete rollback of the transaction is not required as it would be the case in ACID conform RDBs.

Let’s consider a slightly more complex task: the "hello world" program in the world of MapReduce is the counting of word frequencies in a very big number of documents. As it was the case in the previous task, the word count of a single document is independent from the other documents, this makes the task perfectly suitable for scaling out:

A common pattern is emerging here: we use a function which maps each document to a list of independent word counts. The result of this map is a distributed list of word counts. So far our MapReduce programme comprises the following steps:

distribute the documents over multiple computers in a cluster
apply a word count function on each computer
generate a distributed list of word counts

The next step is to aggregate the distributed list of word counts. However, we want to scale out the aggregation again over multiple computers:

This aggregation step is also called reduce. The steps involved are as follows

send the same words to the same computer for aggregation (called shuffle)
apply a sum function to generate the word count over the complete set of documents (reduce)

And there we have the complete MapReduce paradigm:

Some tasks might be more difficult to translate into map and reduce steps and can require multiple rounds of mapreduce. However, the mapreduce ecosystem is growing steadily with new libraries implementing now even complex machine learning algorithms in mapreduce [3,4,5].

Last but not least, comparing mapreduce to RDB we see that mapreduce is using schema at read, which is ideal for messy and inconsistent data, and RDB is traditionally using schemas at write. In the world of Big Data the schema at read approach has the following advantages:

the flexibility to store data of any kind including unstructured or semi-structured data
it allows flexible data consumption
it allows the storage of raw data for future processing and changing objectives
it removes the cost of data formatting at the moment of data creation which results in faster data availability
it allows you to experiment with the data at low risk as the raw data can be kept to correct mistakes

There is always the elephant in the room when speaking about MapReduce: Hadoop!

Most importantly, Hadoop is not MapReduce it is just one implementation of the mapreduce framework! Hadoop is quite a beast and targets the really BIG Big Data. An alternative implementation we are using here at Cambridge Energy Data Lab is Disco.

The main reason we use Disco over Hadoop: Disco jobs are written in Python and Hadoop jobs are mainly written in Java. (Strictly speaking you can also use other languages with Hadoop). Also Disco is much lighter and easier to administrate. [2]

The word count example in Disco is as simple as the underlying problem itself:

  from disco.core import Job, result_iterator

   def map(line, params):
       for word in line.split():
           yield word, 1

   def reduce(iter, params):
       from disco.util import kvgroup
       for word, counts in kvgroup(sorted(iter)):
           yield word, sum(counts)

   if __name__ == '__main__':
       input = ["http://discoproject.org/media/text/chekhov.txt"]
       job = Job().run(input=input, map=map, reduce=reduce)
       for word, count in result_iterator(job.wait()):
           print word, count

I leave it to you to compare this with the Java version for Hadoop: https://wiki.apache.org/hadoop/WordCount

6 screens for monitoring with Raspberry Pi cluster

2014-09-19T16:12:00.000+01:00

Do you monitor?

Key performance indicators (KPI) , project progresses, server status, user logs (and many more) are constantly changing and the amount of data collected is growing bigger everyday. However it is hard to monitor these values, and sharing them with all the members of the team.

Instead, let's constantly display it.

We had too many things to display on a single monitor. Our solution was to buy 6 monitors!

6 screens for monitoring.

In order to handle 6 monitors, we would normally have to buy a massive desktop PC and a top-end graphic board. The solution we found is using Raspberry Pi! Raspberry Pi is one of the smallest computers in the world and it was born in Cambridge UK. Each Pi's performance is not perfect but it can be powerful enough if we assemble them together to form a cluster.

We immediately bought 6 Raspberry Pis. The main issue was to find a rack to store the 6 Pis. It's common to build a case from lego, however, we didn't have any Lego laying around. Our solution we found is using a shoebox!

Raspberry Pis in a Nike shoe case.

Just Do It!

It looks cool. Cambridge style monitoring environment. Try it like us!

Challenge for Excess Generation

2014-09-15T20:26:00.002+01:00

Do you consider to buy a photovoltaic (PV) installation for your house? Or do you already have some solar panels on your roof and you are looking for ways to maximise your return from this investment? Read on as we have some important advice for you:

So far the PV industry's main growth is driven by the government's support with subsidies. Unfortunately, solar panel installations have not yet achieved a competitive advantage due to their still limited efficiency and high cost. This is bound to change in the future. However, if you consider investing into a domestic PV installation today, the governmentally backed financial incentives are key to your investment's return! Here is the secret.

What is Excess Generation?

Sometimes excess generation is also referred to as surplus generation, excess electricity, or exported energy among others. Excess generation is defined as the amount of electricity generated by your rooftop panels (1. Total generation) minus your daytime electricity consumption (2. Electricity used). It is such excess generation which is available for export to a grid system (3. Export energy).

Feed-in-tariff Incentive Scheme.

Feed-in tariffs (FITs) are the most widely used policy in the world for accelerating renewable energy (RE) deployment, accounting for a greater share of RE development than either tax incentives or renewable portfolio standard (RPS) policies. In the European Union (EU), FIT policies have led to the deployment of more than 15,000 MW of solar photovoltaic (PV) power and more than 55,000 MW of wind power between 2000 and the end of 2009. In total, FITs are responsible for approximately 75% of the global PV deployment.

In a grid connected rooftop photovoltaic power station, the generated electricity can be sold to the grid at a higher price than what the grid charges for the consumers. This arrangement provides a secure return for the installer’s investment. Many consumers from across the world are switching to this mechanism due to the revenue yielded. However, the details of the financial mechanism varies depending on countries as illustrated by two examples as follows:

Case Study 1: Desincentive for Excess Generation (UK)

In the UK, consumers have a stronger incentive to minimise Excess Generation, by using the majority of their generated electricity on sunny days. The UK customers receive a guaranteed Feed-in-tariff for all electricity generation (10-14 p/kWh) , plus an 'Export tariff' (4.77 p/kWh) for their excess generation, which, however, is much smaller than the average electricity bill (12-15 p/kWh) . Therefore, customers should consume their generated electricity rather than export it to the grid.

As goes the theory.

However, in reality the ratio of excess generation is fixed to 50% of PV generation due to a lack of smart-meters. Thus, the importance of 'Excess Generation' will definitely emerge with the rollout of smart-meters in the near future.

Case Study 2: Incentive for Excess Generation (JAPAN)

In Japan, FiTs are only paid for 'Excess Generation', not "Total Generation' as it is the case in the UK. FiT’s price is currently much higher (38-42 JPY per kWh) than the average electricity bill (20-25 JPY per kWh), so customers have a strong financial incentive to maximise their amount of 'Excess Generation'. Therefore, customers are willing to change their consumption behaviour by shifting the usage of electricity-heavy appliances, such as dishwashers and wasching machines to the nighttime with cheaper electricity tariff. This individual behavioral change is expected to contribute to a nationwide peak-reduction in the future.

Our Challenge for Excess Generation

We are developing Eneberg, a Domestic PV Generation Forecasting and Trading Software. Eneberg is mainly dealing with aggregated "Excess Generation". Whilst there is a vast body of research and models dealing with PV Generation and Energy Demand, "Excess Generation" is still an open frontier. It is our aim to pioneer in that new field of Excess Generation.

In-depth excess generation analysis is already covered by this post: Energy Surplus Trends from Domestic UK Solar Panels in October 2013 to January 2014

Artificial Neural Networks

2014-09-11T17:32:00.000+01:00

An artificial neural network (ANN) is a computational model inspired by the information processing functionality of the brain. But how does the brain compute?

Generally, the central elements of computation are processing, transmission, and storage. Within the brain the neuron is the central computing element. Neurons receive signals and produce responses. The transmission of information at the neural level involves electrical signals – so called action potentials – based broadly on ions and semi-permeable membranes, and chemical signals at the synapses. In the brain the storage of information corresponds to learning which occurs at the synapses. These synapses are at the interface between neurons and regulate the transmission of information from neuron to neuron.

An ANN widely corresponds to the processing paradigm of neural networks with the nodes of the ANN being the central computing element similar to the neuron. In fact, ANNs are nothing but networks of primitive functions where the chain of function compositions transforms an input to an output. The composition of the computational model is contained implicitly in the interconnections of the nodes and is referred to as the network function. Each node comprises a primitive function transforming its input into an output:

Typically, the inputs of a node have an associated weight w by which the input x i is multiplied. The node integrates all its inputs – usually by adding the different inputs – followed by the evaluation of its primitive function f. The primitive function f computed in the node can be any function but common choices are differentiable functions such as the sigmoid function. Models of ANNs mainly differ in their choice of the primitive function, the topology of the network, and rarely in the timing of the evaluation of the primitive function. In feed-forward ANNs the network is composed of distinctive layers where each neuron only receives input from neurons of the previous layer. Accordingly, a feed-forward network has a distinct input and output layer with the intermediate layers being referred to as hidden layers:

(A second class of ANNs are recurrent networks where connections between nodes form directed cycles.)

The network function of an ANN can be understood as a universal function approximation. However, the difference between ANNs and a Taylor or Fourier series is that the function to be approximated is given not explicitly but implicitly, through a representative set of input-output examples. It will be the task of the learning algorithm to adjust the parameters of the ANN to reflect the input-output examples and to extrapolate to new input patterns in an optimal manner. The learning algorithm is an adaptive method by which the network self-organises to reflect the function to be approximated. The computational effort directly relates to the number of parameters and therefore to the topology of the network and increases substantially for more complicated ANNs. It was not until the proposal of back-propagation as a learning algorithm [Werbos, 1974] that the application of ANNs gained momentum and it has been the most widely used algorithm for neural network learning ever since.

The back-propagation algorithm uses gradient descent on the error function of an ANN in weight space. Thus, the weights of an ANN which minimise its error function are considered to be the solution of the learning problem. As a precondition for gradient descent the error function of an ANN needs to be continuous and differentiable. Since the ANN is simply the composition of its primitive functions the error function becomes differentiable if the networks primitive functions are differentiable themselves.
In the back-propagation algorithm an ANN is initialised randomly with weights. Next, the gradient of the error function is computed recursively and the weights of the ANN are adjusted accordingly using gradient descent. Because an ANN is a complex chain of a sequential function composition the chain rule plays a most important role in calculating the gradient of the network function's error. The back-propagation algorithm implements the chain rule for the recursive calculation of the gradient of the error function in weight space in a very efficient manner.

Learning in an ANN with back-propagation consists of two stages: in the first stage – the feed-forward step – the information progresses form the input layer throughout the network towards the output layer.
Each node of the network evaluates its primitive function $f_j(e)$ and emits the result $y_j$ to the connected nodes in the subsequent layer. Additionally, each node calculates and stores the derivative of its primitive function $df_j(e)/de$.

The second stage -- the back-propagation step -- consists in reversing the flow of information throughout the network whereby a unit input propagates from the output layer towards the input layer with the activation of each neuron now being the back-propagation term $\delta_j$.
At each node the back-propagation term $\delta_j$ is multiplied by the stored derivative of the node's primitive function from the previous feed-forward step which gives the gradient in weight space $(d f_j(e)/de) \delta_j$.

Finally, the weights are updated using gradient descent as given by
$$
w'_{i,j} = w_{i,j} + \alpha y_{i} \frac{d f_j(e)}{de} \delta_j
$$
with $\alpha$ being the learning rate and $w_{i,j}$ being the weight of the feed-forward connection from neuron $i$ in the previous layer to neuron $j$ in the subsequent layer.

[Werbos, 1974] Beyond regression: New tools for prediction and analysis in the behavioural sciences, Pd.D. Thesis, Harvard University (1974).
[Gurney, 1997] An introduction to neural networks, UCL Press (1997).
[Montavon, 1998] Neural Networks: Tricks of the Trade, Springer (1998).

Motivation Over CVs

2014-09-10T14:03:00.000+01:00

As part of our recruitment process we run EnergyDataSimulationChallenge on GitHub which consists of different data science and web application development tasks. Since I started the first challenge on GitHub almost one year ago, the repository has not only been forked 48 times but we also received 36 pull requests. We have reviewed some really great submissions and we hired seven candidates so far and all of them have successfully contributed to our business without exceptions. Under our current recruitment policy, we do not review any candidate's CV before we received and reviewed their pull request on GitHub.

A lot of challenges :)

As we laid out in our previous article "Talent over CVs", our motivation to run the challenges as part of our recruitment is to check an (or to get a better understanding of an) applicant's actual skills in data analysis, programming, and communication. Reviewing the actual code is the easiest way to know the applicants' technical skills. This makes sense as we are looking for people to work with who write great code rather than great CVs. However, this is not the only insight we gain from an applicant's submission. The challenge gives an applicant also the chance to showcase his or her motivation.

Why Motivation?

Why do you want to work as data scientist or a software engineer? Is it solely for the good salary or great career opportunities? Or do you actually love data and you cannot think of any greater joy than to crunch some difficult analysis? Yes, it is the most important! Your motivation is your strongest selling point and one of the most important factors for us during the recruitment process. We all love our jobs here at Cambridge Energy Data Lab and so should you!

Each member of our data science team has very strong opinions and they all are a bit stubborn. One of them is the author of popular data science blog "The Glowing Python", a guru in the world of data analysis in python. They easily have a few hours discussions about our data analysis strategy. Would you enjoy the long and complex data analysis discussions with our data geeks?

Stubborn Data Geek

Also, our developers are crazy about programming. Our CTO loves programming contests like Top Coders, and he can easily spend a whole weekend in the contests. Another developer has his own project. he works almost all of his free moments on "c3", a super cool javascript library for creating fancy charts so easily. Would you enjoy programming with the code geeks for a whole day?

Heroic CTO

I don't think you can enjoy either unless you are really motivated about data science and development. We design our challenges to be like a mini-set of our data analysis and development within our actual business operations. If you love data analysis and development, and you are highly self-motivated, I am confident you will enjoy our challenges. And if you indeed enjoy our challenges I guarantee you will love to work with us as a member of the team! As long as you enjoy working with us, we are confident you will pick up necessary skills and start to contribute to our projects in no time.

Summary

We would like to work with people who love data science and development rather than people who has impressive careers and CVs. If you are highly motivated and passionate, you can easily catch up with other members and we will do our best to support you! We are looking forward to your cool submissions!

We are waiting for your challenge!

Some insights about domestic electricity prices in the IEA countires

2014-08-22T11:31:00.000+01:00

In this post we will provide three interactive visualizations of the latest data released by the International Energy Agency (IEA) about the domestic electricity prices*.

Prices in 2013

In the first figure below we compare the prices of the domestic electricity among the countries monitored by IEA. The plot also shows which fraction of the price is represented by taxes:

In 2013, average domestic electricity prices, including taxes, in Denmark and Germany were the highest in the IEA. We also note that in Denmark the fraction of taxes paid is higher than the actual electricity price whereas in Germany the actual electricity price and the taxes are almost the same. Interestingly, USA has the lowest price and the lowest taxation.

Relationship between taxes and full prices

In this figure we highlight the correlation between taxes and full prices: Here we can see that there is a positive correlation (correlation=0.82) between the prices with taxes and the prices without taxes. This indicates that according to this data, when the full price increases, the taxes also increase. Hovering the pointer on the points we can discover that Germany and Denmark have the highest taxes, while USA, UK and Japan have the lowest. Also, we note that Ireland has expensive electricity and low taxes, while Norway shows the reverse trend.

Evolution of the prices from 2010 to 2013

Here we try to compare the trend of the prices among the five countries with the higest prices in 2013: From this chart we can observe that only in 2013 the cost of the electricity for the domestic consumers has become very similar in Germany and Denmark and that the Danish prices were substantially higher in the past. We can also see that prices in Italy and Ireland have a very similar increasing trend while prices in Austria dropped in 2012 but raised again in 2013.

*the prices are showed as pence per Kwh.

A weather forecast accuracy study

2014-08-22T11:12:00.000+01:00

Can you trust the weather forecast ?

It's a question that can trigger long passionate conversations, especially in this country (UK) where talking about the weather is deeply embedded in the culture. An answer to such a question can therefore be of interest to anybody living in the UK, but also for companies like us who work constantly with weather forecast data in order to produce accurate estimation of renewable energy production. This study will focus on the daily-averaged solar radiation over the Tokyo area from November 2013 to July 2014.

First, we plot in figure 1 the actual solar radiation in kWh over the Tokyo area.

Figure 1: Actual solar radiation over the Tokyo area from November 2013 to June 2014. It's getting hot! The first and third quartile range shows the geographic variability over the area. We observe that the solar radiation is homogeneous over Tokyo.

We can access weather forecasts for the day ahead, the following day or even a week or a month in advance. The number of days separating the date of the prediction and the date the prediction was done for is called the forecast horizon (abbreviated fh in the following). Our dataset consists of 31 different forecasts for every day, from a forecast horizon of 1 to 31. Short term forecasts (0 < fh ≤ 11) are created by weather models. These models take weather data from the past and use it to predict the future of the weather conditions (temperature, pressure, wind, humidity…). Long term forecasts (11 < fh ≤ 31) are determined from considering previous years’ averages. For all these forecasts, we can compute the error by comparing the forecast solar radiation value to the actual value. We represent in figure 2 the error distribution for both long term and short term forecasts.

Figure 2: Error distribution for long and short term forecasts. We notice that the long term distribution is biased towards the negative values (under-estimation) while the short term is slightly biased towards the positive values (over-estimation). The short term distribution shows more of an even distribution around zero, indicating that this forecast is more accurate than the long term one... but not by much!

This figure shows us that the short term forecast performs better than the long term forecast. Now we can question how this error varries with the forecast horizon. We therefore plot in figure 3 the MAPE (measure of the accuracy) and MPE (measure of the bias) as a function of the forecast horizon.

Figure 3: MAPE (measure of the accuracy) and MPE (measure of the bias) as a function of the forecast horizon. The error and the confidence interval decrease as the forecast horizon approaches 0.

We note that the forecast accuracy is improving (the error is decreasing) when the forecast horizon gets closer to 0, for which we have a vanishing error since it corresponds to actual weather values. Not only does the accuracy gets better, but so does the confidence interval (the shaded regions are narrower for short term forecasts). We also confirm that the solar radiation is under-estimated by the long term forecast and slightly over-estimated by the short term one (see right-hand-side plot). This could be due to the fact that the studied period is particularly sunny compared to previous years. However, we also notice that a 10-day weather model forecast is, on average, worse than a prediction based on previous years’ averages... We know since Lorenz that chaotic systems (weather models are good examples of chaotic equations) are very sensitive to initial conditions... Therefore, a middle term forecast based on a weather model has to be considered with caution.

So please, keep chatting about the weather but when it comes to planning a barbecue a week in advance, don't put all your trust in the forecast!

Data-Driven Summary of the Current State of the UK Energy Market

2014-07-01T12:02:00.003+01:00

Data-Driven Summary of the Current State of the UK Energy Market

In March 2014, Ofgem released a comprehensive report on the state of competition in the energy markets in the UK.¹ This report provides an in-depth analysis of how the competition amongst the Big Six and small firms has been functioning. This article aims to show several crucial stats/graphs from the report and provide the summary.

1.Tacit Coordination

The first chart below shows how the households’ average dual fuel bills has changed between 2004 and 2014. The second chart shows the pattern of averaged price announcements over the same period. The red/white circles indicate price increases/decreases. The size of the circle is proportionate to the size of the announced average price increase. Both charts imply “tacit coordination” made by Big 6.

From these charts above, average dual fuel prices increased by 24 per cent between 2009 and 2013. Also, the analysis conducted by Ofgem shows that between 2009 and 2012 unit revenues in the supply of energy by the Big 6 increased by 16.9% while unit total costs increased by 13%. Even with the increased environmental taxes imposed from the beginning of 2013, it have been also observed that there is an increase in the aggregate reported profits of the Big 6 over the last four years from £233 million in 2009 to £1.1 billion in 2012. Although further research would be required, no clear evidence has been found showing suppliers have become more efficient in reducing their own costs.

Price announcements appear to take place in identifiable rounds. Of the 16 price rounds, 12 were associated with price increases while four were associated with price decreases. The Big 6 announced their price changes at almost the same time with nearly the same size of price adjustment. Thus, prices appear to be strongly correlated over the period. Ofgem found that the correlation between firms’ price changes and price changes of competitors that preceded the firm during the period of price announcements are high – 0.93 for gas and 0.84 for electricity.

2. Weak Competition

Next, the intensity of energy market competition is analysed. There is strong evidence that suggests that big firms are not fighting hard to win new customers, which is shown in its switching statistics below.

Source: DNO information request
Note: in this figure, switching is defined as the number of customers of the six largest electricity suppliers' customers who have switched to that supplier in that year, as percentage of all electricity customers.

Both electricity and gas switching rates of the Big 6 have decreased to slightly above 1% for the last 5 years. In contrast, the small suppliers have had steady growth in the number of new customers. This implies that the Big 6 are not competing harder than before to get customers from each other as compared to small suppliers.

Switching rates could be low if most customers are happy with their suppliers, but the other evidence suggests that this is not the case. The two bar charts below show how well consumers trust in suppliers and why they do not switch suppliers.

Source: Ipsos MORI, Customer Engagement with the Electricity Market Survey 2013, p. 52

Source: Ipsos MORI, Customer Engagement with the Electricity Market Survey 2013, p. 22

The figures show that majority of suppliers do not tend to trust suppliers. Slightly above 50% of consumers are happy with their current suppliers and for the rest of them switching suppliers is a hassle.

Other evidence of weak competition is shown in the figure below. This figure shows the annual rates of switching suppliers, tariff or payment method. The number of consumers who switched gas or electricity suppliers has been decreasing from nearly 20% to slightly above 10% while number of those who switched just tariff or payment method with the same suppliers has been rising steadily up to 20%. This means that consumers tend to move to a better deal with the same suppliers rather than switching to the different suppliers.

Source: Ipsos MORI, Customer Engagement in the Energy Market, Tracking Survey 2013, pp. 10-16

3. Growing small suppliers

The total market share of new entrants have been increasing steadily up to 5% and is predicted to continue to rise. Since the energy market liberalization in 1998, there have been 24 entries to the energy retail market and 18 companies are still in business currently (May 2014). The graph below shows the new entries and exits to the energy retail market from 1996.

The graph below is the domestic electricity supply market shares for the last ten years.The market share of the Big 6 has remained high at between 11 and 25 per cent over the period while the market share of the small suppliers has remained low. However, there has been a sharp growth of smaller suppliers, and their market share passed over 5 per cent at January 2014. (This includes the acquisition by Utility Warehouse which was previously Telecom Plus, of 770,000 customer accounts from Npower.)

Source: Meter Point Administration Number (MPAN) data from Distribution Network Operators (DNOs)

4. Conclusion

Ofgem highlights 5 major recurring problems below which limit entry and restrict expansion for small suppliers: 1.Low liquidity in the wholesale market, 2.Credit and collateral requirements, 3.Difficulties in persuading some customers to switch, 4.Regulatory barriers to expansion, 5.Limits to available interconnection capacity

Despite these barriers, evidence suggess that it is still possible to enter the markets and compete with the Big 6, and that many small suppliers are doing so.

5. Acknowledgement
This article is originally written by Taiki Asakawa, an intern student. Throughout his master's dissertation is in UK electricity market reberalisation, he brought us meaningful insights of the electricity market situation and issues. Let me thank him again for his dedicated contribution.

References

State of the Market Assessment

How Do You Use Electricity ?

2014-05-19T10:05:00.000+01:00

Collecting data with smart-meters

Smart-meters, through their ability to communicate data instantly, are re-shaping the electricity market landscape. Indeed, these new-generation meters collect and transmit instantaneous electricity consumption data, which can then be used by various actors ranging from the user (e.g. to monitor its own usage) to the supplier (e.g. to forecast energy demand) via independent companies (like Cambridge Energy Data Lab) which help make more sense of this data.

In this short study, we will focus on identifying generic behaviours of electricity consumption within a dataset of more than 400 users for the February-March 2014 period. Because the raw dataset is impossible to interpret, we will perform what is usually referred to as a "model reduction."

Principal component analysis

The first step in the analysis is to perform a model reduction to define several types of days. Amongst all the unique day time-series, we select few thousands (8000 days exactly, out of the 60 days x 400 users = 24000 total days available) in order to perform a Principal Component Analysis. PCA is a linear algebra method used in order to find directions of largest variance in a dataset composed of several samples of a given variable. See figure 1 for a visual example.

Figure 1: PCA, 2-dimensional example. PCA finds the orthogonal directions which maximise the variance of the samples.

After having performed a PCA, we can order the different samples along the first principal component (PC1 in Figure 1). We perform a PCA on the dataset composed of the 8000 different days of electricity consumption and order the different days along the first principal component. The result is presented in Figure 2.

Figure 2: PCA performed on a dataset of 8000 days of electricity consumption. The days are ordered with respect to their coordinate along the first principal component.

We notice that the days are now ordered with respect to a relevant criterion since we can detect a continuous evolution from users consuming electricity during the day and in the evening (top of Figure 2) to users who mostly use electricity in the evening and at night (bottom of Figure 2). From this observation, we can therefore define different types of days.

We then simplified the full dataset thanks to this criterion, creating around 10 different "types of days." It is therefore possible to simplify the 2-month time-series by attributing a value to each day corresponding to its type. This is represented in the left panel of Figure 3.

Figure 3: Left panel: unordered "type of day" time-series. Right panel: ordered "type of day" time-series obtained by ordering along the first principal component. We notice an evolution from users who mostly use electricity during the day (top) to users who mostly use electricity at night (bottom).

By re-applying the concept of first principal component ordering, we can re-order the simplified "type of day" time-series. This is presented in the right hand side panel of Figure 3. This time, more than ordering the time-series of the days, we manage to order the users. Each separate user can therefore be attributed to a category, depending not only on the type of daily consumption, but also on the longer time-scale (weekly, monthly) behaviour. Indeed, at the top of the right side panel are represented the "type of day" time-series for the users consuming electricity mostly during the day and the evenings, whereas the bottom part of this colour plot is associated with users consuming electricity mostly at night time. We can also notice on this figure a longer time-scale behaviour ordering, and the signature of the week-ends where people tend to stay awake (and use more energy) later at night.

Conclusion

The large amount of data collected by smart-meters can only been visualised and interpreted by using advanced mathematical tools, PCA being one of them. This method allowed us to successfully define different types of days in terms of electricity usage and therefore simplify the complete users' electricity time-series. From this model reduction, another PCA was then performed to directly order the users, therefore gaining insight about the different types of electricity consumption behaviour present in the dataset.

Smart Houses with Batteries and Renewable Generation

2014-04-03T08:42:00.000+01:00

Concept

The preferred method of electricity consumption in the UK is to buy electricity from the grid - through a contract with an electricity retailer - and use it for daily needs such as lighting, heating, cooling, cooking, etc. However, several alternatives to this simple unilateral flow of energy from the grid to devices exist. We will focus on two other types of energy consumption behaviour, enabled by both domestic energy storage and production.

Energy storage consists of storing electricity under an alternative form (potential, chemical, thermal, mechanical, etc.) and, when needed, converting it back to electrical energy. Electricity storage has begun to be applied to houses and buildings, and it shows potential to both reduce customers' energy bills and help bridge the gap between energy demand and supply. Electricity is usually more expensive at peak times, and, generally, is cheaper at night than during the day. Therefore, using batteries to store electricity at night and re-use it during the day can be profitable.

Domestic energy production relies on harvesting energy from natural sources (such as wind, solar radiation, etc.) and using this stream of energy along with energy coming from the grid to meet a home's demands. The UK has installed solar panels on half a million houses so far, and plans to extend this to 10 million by 2020 [1]. With such equipment, households are not only able to produce a portion of the energy they use, but can also directly sell the energy they produce back to the grid through feed-in tariffs.

These two approaches to energy distribution, domestic production and usage can even be combined to create smart houses (see Figure 1), which are powered by incoming electricity from the grid, battery discharge and renewable energy. We can observe that widespread implementation of such strategies smoothes the electricity peak demand, allowing energy producers to more accurately predict the overall needs of the grid.

Figure 1: Schematic of a house equipped with both a solar panel and a battery.

Proof of concept

At Cambridge Energy Data Lab, we prefer crunching actual data to help us make real energy predictions. Therefore, we analysed the electricity consumption of approximately 500 houses, all equipped with lithium-ion batteries, and approximately 40% also equipped with solar panels.

We first focused on the group of houses equipped with batteries only and analysed their daily energy usage. The aggregated (averaged over all the users, for the winter period) results are displayed in Figure 2.

Figure 2: Averaged daily electricity usage for accommodations equipped with battery but no solar panel.

We can clearly see that a sizeable fraction of the energy required during the day (when the electricity rates are expensive) is shifted to the night through the charging of the battery. This stored energy is then released during the day. When electricity becomes cheaper later in the evening, the battery starts to charge anew.

The same analysis can be carried out for the group of users with solar energy generation. The results are presented in Figure 3.

Figure 3: Averaged daily electricity usage for houses equipped with battery and solar panels.

The same observations are made for this group of users: the daytime energy demand is partly moved to nighttime. Moreover, during the day, solar power is produced and allows for less intensive usage of the battery. When a surplus of energy is present, it is sold to the grid and generates income.

When looking at the data for a single day (see Figure 4), we realise that the combination of both solar power generation and energy storage with a battery is very effective at minimising electricity purchase from the grid, especially during the day when it is the most expensive.

Figure 4: Single day analysis: the energy bought from the grid is minimised during the day.

Conclusion and further analysis

This preliminary analysis shows the potential of individual electricity generation and storage. The electricity usage in a house can be distributed along different streams and optimised to reduce the overall cost for the customer. But these installations are very costly too. How long would it take you to reimburse such an investment? We plan to analyse this in the future, so stay tuned!

[1] http://www.theguardian.com/environment/2014/jan/29/uk-10-million-homes-solar-panels-2020

Energy Surplus Trends from Domestic UK Solar Panels in October 2013 to January 2014

2014-04-02T12:08:00.000+01:00

According to the statistics provided by the Department of Energy and Climate Change, around 1,900 solar schemes, Feed-In-Tariff (FiT) for solar panels installation, have been installed every week during the past year in the UK. By 5 January, about half million solar schemes had been installed in total.^1,2 The solar energy revolution has started, but how much energy can actually be produced using solar panels?

At our lab, we analyzed the energy surplus produced from 1 October 2013 to 31 January 2014, and here we report some basic statistics about this selection of customers.

In these 4 months, our customers had an average energy surplus of approximately 827 kWh, which is similar to the average monthly energy consumption of an American house.³ The lowest surplus obtained by a customer was approximately 340 kWh, while the highest was approximately 1483 kWh.

We recorded the highest peaks of the energy surplus between November and December, although two others significant peaks were recorded in the first half of October and at the beginning of January.

References

Solar panels on half a million UK buildings, figures suggest, Jessica Shankleman for BusinessGreen, part of the Guardian Environment Network theguardian.com
Weekly solar PV installation and capacity based on registration date
U.S. Energy Information Administration website

From energy consumption data to energy profiles

2014-03-31T15:54:00.000+01:00

We are all using electricity, non-stop, 24 hours a day. Electricity is our constant companion, but hardly anyone thinks about electricity when we turn on a light or watch the news on TV. Our utility bill is only some abstract amount of electricity we used over a long period of time, so we pay and forget about it.

At Cambridge Energy Data Lab, we think there is much more value in the details of how you use electricity. We believe that if you know and understand your own electricity usage better, you can save money and energy. Our aim is to create insights from your day-to-day electricity consumption to find the best price plan for you, and to convey this knowledge to help you adjust your habits and take control of your electricity usage.

Electricity consumption is not necessarily very regular and depends on many different factors. This is the electricity consumption of a household over a period of days:

The raw energy usage of an exemplary household.

We can see immediately how the energy usage of this household is extremely irregular, so will have to do some further data analysis to provide the insights we are aiming for. Let's try to create an aggregated energy profile for this household. Using time-series analysis, we first decompose the original data into a periodic day-to-day component, a trend component, and a remainder which cannot be explained by the periodic and trend component:

The decomposition of the raw electricity usage data of a particularly low-usage household. The first panel shows the original data set which we decompose into 3 separate components: a periodic day-to-day element (called seasonal in timeseries analysis), the trend component indicating a smooth overall trend, and the remainder which is the partial data which cannot be explained by the other components.

This decomposition based on a daily interval is just the first step. Let's look at the periodic day-to-day component and decompose it a second time on an hourly frequency interval:

The second decomposition of the periodic component from the previous decomposition. We now have an periodic element with a hourly frequency. The remainder has been omitted.

From the seemingly irregular raw data, we arrive at an aggregated energy profile which shows a clear trend of high electricity usage between the morning and evening hours. We can also see the periodic element of appliances, such as a refrigerator, in the seasonal component. This is just the first step, however, and much more sophisticated analysis is still to come. Nevertheless, it demonstrates that even your day-to-day electricity usage, though it doesn't look like much, is full of valuable insights.

Our goal is to develop methods and tools to make your electricity consumption data accessible to you!

Eneberg - Domestic PV Generation Forecasting and Trading Software

2014-03-24T15:48:00.000+00:00

Our Product Eneberg for PV Operators/ aggregators.

Adding "Energy" and "Bloomberg" to evoke the image of an energy-trading software, we proudly present Eneberg as one of our core services. Eneberg is a web-based software for PV operators/aggregators to provide accurate predictions for the amount of electricity will sell to the wholesale market or their consumers (usually industrial or commercial customers). As savvy readers might know already, electricity supply and demand needs to balance at a 30 minute scale for stability in the most countries. Renewable energy generators have paid tremendous resources to forecast energy generation as accurately as possible in order to minimise “imbalance cost," which they are exposed to if they generate more or less electricity than their commitment.

As illustrated below, "Eneberg" is mainly focusing on "Aggregated Domestic PV," which makes the Eneberg unique in the market. Aggregated domestic electricity generations from rooftop PVs, which is still niche but is quickly growing, is our main focus, and having smart meters allows the generation data to be easily available.

Imbalance Mechanism

Prior to generation, an electricity provider must make a commitment to the amount of electricity they will supply. Deviations from this amount are penalized with an "imbalance cost," but the penalties are asymmetrical. If the generator produces too much, it will absorb the additional costs. If the generator produces too little, there are harsh regulatory penalties to pay. Therefore, forecasting accuracy should be conservative, underestimating the required electricity supply provisions to avoid paying the regulatory penalties.

Case study in Japan

We provided this forecast for one of our major Japanese customers. The electricity market in Japan, called JPEX (Japan Electricity Power Exchange), has their imbalance mechanism represented by the blue line. Up to +3% over the supply requirement, JPEX will buy electricity at an agreed rate. Further generation, however, will be unpaid, so the profit will start to decrease after +3%. On the other hand, falling short of the supply requirement is heavily penalised, so profit will be more significantly damaged by not meeting the supply requirement. This demonstrates that the provider's commitment should be more conservative, and our calculated forecast, shown in the red distribution, reflects this.

Our Target and Approach

Our target is to minimise the imbalance cost by optimising the bidding strategies on the electricity wholesale market and improving forecasting algorithms. Our accuracy target is about 5% error, which yields about 90% of maximum profit.

There are many challenges that lie ahead of us; such as quantifying the measures for uncertainty of human behaviour and weather forecasts for geospatially distributed households. In order to have reliable prediction with these uncertainties and without the large datasets of household electricity usage that our model will eventually use, we have adopted Bayesian approaches, some of which will be explained in this blog later.

3 Ways Smart Meters Could Save You Money

2014-03-19T17:32:00.000+00:00

One of the biggest critiques of smart meters is that while they are informative to the user, they do not actively save the user money. This is partly true: currently, to see any savings on a smart meter investment, the user must actively adjust their habits based on the readings. Studies have been inconclusive on the value of smart meters in terms of reducing consumer energy expenditure, and with good reason. A fancy screen on the wall is still a fancy screen on the wall unless the user does something with the information.

The smart meters previously studied require valuable resources. They require time, dedication, and behavioral change at the household level to earn savings. In today’s hectic world that includes any combination of long commutes, hard work days, familial obligations, and much sought after personal time, there is little time left for the consumer to spend micromanaging their energy consumption. Current passive methods involve the whole-home integration of energy-saving, “smart” electronics that either consume less energy or work with the smart meter to periodically turn off at points where the rate rises too high, but these options aren’t cheap.

The question remains: Is it possible, or fair, to expect a smart meter to save us money without adopting a behavioral change or buying brand new appliances and electronics?

The answer is yes! While it may take time to see the savings, the UK smart meter rollout is intended to ultimately help energy consumers and producers alike, and I’d like to speculate on three ways that the smart meter rollout could ultimately help consumers save money without having to adjust habits:

1. Smart meters open the door to short switching periods, allowing for more competitive pricing.

For those without a smart meter, the billing process involves a “meter reader” employee of the energy provider physically coming to the customer’s building and taking the measurement. It takes time—usually weeks—to schedule an appointment, take the measurement, send the final bill, and finish the paperwork to allow a customer to switch. With a smart meter, the reading can be queried instantly, drastically reducing the processing time. With customers able to change their provider in as little as a day, energy producers will need to ensure that their rates remain competitive. Predictive models (such as those being created at Cambridge Energy Data Lab) also increase in accuracy with a known number of consumers, so, in addition to other planning benefits, it is in the producer’s interest to incentivize customers to make a long term commitment to an energy plan. One method of achieving both of these would be in the form of reduced prices on their fixed rate, long-term plans.

2. More accurate, up-to-date energy readings will help energy producers waste less.

With smart meters, companies can gather data on customer energy expenditure habits not only on a monthly basis, but as often as every half-hour. Developing predictive models allows them to more accurately predict the energy they need to generate in a given time period. This allows for less fuel to be wasted and cost savings to be realized, a savings which could theoretically be passed down to the consumer and fuel (pun intended) more competitive rates.

3. Your data is valuable!

Predictive modeling allows for energy providers to minimize costs and provide efficient service. Smart meters (supposedly) encourage consumers to use less energy, which helps to forward the government’s current goals of emissions reduction. Both the energy companies and the government need your data. The widespread, smart meter rollout is recognition of the value of this data in achieving each party’s goals. Both energy providers and the government are literally saying, “We want to know how and when you use energy so badly that we are willing to give you a free smart meter to find out!” Because the use of new smart meters are in the best interests of all parties involved, these meters will be widely available to consumers and will pass these savings unto the user.

Rather empowering, isn’t it? Once the energy providers realize these savings and the government starts to see measurable progress toward emissions reductions, the value of your data goes up over time.

If your data becomes integral to the operation of your energy provider, why give away your data for free? We hope to develop the tools to realize a future scenario where the consumer can be compensated for using a smart meter in the form of significantly reduced energy bills.

4. As an added bonus, customer service will improve.

Most customers who have tried switching providers themselves have been met with lengthy phone calls, paperwork, and frustration. The scenario probably went something like this: You probably had to track down your last bill and the energy provider’s number, call and wait for a representative, wait a few weeks for someone to come read the meter, wait a few more weeks for your final bill to come, and finally get confirmation over a month later that you were taken off the plan, in addition to setting up your next plan with the new energy provider. Of course, a sizeable portion of customers have also reported being double billed for months after because their original supplier did not correctly take them off their supply list, resulting in additional hassle as the consumers fought to correct the issue. Smart meters should help to reduce these errors, or at least reduce the time taken to correct them. As they say, time is money!

Have something to say about these decidedly (if not overly) optimistic speculations? Leave a comment below! For more information on the smart meter rollout, here is the UK government page on the consultations and policies in place/being developed: https://www.gov.uk/government/policies/helping-households-to-cut-their-energy-bills/supporting-pages/smart-meters

Talent over CVs

2014-03-19T16:46:00.002+00:00

As a young and dynamic startup, we are continuously looking for great new talent to join our team of data scientists. But talent is hard to find in a pile of CVs and, as a data science company, it seemed logical to use a data-driven approach to asses applicants. That's why we designed three simple data science challenges (which you can find on GitHub). The 3 different tasks target the different objectives of our company:

Data Analysis and Visualisation
Data Modelling, Machine Learning, and Prediction
Web Development

Each of the tasks is designed to see which programming style you use and how well you document and communicate your code. Code that is not only of high-quality, but also is well-documented and easy to understand is our priority. Please take extra care that you push a polished version of your code.
Second, quality comes before quantity. The objective is not to find the best overall method, so please focus on a single approach rather than trying several methodologies. Remember that you work on an unknown dataset, so don't assume too much. Just try to satisfy the requirements of specialised methodologies.
Finally, we are always happy to see people addressing all three challenges at once, but this is certainly not required!
But enough of the instructions and let's showcase some great examples which we have received:

Cluster analysis of energy consumption data

credit: Dimitry Foures

credit: Philip Squires

Predicting energy production using Bayesian networks

credit: Jan Teichmann

Hello World. We are "Cambridge Energy Data Lab".

2014-03-19T13:06:00.000+00:00

Hello, World!

We are an energetic data-driven start-up based in Cambridge, UK, and this is our first blog post. The company, Cambridge Energy Data Lab (CEDL), was founded in 2013 by a group of young geeks: computer scientists, data scientists, and serial entrepreneurs. As you can easily guess from the company's name, we are focused on "energy data" in the form of electricity generation/consumption numbers, and we claim ourself as a "Lab" since we are research-orientented engineering team. All of our team members either have or are pursuing masters or PhD's in engineering/computer science/mathematics/etc. from an array of international top-tier institutions, including the University of Cambridge.

Our Business

Currently, our business is based in Asia and the UK. CEDL is providing smart-energy data analysis to utility companies, wind and PV developers, and other large energy-sector players. Our first product will involve forecasting electricity generation from domestic solar panels and electricity consumption in domestic households. Large amounts of domestic electricity consumption/generation data have been collected from households via smart-meter throughout our business partners, and we hope to use this data to develop several products to address the critical needs of the energy market.

This Blog's Mission

CEDL has been developing our forecasting algorithm for the past few months, but hereafter, we will incrementally unveil our research and ideas on this blog. It is important to note that all data in our blog posts will be fully anonymized. Within this limitation, we are going to try to share the essence of our technology to the world. Hopefully, it will inspire some of the people who share the same motivation with us: to make the world sustainable by optimising the inefficiency of electricity usage.

Why Cambridge?

A significant portion of our founding members are graduates/students at the University of Cambridge, so unlike London, Tokyo, San Francisco, or Oxford, it feels like home. With the University being perennially ranked as one of the best in the world, especially in engineering/technology, attracting brilliant talent with expertise in Machine Learning, Bayesian Statistics, and Mathematics is significantly easier. Moreover, the spirit of Newton, Darwin and other prominent scientists is still alive here. We feel that the rigorous atmosphere makes us think forward, unleashing us from short-sighted issues. Many startups have found inspiration and success in Cambridge, and we hope to do the same. We want to ignite the next innovation in smart-meter derived data solutions, and we hope you will follow us along the way.

Author

Yohei
Director of Cambridge Energy Data Lab.

The picture is on the Clare Bridge, the oldest remaining bridge in Cambridge (since 1640). Feeling the footsteps of Cambridge giants, I am walking on my own way.