mola.io http://mola.io Tue, 27 Jan 2015 23:17:03 GMT en hourly 1 Vodafone Spain's web blocking system http://mola.io/2015/01/26/vodafone-spain-web-blocking-system http://mola.io/2015/01/26/vodafone-spain-web-blocking-system/#comments Sun, 25 Jan 2015 23:00:00 GMT Santiago M. Mola http://mola.io/2015/01/26/vodafone-spain-web-blocking-system Vodafone Spain is deploying a new web blocking system dubbed Castor. This is the first time that a Spanish ISP has taken the web blocking beyond the DNS or IP level. They're performing large-scale Man-in-the-Middle attacks, intercepting HTTP requests to some website's IPs and serving an HTML redirect to their blocking system instead.

The blocking system has received a lot of public exposure when The Pirate Bay was included in the blocking list. The block affected only a few broadband customers since December 2014, but on January 26th the block was deployed to the whole Vodafone network, affecting most (if not all) customers.

Update: as of January 27th, the Vodafone Spain unblocked The Pirate Bay until it receives a court order.

Vodafone Spain has depoloyed a Deep Packet Inspection (DPI) for parental control by Optenet under codename Castor. This system inspects HTTP traffic from every Vodafone customer looking for patterns that match some rules. In this case, it intercepted HTTP request with the header Host: thepiratebay.se.

In a way, this is much more sophisticated than the traditional DNS or IP block. However, it's still completely ineffective: HTTPS traffic is not affected, http://thepiratebay.se is blocked, but https://thepiratebay.se is not.

More info

More on the TPB block by Vodafone Spain:

More on Optenet's and Vodafone's DPI systems:

The technical details

Here's a sample of what we get when we perform a request to thepiratebay.org (with DNS resolving to the right IP):

$ curl -v http://thepiratebay.org
* About to connect() to thepiratebay.org port 80 (#0)
*   Trying 80.92.65.144... connected
> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: thepiratebay.org
> Accept: */*
> 
< HTTP/1.1 200 OK
< Pragma: no-cache
< Cache-Control: no-cache
< Expires: -1
< Content-Length: 640
< Content-Type: text/html
< 
<html><head><meta http-equiv="refresh" content="0; url=http://castor.vodafone.es/public/stoppages/stop.htmopt?CAT=%5Bp2p_servers-lg%7CVDFLegal-lu%5D&RULE=%5BPoliticaLegal%5D&DATETIME=%5B26/Jan/2015:23:32:14%5D&FILE=-&CODE=06de8af1c97a7e4a6a74a83eba063d40fee3f29ce9f246546bf8b3237503f730a902db232a8b86f9d577b9f62db0c7e8e959f5fd91ea8f8e3607d4c824db1d8f79ef996cba9b33da07a38f63c32211bd39275e99f61120df&LANG=esp&optcheckwfsp=%29%3E%E8%05%EB%E5%A5%2C%2BA%AF%A8u%F7%C1%17%B87%C1e&URL=http://thepiratebay.org/&ui=D6Ime4pzx5jhP%2FHbLzMZomjBakeo0jkVKrJXvhphCbORHbWu7q6J638wQ702CwB1&IP=077.231.234.167&USER=-&CLIENTID=-"></head><body></body></html>
* Connection #0 to host thepiratebay.org left intact
* Closing connection #0

The redirection goes to castor.vodafone.es (212.73.40.87) a server running Optenet Web Server.

A request to google.com adding the HTTP header "Host: thepiratebay.org" triggers the redirection too:

$ curl -v -H "Host: thepiratebay.org" http://google.com
* About to connect() to google.com port 80 (#0)
*   Trying 216.58.211.206... connected
> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Accept: */*
> Host: thepiratebay.org
> 
< HTTP/1.1 200 OK
< Pragma: no-cache
< Cache-Control: no-cache
< Expires: -1
< Content-Length: 640
< Content-Type: text/html
< 
<html><head><meta http-equiv="refresh" content="0; url=http://castor.vodafone.es/public/stoppages/stop.htmopt?CAT=%5Bp2p_servers-lg%7CVDFLegal-lu%5D&RULE=%5BPoliticaLegal%5D&DATETIME=%5B27/Jan/2015:00:03:57%5D&FILE=-&CODE=06de8af1c97a80496a77a83eba063d40fee3f29ce9f246546bf8b3237503f730a902db232a8b86f9d577b9f62db0c7e8e959f5fd91ea8f8e3607d4c824db1d8f79ef996cba9b33da07a38f63c32211bd39275e99f61120df&LANG=esp&optcheckwfsp=%29%3E%E8%05%EB%E5%A5%2C%2BA%AF%A8u%F7%C1%17%B87%C1e&URL=http://thepiratebay.org/&ui=D6Ime4pzx5jhP%2FHbLzMZomjBakeo0jkVKrJXvhphCbORHbWu7q6J638wQ702CwB1&IP=077.231.234.167&USER=-&CLIENTID=-"></head><body></body></html>
* Connection #0 to host google.com left intact
* Closing connection #0
]]> galimatias — The URL Parser For Java http://mola.io/2014/07/02/galimatias-the-url-parser-for-java http://mola.io/2014/07/02/galimatias-the-url-parser-for-java/#comments Tue, 01 Jul 2014 22:00:00 GMT Santiago M. Mola http://mola.io/2014/07/02/galimatias-the-url-parser-for-java Today, I have released the 0.1.0 version of galimatias — The URL Parser For Java. Now that I left the 0.0.x world behind, it's a good time for a post.

Parsing a URL is supposed to be an easy task. It's supposed to be working out of the box by the standard libraries of most programming languages. But the fact is that it isn't something widely supported.

Over the years, web authors have produced all kind of URLs using different features, funky characters, unescaped non-ASCII characters, partially broken constructs, internationalized domain names (AKA IDNA Hell), etcetera. StackOverflow is full of cases illustrating this.

Web browsers have added more and more cases in order to support all cases. And now, there is finally a proper URL Standard by the WHATWG which is gaining traction among web browser vendors. But non-browser libraries are falling behind.

galimatias born to fill this gap in the JVM world. Some highlights:

  • If a web browser can parse a URL as found in the wild, galimatias can parse it too.
  • Provides efficient tools for further normalization of URL beyond parsing, which are particularly useful for web crawlers.
  • It's used by the HTML5 validator (thanks to Michael Smith!).
  • It's the first implementation of the URL Standard in Java.
  • To the best of my knowledge, it's the second idependent implementation of the URL Standard (the first being the JavaScript implementation by Anne van Kesteren.

And finally, an example:

// Let's get a funky URL
String urlString = " http:/日本.jp:80//.././[ FÜNKY ] ";

// Parse it
URL url = URL.parse(urlString);

System.out.println(url);
// OUTPUT: http://xn--wgv71a.jp/[%20F%C3%9CNKY%20]

System.out.println(url.toHumanString());
// OUTPUT: http://日本.jp/[ FÜNKY ]


// URLs can be modified with a fluent API
URL modifiedURL = url.withQuery(" let's do some query about 日本 ").withFragment(" and a fragment");

System.out.println(modifiedURL);
// OUTPUT: http://xn--wgv71a.jp/[%20F%C3%9CNKY%20]?let's%20do%20some%20query%20about%20%E6%97%A5%E6%9C%AC#and a fragment

System.out.println(modifiedURL.toHumanString());
// OUTPUT: http://日本.jp/[ FÜNKY ]?let's do some query about 日本#and a fragment


// And there are convenient canonicalizers to get URLs to a standard form

String differentUrlString = "http:/日本.jp/[20%46%c3%9c%4e%4B%59%20]";
URL differentURL = URL.parse(differentUrlString);
System.out.println(differentURL);
// OUTPUT: http://xn--wgv71a.jp/[%20%46%C3%9C%4E%4B%59%20]

URLCanonicalizer canonicalizer = new DecodeUnreservedCanonicalizer();
URL canonicalizedURL = canonicalizer.canonicalize(differentURL);
System.out.println(canonicalizedURL.toString());
// OUTPUT: http://xn--wgv71a.jp/[%20%46%C3%9C%4E%4B%59%20]
System.out.println(canonicalizedURL.toHumanString());
// OUTPUT: http://日本.jp/[ FÜNKY ]

That's it! Go get it at GitHub or Maven Central!

]]> Bing: Weird Query Rewriting http://mola.io/2013/12/03/bing-weird-query-rewriting http://mola.io/2013/12/03/bing-weird-query-rewriting/#comments Mon, 02 Dec 2013 23:00:00 GMT Santiago M. Mola http://mola.io/2013/12/03/bing-weird-query-rewriting TL;DR: Bing query rewriting is completely absurd.

I rarely use Bing, but now I have to do it from time to time for SEO purposes. It was a completely WTF moment when I queried it for "Chivalo". See the results for Bing in Spanish:

  1. Ch - Wikipedia, la enciclopedia libre - es.wikipedia.org/wiki/Ch
  2. Carolina Herrera - www.carolinaherrera.com
  3. .ch - Wikipedia, la enciclopedia libre - es.wikipedia.org/wiki/.ch

What? No result contains the keyword chivalo in the title, URL, snippet or content. The query has been rewritten from chivalo to just ch.

Let's try for chívalo (note the í which is the correct spelling for the Spanish word)...

  1. Ferdinand Cheval - Wikipedia, la enciclopedia libre - es.wikipedia.org/wiki/Ferdinand_Cheval
  2. Château Cheval Blanc - Wikipedia, la enciclopedia libre - es.wikipedia.org/wiki/Château_Cheval_Blanc
  3. Cheval - Wikipédia - fr.wikipedia.org/wiki/Cheval

Alright. chívalo is rewritten to cheval. Two different words from two different languages (Spanish and French) with no relation at all.

Let's try Bing UK:

  1. George Chuvalo - Wikipedia, the free encyclopedia - en.wikipedia.org/wiki/George_Chuvalo
  2. Chivalo - Discover, collect, buy - chivalo.com
  3. George Chuvalo - IMDb - www.imdb.com/name/nm0161546

Hey. Our website appears as the second result. But no other related results are shown, and completely unrelated results for Chuvalo are included. At least, in this case, I get a message explaining what's happening (Including results for chuvalo. Do you want results for chivalo only?) which was not the case for Bing in Spanish.

To the Bing guys: Chivalo can be a few things:

  1. A Spanish word (chívalo) for gossip it. It's quite unusual, but it's a form of the verb chivar.
  2. A Spanish family name.
  3. Chivalo.com (my startup)
  4. A small village in Mozambique. It's not recognized by Google Maps, but it's there.
  5. A small dutch store.
  6. A few more things that are barely findable.

None of them have any relation with CH, Chuvalo or cheval, at all. You are trying to be too smart, and you look dumb instead.

For search results that make sense, try DuckDuckGo or Google.

]]> Swapping languages in a Gettext PO file http://mola.io/2013/09/17/swapping-languages-in-gettext-po-file http://mola.io/2013/09/17/swapping-languages-in-gettext-po-file/#comments Mon, 16 Sep 2013 22:00:00 GMT Santiago M. Mola http://mola.io/2013/09/17/swapping-languages-in-gettext-po-file TL;DR: Here you have a script for swapping the source and target language of a Gettext PO file. Grab it.

For the first time, I'm working on a project that requires full internationalization (i.e. Chívalo!). So far, I have learned 3 things: 1) Internationalization is much harder than I've ever thought, 2) most tools suck, 3) most Java developers seem to live in a bubble where Gettext never happened. Those will be interesting topics for other posts, but today I'll just explain how to solve a simple problem: swapping source and target languages of a Gettext PO file.

We started Chívalo! as a Spanish-only website, but we always kept in mind that it would go international, eventually. We externalized messages to a file referenced by keys (e.g. nav.home=Home). This is what most people seem to be doing with Play and the officially encouraged mechanism. We quickly realized that this approach sucks big time.1 Then we switched to using Spanish strings as keys, and decided that we would find or create the proper tool to use them at some point.

The point of starting real internationalization arrived. We picked Gettext, extracted the strings, and translated everything to English in a PO file. Now it's time to switch the development to English-first. So we wrote a script to replace every Spanish string with the English equivalent by using the en-US.po file.

The only thing missing is swapping the source language (Spanish) and the target language (English) in en-US.po so that we get the new es-ES.po. I'd thought this would be the easiest part. I was surprised to find out that there seems to be no quick way of doing this.2

Here's a script for doing this with just a simple command:

]]> Unlocking data: Power generation statistics in Spain http://mola.io/2013/08/29/unlocking-data-spain-power-generation http://mola.io/2013/08/29/unlocking-data-spain-power-generation/#comments Wed, 28 Aug 2013 22:00:00 GMT Santiago M. Mola http://mola.io/2013/08/29/unlocking-data-spain-power-generation TL;DR: Spanish Government has a web service to query power generation statistics. But it is locked behind some obscurity and lack of docs. I have put together some scripts to fetch all raw data. Fork it on GitHub or download a CSV with all historic data.

For some time, I have been increasingly interested in electricity demand statistics. Some witty folks are using demanda.ree.es, a Flash visualizer for power generation and demand in Spain, to measure interesting stuff such as the impact of turn-off-the-lights protests or solidarity acts (1) or the success of a general strike (1, 2, 3). These uses are really exciting and I think we can get much more out of this kind of data. So I decided to provide easy access to it.

How?

Let's go to https://demanda.ree.es/demanda.html and fire up Google Developer Tools.

GET https://demanda.ree.es/WSVisionaV01/wsDemanda30Service?WSDL
GET https://demanda.ree.es/WSVisionaV01/wsDemanda30FinoService?WSDL
GET https://demanda.ree.es/WSVisionaV01/wsDemanda30Service?xsd=1
GET https://demanda.ree.es/WSVisionaV01/wsDemanda30FinoService?xsd=1
POST https://demanda.ree.es/WSVisionaV01/wsDemanda30FinoService
POST https://demanda.ree.es/WSVisionaV01/wsDemanda30Service
POST https://demanda.ree.es/WSVisionaV01/wsDemanda30FinoService
POST https://demanda.ree.es/WSVisionaV01/wsDemanda30FinoService

Great! It's just a SOAP web service. This is going to be easy. A Python script using suds will do... well, not so fast.

The first request is a call to the consultaTiempo, a method that returns the current time. It works. But the rest of methods require a parameter clave (Spanish for key). Dammit. And it's not just some constant API key that I can copy from Developer Tools to my script. It changes every few seconds and the older ones are no longer valid. Every time there is a call to consultaTiempo, the key changes. Hey! That means it's time-based!

After a quick analysis between some timestamps and their corresponding times, I realize that keys increase at a constant rate and they are strongly correlated to the timestamp. Each unit increase in the key corresponds to roughly 1 second in time. But I cannot figure it out.

So I downloaded the swf and disasembled the code with swftools's swfdump.

Here it is. It always calls the consultaTiempo method just before retrieving some data:

00024) + 1:1 pushstring "consultaTiempo"
00025) + 2:1 findpropstrict <q>[private]VisionaFlexCO2::_VisionaFlexCO2_Operation2_c
00026) + 3:1 callproperty <q>[private]VisionaFlexCO2::_VisionaFlexCO2_Operation2_c, 0 params
00027) + 3:1 pushstring "prevProgFino"
...
00030) + 5:1 pushstring "valoresMaxMinFino"

Following the function calls, I get to a function called prueba (Spanish for test). That's funny. It looks as if they "ofuscated" the key-generation function under a non-important name. But the key is generated there, and it seems an easy operation:

00023) + 0:1 getlocal_2
00024) + 1:1 pushbyte 5
00025) + 2:1 dup.
00026) + 3:1 callproperty <q>[namespace]http://adobe.com/AS3/2006/builtin::substr, 2 params
00027) + 1:1 coerce_s
00028) + 1:1 setlocal_2
00030) + 0:1 findpropstrict <q>[public]::parseFloat
00031) + 1:1 getlocal_2
00032) + 2:1 callproperty <q>[public]::parseFloat, 1 params
00033) + 1:1 convert_d
00034) + 1:1 setlocal_3
00036) + 0:1 getlocal_3
00037) + 1:1 pushdouble 1.307000
00038) + 2:1 divide.
00039) + 1:1 convert_d
00040) + 1:1 setlocal_3
00042) + 0:1 getlocal_3
00043) + 1:1 convert_i
00044) + 1:1 setlocal r4
00046) + 0:1 findproperty <q>[private]VisionaFlexCO2::clave

Ok. Got it. So I already have the Python equivalent:

key = str(int(float(timestamp[5:10])/ 1.307000))

Yay! It works! The only thing missing are a few SOAP calls and some boring JSON dumping code.

Show me the code!

The resulting script for fetching all historic data and generatic some graphs can be found in ree-demanda repo at GitHub. Everything it's pretty basic, but it gets the job done.

Data, NOW

I fetched all historic data from the web service and consolidated it in a single CSV file. You can find it at https://github.com/smola/ree-demanda. I plan to update it regularly and, eventually, set up a simple REST API to query it. In the mean time, if you find it is outdated, poke me at santi@mola.io and I will update it.

Visualization example

Here is a visual example of the data. Power generated since 2007 in Spain. Each line is a different power source. Data is relative to the total power demand at a given time.

Power generation by source in Spain; 2007-2013

Ideas?

What would you do with this data? Are you interested in a specific graph, crossing it with other data sources? Would it be useful for an online service? What knowledge can be obtained out of it?

If you have any idea, please, use the comments or write me at santi@mola.io.

]]>