EU unlocks a great new source of online innovation

Today the European Parliament voted to formally agree new rules on open data – effectively making a reality of the proposal which I first put forward just over 18 months ago, and making it easier to open up huge amounts of public sector data.

Great news — wonder how it’ll affect the Ordnance Survey of Ireland?

(tags: osi mapping open-data open data europe eu neelie-kroes)

UK ISPs Secretly Start Blocking Torrent Site Proxies | TorrentFreak

The next step of cat-and-mouse. Let’s see what the pirate sites do next…

The blocking orders are intended to deter online piracy and were requested by the music industry group BPI on behalf of a variety of major labels. Thus far they’ve managed to block access to The Pirate Bay, Kat.ph, H33T and Fenopy, and preparations are being made to add many others. The effectiveness of these initial measures has been called into doubt, as they are relatively easy to bypass. For example, in response to the blockades hundreds of proxy sites popped up, allowing subscribers to reach the prohibited sites via a detour. However, as of this week these proxies are also covered by the same blocklist they aim to circumvent, without a new court ruling. The High Court orders give music industry group BPI the authority to add sites to the blocklist without oversight. Until now some small changes have been made, mostly in response to The Pirate Bay’s domain hopping endeavors, but with the latest blocklist update a whole new range of websites is being targeted.

(tags: bittorrent blocking filesharing copyright bpi piracy pirate-bay proxies fenopy kat.ph h33t filtering uk)

There’s a map for that

‘Not long ago, we began rendering 3D models on GitHub. Today we’re excited to announce the latest addition to the visualization family – geographic data. Any .geojson file in a GitHub repository will now be automatically rendered as an interactive, browsable map, annotated with your geodata.’ As this HN comment notes, https://news.ycombinator.com/item?id=5875693 — ‘I’d much rather Github cleaned up the UI for existing features than added these little flourishes that I can’t imagine even 1% of users use.’ Something is seriously wrong in how GitHub decides product direction if this kind of wankology (and that Judy-array crap) is what gets prioritised. :( (via Marc O’Morain)

(tags: via:marc github mapping maps geojson hacking product-management ui pull-requests)

Lawsuit Filed To Prove Happy Birthday Is In The Public Domain; Demands Warner Pay Back Millions Of License Fees | Techdirt

The issue [...] is that it’s just not cost effective for anyone to actually stand up and challenge Warner Music, who has strong financial incentive to pretend the copyright is still valid. Well, apparently, someone is pissed off enough to try. The creatively named Good Morning to You Productions, a documentary film company planning a film about the song Happy Birthday, has now filed a lawsuit concerning the copyright of Happy Birthday and are seeking to force Warner/Chappell to return the millions of dollars it has collected over the years. That’s going to make this an interesting case.

(tags: music copyright law via:bwalsh public-domain happy-birthday songs warner-music lawsuits)

graphite-metrics

metric collectors for various stuff not (or poorly) handled by other monitoring daemons Core of the project is a simple daemon (harvestd), which collects metric values and sends them to graphite carbon daemon (and/or other configured destinations) once per interval. Includes separate data collection components (“collectors”) for processing of: /proc/slabinfo for useful-to-watch values, not everything (configurable). /proc/vmstat and /proc/meminfo in a consistent way. /proc/stat for irq, softirq, forks. /proc/buddyinfo and /proc/pagetypeinfo (memory fragmentation). /proc/interrupts and /proc/softirqs. Cron log to produce start/finish events and duration for each job into a separate metrics, adapts jobs to metric names with regexes. Per-system-service accounting using systemd and it’s cgroups. sysstat data from sadc logs (use something like sadc -F -L -S DISK -S XDISK -S POWER 60 to have more stuff logged there) via sadf binary and it’s json export (sadf -j, supported since sysstat-10.0.something, iirc). iptables rule “hits” packet and byte counters, taken from ip{,6}tables-save, mapped via separate “table chain_name rule_no metric_name” file, which should be generated along with firewall rules (I use this script to do that).

Pretty exhaustive list of system metrics — could have some interesting ideas for Linux OS-level metrics to monitor in future.

(tags: graphite monitoring metrics unix linux ops vm iptables sysadmin)

Former NSA Boss: We Don’t Data Mine Our Giant Data Collection, We Just Ask It Questions

‘Well, that’s – no, we’re going to use it. But we’re not going to use it in the way that some people fear. You put these records, you store them, you have them. It’s kind of like, I’ve got the haystack now. And now let’s try to find the needle. And you find the needle by asking that data a question. I’m sorry to put it that way, but that’s fundamentally what happens. All right. You don’t troll through the data looking for patterns or anything like that. The data is set aside. And now I go into that data with a question that – a question that is based on articulable(ph), arguable, predicate to a terrorist nexus.’

Yep, that’s data mining.

(tags: data-mining questions haystack needle nsa usa politics privacy data-protection michael-hayden)

fastutil

fastutil extends the Java™ Collections Framework by providing type-specific maps, sets, lists and queues with a small memory footprint and fast access and insertion; provides also big (64-bit) arrays, sets and lists, and fast, practical I/O classes for binary and text files. It is free software distributed under the Apache License 2.0. It requires Java 6 or newer.

used by Facebook (along with Apache Giraph, Netty, Unsafe) to speed up “weekend Hive jobs” to “coffee breaks”. http://www.slideshare.net/nitayj/2013-0603-berlin-buzzwords

(tags: via:highscalability facebook giraph optimization java speed fastutil collections data-structures)

Big Memory, Part 4

good microbenchmarking of a bunch of Java collections; Trove, fastutil, PCJ, mahout-collections, hppc

(tags: java collections benchmarks performance speed coding data-structures optimization)

Spamalot reigns: the spoils of Ireland’s EU kingship | The Irish Times – Thu, Jun 13, 2013

The spam presidency. As European citizens are made the miserable targets of unimpeded “direct marketing”, that may be how Ireland’s stint in the EU presidency seat is recalled for years to come. Under the guiding hand of Minister for Justice Alan Shatter, the Council of the European Union has submitted proposals for amendments to a proposed new data protection regulation, all of which overwhelmingly favour business and big organisations, not citizens. The most obviously repugnant and surprising element in the amendments is a watering down of existing protections for EU citizens against the willy-nilly marketing Americans are forced to endure. In the US there are few meaningful restrictions on what businesses can do with people’s personal information when pitching products and services at them. In the EU, this has always been strictly controlled; information gathered for one purpose cannot be used by a business to sell whatever it wants – unless you have opted in to receive such solicitations. This means you are not constantly bombarded by emails and junk mail, nor do you get non-stop phone calls from telemarketers. Under the proposed amendments to the draft data protection regulation, direct marketing would become a legal form of data processing. In effect, this would legitimise spam email, junk print mail and marketing calls. This unexpected provision signals just how successful powerful corporate lobbyists have been in convincing ministers that business matters more than privacy or giving citizens reasonable control over their personal information. Far worse is contained in other amendments, which in effect turn the original draft of the regulation upside down.

Fantastic article from Karlin Lillington in today’s Times on the terrible amendments proposed for the EU’s data protection law.

(tags: eu law prism data-protection privacy ireland ec marketing spam anti-spam email)

Vagrant and Chef to provision dev test environments

We have recently switched from a manually configured development environment to a nearly fully automated one using Vagrant, Chef, and a few other tools. With this transition, we’ve moved to an environment where data on the dev boxes is considered disposable and only what’s checked into the SCM is “real”. This is where we’ve always wanted to be, but without the ability to easily rebuild the dev environment from scratch, it’s hard to internalize this behavior pattern.

(tags: dev osx chef vagrant testing vms coding)

Rapid Response: The NSA Prism Leak

‘The biggest leak in the history of US security or nothing to worry about? A breach of trust and a data protection issue or a necessary secret project to protect American interests? [Tomorrow] lunchtime Science Gallery Rapid Response event [sic] will pick through the jargon, examine the minutiae of the National Security Agency’s PRISM project and the whistle blower Edward Snowden’s revelations, and discuss what it means for you and everyone. And we’ll look at the bigger picture too. Journalist Una Mullally will chair a panel of guests on the story that everyone is talking about. ‘

(tags: science-gallery panel-discussions dublin nsa prism panel)

Music firms secure orders blocking access to Pirate Bay – Crime & Law News from Ireland & Abroad | The Irish Times – Wed, Jun 12, 2013

Four major music companies have secured court orders requiring six internet service providers to block access by subscribers to various Pirate Bay websites within some 30 days in a bid to prevent illegal downloading of copyright music and other material. [...] Today, Mr Justice Brian McGovern said he was satisfied to make the order in circumstances including that new copyright laws here and in the EU permitted such orders to be made. He said he fully agreed with a previous High Court judge who had said he would make such blocking orders if the law permitted and noted the law now allowed for such orders. The form of the orders means the music companies will not have to make fresh applications to court if Pirate Bay changes its location on the internet.

(tags: pirate-bay blocking filtering internet ireland upc eircom vodafone digiweb three imagine o2 copyright)

Labour TD ignores tough questions on web case

I [Tom Murphy] have asked [Sean Sherlock] a question: Does he have any comment about the lawsuit between EMI and UPC (and a raft of other ISPs too btw) which is using his SI to attempt to block PirateBay? A court case he said would not happen. Now, I am blocked from following him on Twitter. This is not how a proper political system works.

(tags: politics ireland twitter sean-sherlock tom-murphy boards devore copyright)

PRISM explains the wider lobbying issues surrounding EU data protection reform | EDRI

The US has very successfully and expertly lobbied against the [EU] data protection package directly, it has mobilised and supported US industry lobbying. US industry has lobbied in its own name and mobilised malleable European trade associations to lobby on their behalf to amplify their message, “independent” “think tanks” have been created to amplify their message again. The result is not just the biggest lobbying effort that Brussels has ever seen, but also the broadest. Compliant Members of the European Parliament (MEPs) and EU Member States [...] have been imposing a “death by a thousand cuts” on the Regulation. Where previously there was a clear obligation to collect the “minimum necessary” data for any given service, the vague requirement to retain “not excessive” data is now preferred. Where previously companies could only use data for purposes that were “compatible” with the original reason for collecting the data, the Irish EU Presidency (pdf) has proposed a comical definition of “compatible” based on five elements, only one of which is related to the dictionary definition of the word. Members of the European Parliament and EU Member States are falling over themselves to ensure that the EU does not maintain its strategic advantage over the US. In addition to dismantling the proposed Regulation, countries like the UK desperately seek to delay the whole process and subsume it into the EU-US free trade agreement (the so-called “investment partnership” TTIP/TAFTA), which would subordinate a fundamental rights discussion in a trade negotiation. The UK government is even prepared to humiliate itself by arguing in favour of the US position on the basis that two and a half years (see Communication from 2010, pdf) of discussion is too fast!

(tags: edri data-protection eu ec ireland politics usa meps privacy uk free-trade)

Microsoft admits US government can access EU-based cloud data

interesting point from an MS Q&A back in 2011, quite relevant nowadays:

Q: Can Microsoft guarantee that EU-stored data, held in EU based datacenters, will not leave the European Economic Area under any circumstances — even under a request by the Patriot Act? A: Frazer explained that, as Microsoft is a U.S.-headquartered company, it has to comply with local laws (the United States, as well as any other location where one of its subsidiary companies is based). Though he said that “customers would be informed wherever possible,” he could not provide a guarantee that they would be informed — if a gagging order, injunction or U.S. National Security Letter permits it. He said: “Microsoft cannot provide those guarantees. Neither can any other company.” While it has been suspected for some time, this is the first time Microsoft, or any other company, has given this answer. Any data which is housed, stored or processed by a company, which is a U.S. based company or is wholly owned by a U.S. parent company, is vulnerable to interception and inspection by U.S. authorities. 

(tags: microsoft privacy cloud-computing eu data-centers data-protection nsa fisa usa)

IAB Europe awards MEP Sean Kelly for standing up for data privacy rights (video) – Ireland’s CIO and strategy news and reports service – Siliconrepublic.com

Irish MEP serving as a rapporteur on reform of the EU data protection regime, was given an award by an advertising trade group last month:

Sean Kelly, Fine Gael MEP for Ireland South [who serves as the EU’s Industry Committee Rapporteur for the General Data Protection Regulation], has been selected to receive the prestigious IAB Europe Award for Leadership and Excellence for his approach to dealing with privacy concerns over shortcomings in the European Commission’s data protection proposal. IAB Europe represents more than 5,500 online advertising media, research and analytics organisations.

(tags: iab-europe awards spam sean-kelly ireland meps politics eu data-protection privacy ec)

The CAP FAQ by henryr

No subject appears to be more controversial to distributed systems engineers than the oft-quoted, oft-misunderstood CAP theorem. The purpose of this FAQ is to explain what is known about CAP, so as to help those new to the theorem get up to speed quickly, and to settle some common misconceptions or points of disagreement.

(tags: database distributed nosql cap consistency cap-theorem faqs)

seeing into the UV spectrum after Cataract Surgery with Crystalens

I’ve been very happy so far with the Crystalens implant for Cataract Surgery [...] one unexpected/interesting aspect is I see a violet glow that others do not – perhaps I’m more sensitive to the low end of the visible light spectrum.

(via Tony Finch)

(tags: via:fanf science perception augmentation uv light sight cool cataracts surgery lens eyes)

Instagram: Making the Switch to Cassandra from Redis, a 75% ‘Insta’ Savings

shifting data out of RAM and onto SSDs — unsurprisingly, big savings.

a 12 node cluster of EC2 hi1.4xlarge instances; we store around 1.2TB of data across this cluster. At peak, we’re doing around 20,000 writes per second to that specific cluster and around 15,000 reads per second. We’ve been really impressed with how well Cassandra has been able to drop into that role.

(tags: ram ssd cassandra databases nosql redis instagram storage ec2)

Council of the European Union Releases Draft Compromise Text on the Proposed EU Data Protection Regulation

Oh god. this sounds like an impending privacy and anti-spam disaster. “business-focussed”:

Overall, the [Irish EC Presidency’s] draft compromise text can be seen as a more business-focused, pragmatic approach. For example, the Presidency has drafted an additional recital (Recital 3a), clarifying the right to data protection as a qualified right, highlighting the principle of proportionality and importance of other competing fundamental rights, including the freedom to conduct a business.

and some pretty serious relaxation of how consent for use of personal data is measured:

The criterion for valid consent is amended from “explicit” to “unambiguous,” except in the case of processing special categories of data (i.e., sensitive personal data) (Recital 25 and Article 9(2)). This reverts to the current position under the Data Protection Directive and is a concession to the practical difficulty of obtaining explicit consent in all cases. The criteria for valid consent are further relaxed by the ability to obtain consent in writing, orally or in an electronic manner, and where technically feasible and effective, valid consent can be given using browser settings and other technical solutions. Further, the requirement that the controller bear the burden of proof that valid consent was obtained is limited to a requirement that the controller be able to “demonstrate” that consent was obtained (Recital 32 and Article 7(1)). The need for “informed” consent is also relaxed from the requirement to provide the full information requirements laid out in Article 14 to the minimal requirements that the data subject “at least” be made aware of: (1) the identity of the data controller, and (2) the purpose(s) of the processing of their personal data (Recitals 33 and 48).

(tags: anti-spam privacy data-protection spam ireland eu ec regulation)

LobbyPlag

wow, great view of which MEPs are eviscerating the EU’s data protection regime:

Currently the EU is negotiating about new data privacy laws. This new EU Regulation will replace all existing national laws on data privacy. Here you can see a general overview which Members of the European Parliament (MEPs) are pushing for more or less data privacy. Choose a country, a political group or a MEP from the “Top 10” list to find out more.

(tags: europe eu privacy data-protection datap ec regulation meps)

EDRI’s comments on EU proposals to reform privacy law

Amendments 762, 764 and 765 in particular seem to move portions of the law from “confirmed opt-in required” to “opt-out is ok” — which sounds like a risk where spam and unsolicited actions on a person’s data are concerned

(tags: law privacy anti-spam eu spam edri)

EC2Instances.info

‘Easy Amazon EC2 Instance Comparison’. a nice UI on the various EC2 instance types on offer with their key attributes. Misses out availability of EBS-optimized instances though

(tags: amazon ec2 aws comparison pricing)

HyperLevelDB: A High-Performance LevelDB Fork

‘HyperLevelDB improves on LevelDB in two key ways: Improved parallelism: HyperLevelDB uses more fine-grained locking internally to provide higher throughput for multiple writer threads. Improved compaction: HyperLevelDB uses a different method of compaction that achieves higher throughput for write-heavy workloads, even as the database grows.’

(tags: leveldb storage key-value-stores persistence unix libraries open-source)

EU Council deals killer blow to privacy reforms

‘In an extraordinary result for corporate lobbying, direct marketing would by default be considered a legitimate data process and would therefore – by default – be lawful.’

(tags: eu politics data-protection privacy anti-spam spam eu-council direct-marketing)

Care and Feeding of Large Scale Graphite Installations [slides]

good docs for large-scale graphite use: ‘Tip and tricks of using and scaling graphite. First presented at DevOpsDays Austin Texas 2013-05-01′

(tags: graphite devops ops metrics dashboards sysadmin)

Low-latency stock trading “jumps the gun” due to default NTP configuration settings

On June 3, 2013, trading in SPY exploded at 09:59:59.985, which is 15 milliseconds before the ISM’s Manufacturing number released at 10:00:00. Activity in the eMini (traded in Chicago), exploded at 09:59:59.992, which is 8 milliseconds before the news release, but 7 milliseconds after SPY. Note how SPY and the eMini traded within a millisecond for the Consumer Confidence release last week, but the eMini lagged SPY by about 7 milliseconds for the ISM Manufacturing release. The simultaneous trading on Consumer Confidence is because that number is released at the same time in both NYC and Chicago. The ISM Manufacturing number is probably released on a low latency feed in NYC, and then takes 5-7 milliseconds, due to the speed of light, to reach Chicago. Either the clock used to release the ISM number was 15 milliseconds fast, or someone (correctly) jumped the gun. Update: [...] The clock used to release the ISM was indeed, 15 milliseconds fast. This could be from using the default setting of many NTP clients, which allows the clock to drift up to about 16 milliseconds before adjusting time.

(tags: ntp time synchronization spy trading stocks low-latency clocks internet)

the infamous 2008 S3 single-bit-corruption outage

Neat, I didn’t realise this was publicly visible. A single corrupted bit infected the S3 gossip network, taking down the whole S3 service in (iirc) one region:

We’ve now determined that message corruption was the cause of the server-to-server communication problems. More specifically, we found that there were a handful of messages on Sunday morning that had a single bit corrupted such that the message was still intelligible, but the system state information was incorrect. We use MD5 checksums throughout the system, for example, to prevent, detect, and recover from corruption that can occur during receipt, storage, and retrieval of customers’ objects. However, we didn’t have the same protection in place to detect whether [gossip state] had been corrupted. As a result, when the corruption occurred, we didn’t detect it and it spread throughout the system causing the symptoms described above. We hadn’t encountered server-to-server communication issues of this scale before and, as a result, it took some time during the event to diagnose and recover from it. During our post-mortem analysis we’ve spent quite a bit of time evaluating what happened, how quickly we were able to respond and recover, and what we could do to prevent other unusual circumstances like this from having system-wide impacts. Here are the actions that we’re taking: (a) we’ve deployed several changes to Amazon S3 that significantly reduce the amount of time required to completely restore system-wide state and restart customer request processing; (b) we’ve deployed a change to how Amazon S3 gossips about failed servers that reduces the amount of gossip and helps prevent the behavior we experienced on Sunday; (c) we’ve added additional monitoring and alarming of gossip rates and failures; and, (d) we’re adding checksums to proactively detect corruption of system state messages so we can log any such messages and then reject them.

This is why you checksum all the things ;)

(tags: s3 aws post-mortems network outages failures corruption grey-failures amazon gossip)