Hacker News 50

APNewsBreak: New gender options for Facebook users - US News

Thu, 13 Feb 2014 10:12:04 -0800

Comments: "APNewsBreak: New gender options for Facebook users - US News"

URL: http://www.usnews.com/news/business/articles/2014/02/13/apnewsbreak-new-gender-options-for-facebook-users

By MARTHA MENDOZA, AP National Writer

MENLO PARK, Calif. (AP) — You don't have to be just male or female on Facebook anymore. The social media giant is adding a customizable option with about 50 different terms people can use to identify their gender as well as three preferred pronoun choices: him, her or them.

Facebook said the changes, shared with The Associated Press before the launch on Thursday, initially cover the company's 159 million monthly users in the U.S. and are aimed at giving people more choices in how they describe themselves, such as androgynous, bi-gender, intersex, gender fluid or transsexual.

[READ: How Twitter's User Growth Compares to Facebook's]

"There's going to be a lot of people for whom this is going to mean nothing, but for the few it does impact, it means the world," said Facebook software engineer Brielle Harrison, who worked on the project and is herself undergoing gender transformation, from male to female. On Thursday, while watchdogging the software for any problems, she said she was also changing her Facebook identity from Female to TransWoman.

"All too often transgender people like myself and other gender nonconforming people are given this binary option, do you want to be male or female? What is your gender? And it's kind of disheartening because none of those let us tell others who we really are," she said. "This really changes that, and for the first time I get to go to the site and specify to all the people I know what my gender is."

Facebook, which has 1.23 billion active monthly users around the world, also allows them to keep their gender identity private and will continue to do so.

The Williams Institute, a think tank based at the University of California, Los Angeles, estimate that there are at least 700,000 individuals in the U.S. who identify as transgender, an umbrella term that includes people who live as a gender different from the one assigned to them at birth.

The move by Facebook represents a basic and a yet significant form of recognition of the nation's growing transgender rights movement, which has been spurred by veteran activists and young people who identify as transgender at younger and younger ages. The Human Rights Campaign last year found that 10 percent of the 10,000 lesbian, gay, bisexual transgender youths it surveyed used "other" or wrote in their own gender terms.

"Over the past few years, a person's Facebook profile truly has become their online identity, and now Facebook has taken a milestone step to allow countless people to more honestly and accurately represent themselves," HRC President Chad Griffin said. "Facebook's action is one that I hope others heed in supporting individuals' multifaceted identities."

[ALSO: Facebook Celebrates Its 10th Birthday]

The change to the gender selection option is seen as a major step toward acceptance for people who don't self-identify as male or female, but the high-profile development seemed senseless to those who believe in two genders, no more.

"Of course Facebook is entitled to manage its wildly popular site as it sees fit, but here is the bottom line: It's impossible to deny the biological reality that humanity is divided into two halves - male and female," said Jeff Johnston, an issues analyst for Focus on the Family, an influential national religious organization based in Denver. "Those petitioning for the change insist that there are an infinite number of genders, but just saying it doesn't make it so. That said, we have a great deal of compassion for those who reject their biological sex and believe they are the opposite sex."

Masen Davis, executive director of the San Francisco-based Transgender Law Center, said it may be hard for some people to understand the importance of having the ability to select from multiple genders online. But he said many transgender people will be thrilled with the change.

"We applaud Facebook for making it possible for people to be their authentic selves online," he said.

In the past decade, the transgender movement has become much more organized and outspoken, demanding the kind of civil rights and respect already sought by gay activists. During this time, the transgender umbrella has been growing well beyond transsexuals to encompass a wide variety of gender identities.

The move by Facebook came after years of lobbying from users, some who started Facebook pages to petition for the change. Google+ offers male, female and "other" as choices, but transgender advocates said Facebook's many specific options puts the platform well ahead of any other online community. About 1 percent of Google+ users identify as other.

The idea of expanding gender choices percolated at Facebook for about a year and started to come to fruition during an in-house brainstorming four months ago, project manager Lexi Ross said.

Transgender activist Nori Herras-Castaneda, a spokeswoman for the Billy DeFrank LGBT Community Center in San Jose, said her community has been waiting for this to happen for a long time.

"We always talk about how gender is a spectrum," she said. "I can see a lot of people being extremely happy about this."

[MORE: Facebook Adds Online Bullying Prevention Hub]

At this point, Facebook targets advertising according to male or female genders. For those who change to something neutral, ads will be targeted based on the pronoun they select for themselves. Unlike getting engaged or married, changing gender is not registered as a "life event" on the site and won't post on timelines. Therefore, Facebook said advertisers cannot target ads to those who declare themselves transgender or recently changed their gender.

Users also can select "neither" or "other" and separately indicate whether they want to be referred to as he, she or they.

Facebook came up with its range of terms after consulting with leading gay and transgender activists, and the company plans to continue working with them. Facebook started the options in the U.S. and plans to take it global after working with activists abroad to come up with terms appropriate in other countries.

Herras-Castaneda said she did expect some anger.

"Any time the transgender community makes advances, there is backlash, and this is a very big advance, so yes, we'll face some problems, no doubt," she said.

At Facebook, staffers said the expanded options were never questioned, from CEO Mark Zuckerberg on down.

"Really, there was no debate within Facebook about the social implications at all," said Alex Schultz, director of growth. "It was simple: Not allowing people to express something so fundamental is not really cool so we did something. Hopefully a more open and connected world will, by extension, make this a more understanding and tolerant world."

---

Follow Martha Mendoza at https://twitter.com/mendozamartha

---

Associated Press Writer Lisa Leff contributed to this story.

Can’t We All be Reasonable and Speak English? « Blog – Stack Exchange

Thu, 13 Feb 2014 10:19:04 -0800

Comments: "Can’t We All be Reasonable and Speak English? « Blog – Stack Exchange"

URL: http://blog.stackoverflow.com/2014/02/cant-we-all-be-reasonable-and-speak-english/

Two weeks ago, we announced the public launch of Stack Overflow in Portuguese, our first-ever non-English Stack Overflow community. Which raises one very obvious question:

Have we lost our minds?

Wasn’t the whole point of Stack Overflow to aggregate as much developer knowledge as possible in one place? To get all the potential solutions together, and provide one canonical set of answers?

We are aware that, “Let’s all try speaking speaking different languages!” hasn’t always worked out for the best.

Yup. When we set out to “collectively increase the sum total of good programming knowledge in the world,” a big part of the plan was de-fragmenting information previously spread across myriad books, sites, and your brains. It’s why we mark things as duplicates – we want all the precious gems of knowledge stored in the same cave of wonders.

So know this: we are at least as worried about fragmentation as you are. And we have a plan:

Eventually, all of you are going to have to learn Portuguese.

Okay, not really. But, given that one of our core goals was knowledge aggregation, it does seem just a little bit crazypants to start launching sites in new languages, assuming that one very important fact is true:

Assumption: All of the serious developers in the world are highly proficient in English.

Which… actually sounds plausible. But it’s wrong.

Not every developer in the world speaks English. Just reading the comments from our announcement, you’ll see multiple readers sharing that they or their colleagues (and one dad) couldn’t participate on SO due to language constraints. But data beat anecdotes. We don’t have recent numbers for Brazil and Portugal, but we do for China, and they illustrate the same point: So, if the data tell us that we’re getting roughly 80% less activity from Asia than we should in the absence of language constraints, why does it feel so obvious that all serious programmers speak English? This may help: Quick – name any famous developer who doesn’t write well in English. I couldn’t. I can name over a dozen famous English-speaking coders. But even if you frequent all the hacker sites and conferences, how many devs have you met who aren’t solid in English? Roughly none, right? There’s just one problem. Try this: Without Googling, name any famous developer from Japan. Or China. Or Russia. Again, I couldn’t. Well, I came up with Shigeru Miyamoto. But he’s apparently a designer. I couldn’t name even one. Not like I can name Carmack or Stallman, or Hopper, or even “DHH.” (Does DHH have an actual name? I personally imagine him as a very handsome, talented, fast-driving set of initials. But I digress.) Is it plausible that there aren’t any devs good enough to be famous from those countries? Nope. Here’s what’s happening: It’s easy to assume that there aren’t any devs who can’t speak English because I never see any. But I never see any because I’m hanging around places where devs go to talk to each other in English. The startling truth is this: On the internet, If you don’t speak English, you’re completely invisible to me. I also assumed that since developers have to learn English-like syntax, they must speak English. Which is a bit like assuming that because I can order Uni, Hamachi, and Aji by their Japanese names, I could probably toss back some sake with Morimoto and discuss knife techniques in Japanese. Even when programming languages use words like “if” or “function,” they’re just terms to memorize, and don’t always even mean the same thing in English that they do in programming. It’s almost impossible to feel like part of a community if you’re not highly proficient in the language. Even non-native speakers who are fluent enough to read posts in their second or third languages often aren’t comfortable enough to write in them.I imagine myself at a professional meetup where everyone is speaking French (which I studied through college). How many jokes would I tell? How many would I even understand? Sure, I can function, and understand all the words, but I don’t feel like I belong to the group. Don’t get me wrong – some of our best users aren’t native English speakers, but they’re in that rare group who have achieved a far higher mastery of a language than their peers. When I hear, “Well, I didn’t need a site like this – English is my third language, and I’m in the top 1% on Stack Overflow!” I think: “Yes, that makes sense. You are insanely good at two difficult, language-based things. Most people will find both of them to be a lot more challenging than you did.” The truth is, by requiring fluency in English, we’re shutting out of a lot of developers who may know enough English to read it but not enough to feel comfortable participating. Requiring that all aspiring devs “just go learn English” first isn’t who we want to be. No child should be denied their chance to revolutionize tomorrow’s input technologies. Even if I believed that every programmer must eventually master English, it still wouldn’t make any sense to make them do it first. I believe that everyone – everyone – who can really fall in love with programming should get a chance to. So pre-filtering for the ones willing to learn a foreign freaking language before they first sit down with a code editor to see if it lights some spark in them just feels wrong.Think of the children. The children!! Okay, last quiz, just for the native English speakers: How old were you when you first realized you could type things on a keyboard and control machines? Great. Now, at that age, were you proficient enough in another language to have learned to code without any English? When I tell someone I work at Stack Exchange, my absolute favorite response is: “I basically learned to code from posts I found on Stack Overflow” We want that for every young programmer. Not just the ones lucky enough to be born somewhere that English gets taught in grammar school.

Okay, that all makes some sense. But why Portuguese?

To be clear, we still don’t think there needs to be a Stack Overflow in every language. We do want as much centralization as possible, and we know that devs who have mastered English will mostly keep going to the English site, since it has the most critical mass. Just like we want them to. So, you won’t need to learn new languages to find good answers – we expect almost every question asked on the Portuguese site to also be asked (and answered) on the English site.

We’re really only considering launching sites in languages that:

Have large, strong communities of high-talent developers, where
A meaningful percent of them aren’t comfortable enough to participate in an English-only community

That probably limits the list of potential candidates to Mandarin, Japanese, Portuguese, Russian, Turkish, and Spanish. From there, Portuguese was a no-brainer. The developer community in Brazil is awesome, and growing fast. And we wanted to start with a language with a similar alphabet, to minimize the localization work.

And it’s worth a shot. We’ve learned that it’s easier to just watch the future than to try to predict it. So we’re big on just trying stuff out (assuming it can’t break our other stuff). And we’re huge on getting stuff crazy-wrong, refusing to admit it, and instead doubling down on our wrong-minded idea, while nodding crazily er… admitting we made a mistake, and reversing course. So, given the number of user requests, we figured, “why not give it a it a try?” We’re committed to supporting one or two languages and seeing how they develop before we push any further.

And so far, it’s an incredible success. Despite an audience limited to portuguese-speaking devs, the site’s activity in its first week was higher than all but 4 out of 120 sites we’ve launched to date, including the original trilogy.

More importantly, people who couldn’t ask questions are asking them, and getting great answers. When in doubt, we want to err on the side of helping more people. If just one little girl in Brazil sticks with programming because an answer on this site helped her finish her first project, well… that’s not good enough! I want to help thousands of them. And the boys, too.

Still, it’s a good start.

"Strong" stack protection for GCC [LWN.net]

Thu, 13 Feb 2014 11:10:06 -0800

Comments: ""Strong" stack protection for GCC [LWN.net]"

URL: http://lwn.net/Articles/584225/

By Jake Edge
February 5, 2014

Stack buffer overflows are a longstanding problem for C programs that leads to all manner of ills, many of which are security vulnerabilities. The biggest problems have typically been with string buffers on the stack coupled with bad or missing length tests. A programmer who mistakenly leaves open the possibility of overrunning a buffer on a function's stack may be allowing attackers to overwrite the return pointer pushed onto the stack earlier. Since the attackers may be able to control what gets written, they can control where the function returns—with potentially dire results. GCC, like many compilers, offers features to help detect buffer overflows; the upcoming 4.9 release offers a new stack-protection mode with a different tradeoff between security and performance impact.

GCC has supported stack protection for some time. It currently supports two different types of stack protection. Recently, Google engineers have come up with another style that tries to chart a middle course between the two existing options. It has made its way into GCC 4.9 (expected later this year) and the upcoming 3.14 kernel has support for building with that option.

The basic idea behind stack protection is to push a "canary" (a randomly chosen integer) on the stack just after the function return pointer has been pushed. The canary value is then checked before the function returns; if it has changed, the program will abort. Generally, stack buffer overflow (aka "stack smashing") attacks will have to change the value of the canary as they write beyond the end of the buffer before they can get to the return pointer. Since the value of the canary is unknown to the attacker, it cannot be replaced by the attack. Thus, the stack protection allows the program to abort when that happens rather than return to wherever the attacker wanted it to go.

There is a downside to using canaries. The value must be generated and checked, which takes some time, but more importantly there must be code added to handle the canary for each function that is protected that way. That extra code results in some level of performance degradation, perhaps mostly due to a larger cache footprint. For this reason, it can make sense to restrict stack protection to a subset of all the functions in a program.

So the question has always been: "Which functions should be protected?" Putting stack protection into every function is both overkill and may hurt performance, so one of the GCC options chooses a subset of functions to protect. The existing -fstack-protector-all option will protect all functions, while the -fstack-protector option chooses any function that declares a character array of eight bytes or more in length on its stack. Some distributions have lowered that threshold (e.g. to four) in their builds by using the --param=ssp-buffer-size=N option.

That "character array" test catches the most "at risk" functions, but it leaves a number of other functions behind. As Kees Cook pointed out in a recent blog post, the Google Chrome OS team had been using -fstack-protector-all since the team is "paranoid", but a new -fstack-protector-strong option has been developed to broaden the scope of the stack protection without extending it to every function in the program.

In addition to the protections offered by -fstack-protector, the new option will guard any function that declares any type or length of local array, even those in structs or unions. It will also protect functions that use a local variable's address in a function argument or on the right-hand side of an assignment. In addition, any function that uses local register variables will be protected. According to Cook, Chrome OS has been using -fstack-protector-strong (instead of protecting all functions) for ten months or so.

During the 3.14 merge window, Linus Torvalds pulled Cook's patches to add the ability to build the kernel using the strong stack protection. In Ingo Molnar's pull request (and Cook's post), the results of using strong protection on the kernel were presented. The kernel with -fstack-protector turned on is 0.33% larger and covers 2.81% of the functions in the kernel. For -fstack-protector-strong, those numbers are an increase of 2.4% in code size over an unprotected kernel, but 20.5% of the functions are covered.

The CONFIG_CC_STACKPROTECTOR_STRONG kernel configuration option adds the strong protection, while the CONFIG_CC_STACKPROTECTOR option for the "regular" protection has been renamed to reflect that: CONFIG_CC_STACKPROTECTOR_REGULAR. The default CONFIG_CC_STACKPROTECTOR_NONE does just what its name would imply.

While stack protection certainly isn't a panacea for security woes, it will catch a significant portion of real-world attacks. Having an option that strikes a balance between the ultra-paranoid "all" and the regular variant (not to mention the wide-open "none" option) is likely to catch more bugs—and attack vectors. We will likely see some of the more security-conscious distributions building their user-space programs and kernels with the "strong" option moving forward.

(Log in to post comments)

Ron Paul Launches Snowden Clemency Petition - Hit & Run : Reason.com

Thu, 13 Feb 2014 11:29:18 -0800

Comments: "Ron Paul Launches Snowden Clemency Petition - Hit & Run : Reason.com"

URL: http://reason.com/blog/2014/02/13/ron-paul-launches-snowden-clemency-peti

Credit: Gage Skidmore/wikimedia

Today, it was reported that former Rep. Ron Paul (R-Texas) launched a petition calling for NSA whistle-blower Edward Snowden to be granted clemency.

On the page on the Ron Paul Channel’s website where visitors can sign the petition, the former congressman says,

Edward Snowden sacrificed his livelihood, citizenship, and freedom by exposing the disturbing scope of the NSA’s worldwide spying program. Thanks to one man’s courageous actions, Americans know about the truly egregious ways their government is spying on them.

The news of Paul’s petition comes on the same day it was reported that the European Parliament had voted against calling for Snowden to be granted asylum.

The New York Times and The Guardian both urged U.S. officials to grant Snowden clemency last month.

According to the Ron Paul Channel, Snowden’s temporary asylum in Russia will expire at the end of July.

Yesterday, Intelligence Squared hosted a debate on the motion “Snowden was Justified.” Speaking for the motion were legal adviser to Edward Snowden and ACLU attorney Ben Wizner and Pentagon Papers whistle-blower Daniel Ellsberg. Speaking against the motion were former CIA Director James Woolsey and former federal prosecutor and contributing editor to National Review Andrew McCarthy.

Wizner said Snowden was justified “because he provided to journalists and through them to us information that we had a right to know and that we had a need to know. The government had not just concealed this information, it had lied to us about it.”

Woolsey claimed that Snowden had released information to “Hezbollah, Al Qaeda, Hamas, Pyongyang, Tehran, and so on.” Thankfully, Ellsberg pointed out that Snowden released information to journalists who have since reported on the documents.

Before the debate began 29 percent were for the motion, 29 percent were against the motion, and 43 percent were undecided. After the debate, 54 percent were for the motion, 35 percent were against the motion, and 11 percent were undecided.

Watch the debate below:

Quant Strategies Implemented by the Quantopian Community | Quantopian Blog

Thu, 13 Feb 2014 11:33:18 -0800

Comments: " Quant Strategies Implemented by the Quantopian Community | Quantopian Blog"

URL: http://blog.quantopian.com/5-basic-quant-strategies-implemented-by-the-quantopian-community/

Quant Strategies Implemented by the Quantopian Community

Posted by: Jess Stauth

Last week I gave a quant finance meet-up talk at the Hacker Dojo in Mountain View, CA. The format was inspired by some analysis I did on the types of algorithms shared and cloned in the Quantopian community – initially I wanted to ask: What are the most popular strategies coded up on Quantopian? To answer this question I ranked all public forum posts three ways, first on number of replies, second on number of views, and third on number of times cloned. I averaged these scores and re-ranked the list to come up with the top 25 ‘Most popular posts of all time’. (NB: I did not do any correction for the date of the original post, so the amount of time the thread has been alive has not been normalized.)

Starting from this list, I worked backwards and used examples from the Quantopian community to introduce 5 basic quant strategy types: Mean Reversion, Momentum, Value, Sentiment and Seasonality. While this list is not technically ‘mutually exclusive and collectively exhaustive’, it covers a large fraction of intraday to lower frequency quant strategies and provides a good overview of the way equity focused quants think about predicting market prices. I went back to my Top 25 list and categorized each algo into one of these five buckets and then created this pie chart based on the aggregated number of views for each strategy type.

There are a number of interesting conclusions to be drawn from this initial overview of community activity. Perhaps the most obvious and predictable of these is that price based strategies are currently in the lead by a large margin – due, I expect, to the easy access to minute-level equity pricing and the accessibility of the logic for momentum and mean-reversion. Indeed there were no value-based strategies that made their way into the Top 25 – which in my view represents a key opportunity space right now.

More subtle and, from my admittedly biased point of view, more compelling is the diversity and quality of content and collaboration in the public sphere. Having joined the Quantopian team from a large corporate setting working with a small group of institutional clients, seeing that the Top 25 algos have been cloned over 13,000 times, an average of over 500 clones per strategy is… well it’s pretty damn cool.

Below you can find the slide deck from my presentation:

This entry was posted on Monday, February 10th, 2014 at 6:05 pm and is filed under Quantopian. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Good Samaritan Backfire — Medium

Thu, 13 Feb 2014 12:29:44 -0800

Comments: "Good Samaritan Backfire — Medium"

URL: https://medium.com/p/9f53ef6a1c10/

Solitary Confinement — Safety Cell

I was led into a corner.

“First we have to get you ready,” one of the deputies said. His arm undid the button of my pants, which at first I thought was a cruel joke, and then he yanked them down to my ankles.

They pushed me forward against the wall. I stumbled in my handcuffs and pant shackles.

“Step out of your pants,” they ordered. And as soon as I did: “Step out of your socks!”

Naked from the waist down, someone said, “Take off your shirt.” It was topologically impossible, given the cuffs. One of the deputies said, “I’ll do it.” I was uncuffed, my shirt was stripped with force, getting caught on my neck, tugging my head backwards, then up, then off.

The night shift deputies were cruel. They responded to questions in the tone of schoolyard bullies—tauntingly. They giggled as they slammed the door behind me. “You’ll see the doctor alright.”

On the floor lay a straight jacket made from the material used to pad furniture when it is being moved, and a second piece of the same fabric that I later used to cover the dirty floor in an attempt to sleep.

There were no knobs or protrusions in the room, just soft corners. The toilet was a hole in the ground, no toilet paper. The hole dropped down a few feet where it was intersected by a grate of prison bars. The flushing happened automatically, periodically, though I never felt the urge. Even one’s feces left prison upon evacuation, presumably to leave the subject without anything to play with.

I say this, because while the room was dirty, it was not as dirty as the next two cells I experienced the following day, which were smeared with feces and peanut butter. Approximately every 6 hours, a pushcart made its way around the prison with regulation peanut butter sandwiches. Only a fraction were consumed. Many were used for wall decoration or splattered against the ceiling.

I couldn’t bear to eat, so I took my rations home as a souvenir. Aside from the milk, they still seemed edible a month later. This is their strength.

Trapped in a rendition of One Flew Over the Cuckoo’s Nest

While the metal door was too thick for me to be heard if I did not scream, I could hear the muted screams of others across the jail. The din was anything but soothing.

When I asked for water, I was given enough (a couple Dixie cups’ worth) to barely keep my throat lubricated.

I was cold. The two pieces of fabric were not enough to spread on the filthy ground and also cover my naked body. I tried to sleep but it proved fruitless. Every 15 minutes, the metal peephole was creaked open, and I was expected to react, presumably to confirm that I was still alive. This was noted on a clipboard hanging beside the door.

Eventually, I found it most comfortable to stand by the cell door with the coarse fabric draped over my body. I looked out through a narrow slit of Plexiglas and tried to call attention from passers’ by. “Sir, Ma’am, could you please tell me… how long should I expect to be in here?”

A streak of being ignored was broken by a couple disheartening responses. “Usually we put people in there for 24 hours.”

Now I really felt like I was going crazy. Those weren’t the reassuring answers my inner optimist had hoped for. When I had told the arresting officers that I accepted my lot, this wasn’t the lot I was referring to. I didn’t expect a medal for fulfilling my civic duty, but I still felt like I had some fleeting right to something other than this. I banged on the metal door repeatedly until Deputy Terry showed up.

“Why am I in here?”

“You are crazy. You are a lunatic,” he pronounced.

“Do you know how I got here?”

“Doesn’t matter.”

“This place—being in here—will make me crazy,” I pleaded.

“Good. That’s what you are and where you belong.” He spiraled his index finger by his muscular temple.

I tried to respond as he started walking way.

“Sir, might you consider for a moment that I am having a sane response to the conditions I’m being subjected? I was arrested by the very police I called to the scene of a medical emergency less than a block from my house, while heading home for the night.”

He stared at me bewildered, and never came near again.

Part 2: Lessons learned tuning TCP and Nginx in EC2 « Chartbeat Engineering Blog

Thu, 13 Feb 2014 11:54:18 -0800

Comments: "Part 2: Lessons learned tuning TCP and Nginx in EC2 « Chartbeat Engineering Blog"

URL: http://engineering.chartbeat.com/2014/02/12/part-2-lessons-learned-tuning-tcp-and-nginx-in-ec2/

In part 1 of our post, one of the items we discussed was our issues with using DNS as a load balancing solution. To recap, at the end of our last post we were still setup with Dyn’s load balancing solution and our servers were receiving a disproportionate amount traffic to them. Server failures were still not as seamless as we wanted, due to the issues with DNS TTLs not always being obeyed and our response times were a lot higher than we wanted them to be, hovering around 200-250ms.

In part 2 of this post, I’ll cover the following

How we improved our issues with server failure and response time by using Amazon’s ELB service
The performance gains we saw from enabling HTTP keepalives.
Future steps for performance improvements

But first before I dive into the ELB, there’s one topic I left out of my last post that I wanted to mention.

TCP Congestion Window (cwnd)

In TCP congestion control, there exists a variable called the “congestion window” , commonly referred to as cwnd. The initial value of cwnd is often referred to as “initcwnd”. After the initial TCP handshake is done, we begin to send data, the cwnd determines how many bytes we can send before the client needs to respond with an ACK. Let’s look at a graphic of how different initcwnd values affect TCP latency from a paper Google released.

At Chartbeat, we’re currently running Ubuntu 10.04 LTS (I know, I know, we’re in the process of upgrading to 12.04 as this is being written), which ships with Kernel 2.6.32. Starting in Kernel 2.6.39, thanks to some research from Google, the default initcwnd was changed from 3 to 10. If you are serving up content greater than 4380 bytes (3 * 1460), you will benefit from increasing your initcwnd due to the ability to have more data in flight (BDP or bandwidth delay product) before having to reply with an ACK. The average response size from ping.chartbeat.net is way under that, at around 43 bytes, so this change had no benefit to us at the time when the servers were not behind the ELB. We’ll see why increasing the initcwnd helped us later in the post when we discuss HTTP keepalives.

ELB (Elastic Load Balancer)

The options for load balancing traffic on AWS are fairly limited. Your choices are

DNS load balancing service such as Dyn
Homegrown solution using HAProxy, nginx or <insert favorite load balancing software here>

Each of these solutions have their limitations and depending on your requirements, some may not be suitable at all for you. I won’t go into all of the pros and cons of each solution here since there are plenty of articles on the web discussing these already. I’ll just go over a few that directly affected our choice.

In choosing a homegrown solution, support for high availability and scalability is difficult. Currently with AWS, there’s no support for gratuitous ARP, which is traditionally used in handling of fail overs both in software and hardware load balancers. In order to work around this issue, you can utilize Elastic IPs and homegrown scripts to move the Elastic IP between instances when it detects a failure. In our experience we’ve seen lag times from 30 seconds to a few minutes when moving an Elastic IP. During this time, you would be down hard and not serving up any traffic. The above solution also only works when all your traffic can be handled by one host and you can accept the small period of downtime during fail over.

But how would you handle a situation where your traffic was too high for one host? You could launch multiple instances of your home grown solution but you would then need to handle balancing the traffic between these instances. We already discussed in part 1 the issue we had with using DNS to handle the balancing of traffic. The only other solution would be to actually use an ELB in front of these instances. If we went with this solution, it meant adding another layer of latency to the request. Did we really need to do something like this?

The reason why most people end up going with a solution like HAProxy is because they have more advanced load balancing requirements. ELB only supports round robin request balancing and sticky sessions. Some folks require the ability to do request routing based on URI, weight based routing or any of the other various algorithms that HAProxy supports. Our requirements for a load balancing solution were fairly straightforward:

Evenly distribute traffic (better than our current DNS solution)
Highly available
Handle our current traffic peak(200k req/sec) and scale beyond that
End-to-End SSL support

ELB best met all these requirements for us. A homegrown solution would have been overkill for our needs. We didn’t need any of the advanced load balancing features, SSL is currently only supported in HAProxy’s development branch (1.5.x) and requires using stunnel or nginx for support in the stable branch (1.4.x) and we didn’t need to add any additional layers that would increase our latency even further.

Moving to ELB

The move to using an ELB was fairly straight forward. We contacted Amazon support and our technical account manager to coordinate pre-warming the ELB. According to the ELB best practices guide, ELBs will scale gradually as your traffic grows (should handle 50% of traffic increase every 5 minutes), but if we suddenly switched 100% of our traffic to the ELB, it would not be able to scale quickly enough and start throwing errors. We weren’t planning on doing a cutover in that fashion anyway, but to be safe we wanted to ensure the ELB was pre-warmed ahead of time even as we slowly moved over traffic. We added all the servers into the ELB and then did a slow roll out utilizing Dyn’s traffic director solution, which allowed us to weight DNS records. We were able to raise the weight of the ELB record and slowly remove the individual server’s IPs from ping.chartbeat.net to control the amount of traffic flowing through the ELB.

Performance gains

We saw large, immediate improvements in our performance with the cutover to the ELB. We saw less TCP timeouts and a huge decrease in our average response time.

We went from roughly 200 ms average response times, to 3 ms response times. That’s a 98.5% decrease in response time!

Our traffic was now more evenly distributed than our previous DNS based solution. We were able to further distribute our traffic shortly after, when Amazon released “Cross-Zone load balancing”

Enabling cross-zone load balancing got our request count distribution extremely well balanced, the max difference in requests between hosts sits currently around 13k requests over a minute.

KeepAlives

With our servers now behind the ELB we had one last performance tweak we wanted to enable, HTTP keepalives between our servers and the ELB. Keepalives work by allowing multiple requests over a single connection. In cases where users are loading many objects off your site, this can greatly reduce latency by removing the overhead of having to re-establish a connection for each object you are loading off the site. CPU savings are seen on the server side since less time will be spent opening and closing connections. All this sounds pretty great, so why didn’t we have it enabled before hand?

There are a few cases where you may not want keepalives enabled on your web server. If you’re only serving up one object from your domain, it doesn’t make much sense to keep a connection hanging around for more requests. Each connection uses up a small amount of RAM. If your web servers don’t have a large amount of RAM and you have a lot of traffic, enabling keepalives could get you in a situation where you will consume all RAM on the server, especially with a high default timeout for the keepalive connection. For Chartbeat, our data comes from clients every 15 seconds, holding a connection open just to get a small amount of data every 15 seconds would be a waste of resources for us. Fortunately we were able to offload that to the ELB which enables keepalive connections by default for any HTTP 1.1 client.

With our servers no longer being directly exposed to the clients, we could re-visit enabling keepalives. We are doing a high amount of requests between the ELB and our servers , with the connections coming from a limited set of servers on Amazon’s end. We want the ELBs to be able to proxy as much information as possible to us over one connection and keep that connection open for as long as possible. This is where having a larger initcwnd comes into play. Having a larger initcwnd lowers our latency and gets our bandwidth up to full speed between the servers and the ELB. We expected to see a drop off in the amount of traffic going through the servers as well as some CPU savings. To ensure there were no issues, we did a “canary” test with one server enabled with keepalive and put it into production. The results were not at all what we expected. Traffic to the server became extremely spiky and average response time increased a bit when keepalives were enabled on the canary server. After talking to Amazon about the issue, we learned that the ELB was favoring the host with keepalive enabled. More traffic was being sent to that host causing its latency to increase. When the latency increased, the ELB would then send less traffic through the host and the cycle would start over again. Once we confirmed what the issue was, we proceeded with the keepalive rollout and the traffic went back to being evenly distributed. The amount of sockets we had sitting in TIME_WAIT went from around 200k to 15k after enabling keepalives and CPU utilization dropped by about 20%.

Keepalives and Timeouts

There are a few important things to be aware of when configuring keepalives with your ELB with regards to timeouts. Unfortunately there’s a lack of official documentation on ELB keepalive configuration and behavior, so the information below could only be found through various posts on the official AWS forums.

The default keepalive idle connection timeout is 60 seconds
The keepalive idle connection timeout can be changed to values as low as 1 second and as high as 17 minutes with a support ticket
The keepalive timeout value on your backend server must be higher than that of your ELB connection timeout. If it is lower, the ELB will re-use the idle connection when your server has already dropped the connection, resulting in the client being served up a blank response. The default nginx keepalive_timeout value is safe at 75 seconds with the default ELB timeout of 60 seconds.

Downsides

While the ELB has worked out great for us and we’ve seen huge performance improvements from switching to using one in front of our servers, there are a few issues we’d love to see addressed in future roll-outs of the ELB:

Lack of bandwidth graphs in CloudWatch. I’m surprised the ELB has been around for this long without this CloudWatch metric. You get charged per GB processed through the ELB, yet there’s no way to see from Amazon’s view, how much bandwidth is going through your ELB. This could also help identify DoS attacks that don’t involve making actual requests to the ELB. No Ability to pre-warm an ELB without going through support. Right now it’s a process of having to contact Amazon support to get an ELB pre-warmed, and answering a bunch of questions related to your traffic. Even if this process was moved to a web form like how requests for service limit increases are done, it would be better than the current method. No ability to clone an ELB. Why would you want that? If you have an ELB that is handling a large amount of traffic and you are experiencing issues with it, you cannot easily replace the faulty ELB in a hurry due to the need for new ELBs to scale up slowly. It would be extremely useful to clone an existing one, capturing it’s fully warmed configuration and then be able to flip traffic over to it. Right now if there’s an issue, AWS support needs to get involved, and unless you are paying for higher end support, you may not get a fast enough response from support. No access to the raw logs. A feature to send the ELB logs to an S3 bucket would be very valuable. This would open up a bunch of doors with the ability to setup AWS Data Pipeline to fire off an EMR job or move data into Redshift. Currently all that must be done on the servers behind the ELB. No official documentation on keepalive configuration or behavior. Ability to change the default keepalive timeout value is not exposed through the API and requires a support ticket.

Conclusions

We learned an important lesson by not monitoring some key metrics on our servers that were having an affect on our performance and reliability. With increasing traffic it’s important to re-evaluate your settings periodically to see if they still make sense for the level of traffic you are receiving. The default TCP sysctl settings will work just fine for a majority of workloads but when you begin to push your server resources to there limits, you can see big performance increases by making some adjustments to variables in sysctl. Through TCP tuning and utilizing AWS Elastic Load Balancer we were able to

Decrease our response time by 98.5%
Decrease our server footprint by 20% on our front end servers
Have failed servers removed from service within seconds
Eliminate dropped packets due to listen queue socket overflows

Next Steps

Since the writing of this article, we’ve done some testing with Amazon’s new C3 instance types and are planning to move from the m1.large instance type to the c3.large. The c3.large is almost 50% cheaper and gives us more compute units which in turn yields slightly better response times.

Our traffic is very cyclical which lends itself perfectly to take advantage of Amazon’s auto scaling feature. Take a look at a graph from a weeks worth of traffic.

In the middle of the night (EDT), we see half of what our peak traffic was earlier in the day and on weekends we see about 1/3 less traffic than a weekday.

In the next coming months we’ll be looking to implement auto scaling to achieve additional cost savings and better handle large, unexpected spikes of traffic.

Additional resources:

Special thanks to the following folks for feedback and guidance on this post

Silk Road 2 Hacked, All Bitcoins Stolen – $2.7 Miliion | Deep Dot Web

Thu, 13 Feb 2014 12:29:44 -0800

Comments: "Silk Road 2 Hacked, All Bitcoins Stolen – $2.7 Miliion | Deep Dot Web"

URL: http://www.deepdotweb.com/2014/02/13/silk-road-2-hacked-bitcoins-stolen-unknown-amount/

Update 2: As the time passes there are more and more suspicions that this was in fact a SCAM by the Silk Road staff – and not a hack, we will post more details about it once, and if we get the full picture.

Update: The amount of BTC that was stolen was calculated by Nicholas Weaver @NCWeaver – Computer Security Researcher, to be around: 4474.266369160003BTC that are with the value of about $2.7 Million.

It was just announced in a post by Defcon the Silk Road administrator (this post will be updated as soon as we get more info) -
Yes, what seemed to be an imaginary situation until not long ago, just became true, the silk road2 – the site who counted to be the security fortress of the deep web just has been hacked with its bitcoin stolen. as he announced on the sites forums, we pasted his post here:

Link to the original thread on Silk Road 2 Forums: http://silkroad5v7dywlc.onion/index.php?topic=25091.msg491029#msg491029

=====Start Quote====

I am sweating as I write this.

Christmas brought grave news. I cannot adequately express how deeply honored I was by your unconditional support of my staff.

I do not expect the same reaction to today’s revelations. This movement is built on integrity, and I feel obligated to be forthright with you.

I held myself to a high standard as your leader, yet now I must utter words all too familiar to this scarred community:

We have been hacked.

Nobody is in danger, no information has been leaked, and server access was never obtained by the attacker.

Our initial investigations indicate that a vendor exploited a recently discovered vulnerability in the Bitcoin protocol known as “transaction malleability” to repeatedly withdraw coins from our system until it was completely empty.

Despite our hardening and pentesting procedures, this attack vector was outside of penetration testing scope due to being rooted in the Bitcoin protocol itself.

This attack hit us at the worst possible time. We were planning on re-launching the new auto-finalize and Dispute Center this past weekend, and our projections of order finalization volume indicated that we would need the community’s full balance in hot storage.

In retrospect this was incredibly foolish, and I take full responsibility for this decision.

I have failed you as a leader, and am completely devastated by today’s discoveries. I should have taken MtGox and Bitstamp’s lead and disabled withdrawals as soon as the malleability issue was reported. I was slow to respond and too skeptical of the possible issue at hand. It is a crushing blow. I cannot find the words to express how deeply I want this movement to be safe from the very threats I just watched materialize during my watch.

I’ve included transaction logs at the bottom of this message. Review the vendor’s dishonest actions and use whatever means you deem necessary to bring this person to justice. More details will emerge as we continue to investigate.

Given the right flavor of influence from our community, we can only hope that he will decide to return the coins with integrity as opposed to hiding like a coward.

It takes the integrity of all of us to push this movement forward. Whoever you are, you still have a chance to act in the interest of helping this community. Keep a percentage, return the rest. Don’t walk away with your fellow freedom fighters’ coins. DPR2 returned the cold storage. I didn’t run with the gold. But two people alone cannot move us forward. It takes an entire community committing to integrity – and though this crushing blow will not stop us, it sure is a testament to how greedy some bastards truly are.

Being a part of this movement might be the most defining thing you do with your entire life.

Don’t trade that for greed, comrades.

I will fight here by your side, even the greedy bastards amongst us.

This community has suffered great financial loss over and over again, and I am devastated that it has happened again under my watch.

Hindsight is already suggesting dozens of ways this could have been prevented, but we must march onward.

The only way to reverse a community’s greed is through generosity. Our true character is revealed during trying times.

If this financial hardship places you at risk of physical harm, contact me directly and I will do my best to help you with my remaining personal funds.

Now what.

Never again store your escrow bitcoins on a server.

Silk Road will never again be a centralized escrow storage.

This week has shown the collateral damage we can cause by being a huge target and failing in just one unforeseen area.

I am now fully convinced that no hosted escrow service is safe.

If I cannot trust myself to keep a hosted escrow solution safe, I cannot trust anyone.

Multi-signature transactions are the only way this community will be protected long-term.

I am aggressively tasking our devs on building out multi-sig support for commonly-used bitcoin clients. Expect a generous bounty if you have the skill to implement this.

Until then.

1. We will never again allow ourselves to be a single point of failure. We will never again host your Escrow wallets.

2. Vendor registration is closed while we regroup.

3. All listings on Silk Road are now No-Escrow (Finalize-Early) for 1-2 months while we implement multi-signature transactions and lobby for mainstream Bitcoin client multi-sig support.

4. All unshipped orders have been cancelled.

5. Vendors may link to other marketplaces on a trail basis until we launch multi-sig, then we will re-evaluate based on community input. We do not want to be a centralized point of failure, but we also do not want to lead our buyers into dangerous waters.

6. From this point forward DO NOT trust markets with centralized escrow. Use multi-signature transactions whenever possible, with trusted third parties as escrow providers.

Everything will be offline for 24-48 hours to minimize variables as we continue to investigate. The evidence we have below will be expanded based on our findings.

- ——————

No marketplace is perfect. Expect any centralized market to fail at some point. This is precisely why we must unite in the decision to decentralize.

We are relieved that our security procedures protected user identities, and that no servers were compromised. This was not a worst-case scenario: nobody will be getting arrested from this. Financial loss is terrible, but will not put all of us behind bars.

The details we have on the hacker are below. Stop at nothing to bring this person to your own definition of justice.

Humbled and furious,

Defcon

- ——————

# Attacker Intel as of 2014-02-13 18:00:00 UTC

We normally do not doxx anyone, and hold user information sacred. But this is an extreme situation affecting our entire community, and all three users who have exploited this vulnerability are very much at risk until they approach us directly to assist with any information.

Do not reveal any details of the attack. This will jeopardize your reward. Contact us directly.

If anyone has purchased or sold to these usernames, expect generous bounties for any information you can contribute which leads to identification.

# Attacker 1: (Responsible for 95% of theft)
Suspected French, responsible for vast majority of the thefts. Used the following six vendor accounts to order from each other, to find and exploit the vulnerability aggressively.

## Usernames used:
narco93
ketama
riccola
germancoke
napolicoke
smokinglife

Transactions listed at bottom of this file. Finding Attacker 1 is top priority.

# Attacker 2: (Responsible for ~2.5% of theft, using same methods towards end of attack lifecycle, likely knows Attacker 1)
LethalWeapon – Australia – “stumbled upon” large amount of BTC

# Attacker 3: (Responsible for ~2.5% of theft, using same methods towards end of attack lifecycle, likely knows Attacker 1)
mrkermit – Australia

# Theft Withdrawal Transactions and historical withdrawals by Attacker 1
address,txid_cleaned
{Here some big list of withdrawal addresses with the stolen bitcoins}

=====End Quote====

Aside from the endless marketplaces being hacked every day now, this is the most shocking event we have encountered – as Silk Road being the largest DarkNet market nowadays was probably holding the largest sum of money of them all – it is not yet clear how many Bitcoins were stolen exactly, but its almost certain that this is about to become the largest theft in the Deep Web history – bigger than the Sheep Marketplace Scam that had amount equal at the time to $40 million in bitcoins stolen by its admins.

This case only serves as ANOTHER, Very Painful lesson about – why on-site escrows are bad, and should not be used! only direct transaction or mulsig escrow like the one offered at themarketplace.i2p are the safe way to conduct business on these sites.

Is this the end of the centralized marketplaces?

We sure hope so! as we posted here again and again, they are not safe, and will always end up being hacked or having the money stolen by their admins.

So who were the hackers?

Few hours before the announcement we at DeepDotWeb received a mail saying: “SilkRoad hacked, 150 BTC stolen, you heard it first from me” this was sent to us by a reddit user who claimed since yesterday he was going to hack SR and steal the sites money – we are trying to verify if this amount matches the amounts that were stolen by the “smaller” hackers that Defcon reported in his post, the others remain unknown.

The Silk Road moderators ranged from pleading or threatening the hackers:

To a complete shock:

To an Apology:

The users reaction was not much different obviously and ranged between shocked / angry / desperate or accusing the sites admins to the thief’s themselves:

IS ANYONE ELSE BUYINGGGG THIS? !!! WE ARE FIXING ESCROW WE ARE FIXING VENDOR REFUNDS? WE ARE DOING ALL WE CAN THIS SHEEP !!!! STYLE FUCKING BY OUR TRUSTED SR GUYS , ITS FUCKING PLAIN AND SIMPLE ESCROW SYSTEM WAS A SCAM SO EVERY COCKSUCKER WHO DIDNT FINALZE THE COINS STAYED IN THE BANK AND OPPS WE HAVE BEEN HACKED !!! WE ARE FIXING THE VENNDOR REFUND ? YEAH RIGHT RIGHT ANOTHER PERFECT SCAM, MORE COIN IN THE BANK AND AT THE RIGHT TIME AGAIN OOPSS WE HAVE BEEN HACKED \ DEFCON GO FUCK YOUR SELF , U GUYS HAVE NOT DOMNE NOTHING ABOUT THE ESCROW SYSTEM , U HAVE DONE NOTHIGN ABOUT VENDOR REFUND , ALL U GUYS DID IS LET THE FUCKING BANK BUILD UP AND SORRY GUYS WE HAVE BEEN HACKED EVERY DOG GETS THERE DAY AND I CANT WAIT TILL I SEE ONE OF U FALL

Some even tried to help in some way.

For us – the big question is “how much”? , we will keep following up on this and updating this post as we get new information – for now, you can check out other site on this list.

Share and Enjoy

Official Enterprise Blog: VMWare to Bring Traditional Windows Apps and Desktops to Chromebooks

Thu, 13 Feb 2014 12:33:44 -0800

Comments: " Official Enterprise Blog: VMWare to Bring Traditional Windows Apps and Desktops to Chromebooks "

URL: http://googleenterprise.blogspot.com/2014/02/vmware-to-bring-traditional-windows_12.html

Posted by Rajen Sheth, Director of Product Management, Chrome

Cloud applications allow flexibility, scalability and security and enable a work-anywhere environment, but many of our customers still use traditional desktop applications. Desktop as a Service (DaaS) helps bridge the gap between the cloud and a traditional desktop by allowing you to run your traditional software in the cloud and have applications appear on your Chromebook similarly to how they run today. An example might be your Windows based accounting application.

Chromebooks have continued to help more and more customers over the last year. According to NPD, Chromebooks made up 21% of U.S. commercial laptop sales in 2013. Customers prefer the low total cost of ownership, the central web-based management console and the fact that they don't need any additional security or anti-virus software. These are reasons why businesses large and small are adopting Chromebooks.

Today, customers can fully embrace the cloud with Chromebooks using VMware Horizon™ DaaS®. VMware and Google are working together to make the migration of legacy applications even easier, by using the HTML5/Blast experience from Chromebooks. This means you can work with Chromebooks and connect to a Windows experience running VMWare™ Horizon View.

As the countdown to Windows XP end of life continues, deploying Chromebooks and taking advantage of a DaaS environment ensures that security vulnerabilities, application compatibility and migration budgets will be a thing of the past.

VMware Horizon DaaS enables customers to centralize other desktop environments and manage these as a cloud service. Initially available to customers as an on-premise service or by VMWare vCloud Service Provider Partners (VSPPs) offering DaaS in the cloud or within hybrid deployments. Users will be able to access their Windows applications, data and desktops using VMware’s Blast HTML5 technology to their Chromebook.

This technology is available now by bringing together VMware Horizon View 5.3 and Chromebooks as an on-premise service and will be available soon as an application that can be installed from the Chrome Web Store. To find out more follow this link.

Startup Sales Negotiations 101 - How to Respond to Discount Inquiries

Thu, 13 Feb 2014 12:59:44 -0800

Comments: "Startup Sales Negotiations 101 - How to Respond to Discount Inquiries"

URL: http://blog.close.io/startup-negotiation-101-how-to-respond-to-discount-inquiries

People will sometimes reach out and ask for a discount on your product before they took the time to sign up for a trial and use it at all. What do you do when that happens?

Instead of debating if you should or shouldn't offer them a discount right away, you need to refocus their energy on what really matters: your product!

Let's explore the 3 core reasons why you never want to negotiate pricing before someone had a chance to trial your product and determine that it's a good fit.

#1: You're starting the relationship on the wrong foot

People who ask you to lower your prices before having invested any time using your product are usually trouble.

This can often lead to winning a new customer that is going to expect you to give 24/7 premium phone support, prioritize features based on their needs all while trying to pay you pennies on the dollar. If you start the relationship by giving them everything they ask for, don't be surprised if they keep asking for more in an unreasonable fashion. This is ultimately unsustainable and unhealthy for both sides.

#2: They're buying for the wrong reason

At this point they can't tell if your product is a good fit for them since they never used it. Your first priority should always be to help people explore and discover that your product is really solving their problem before negotiating what the final pricing should be.

Discounting your product upfront might help you close some deals faster but will often lead to these customers ultimately discovering that they should have never bought in the first place. Always be wary of prospects that don't want to do their homework upfront. Nothing sucks more than a new customer that cancels immediately after having created a ton of support and onboarding cost.

#3: You're negotiating on price vs. value

The problem with people trying to negotiate pricing before testing your product is that you are forced to negotiate on price rather than value.

They didn't have a chance to build up any desire to buy and discover the massive value your product could deliver to them. All of the sudden your product turns into a commodity and your only differentiation is offering them the lowest price possible.

#4: You're negotiating without leverage

The more time people invest in your product the more "invested" they become and naturally the harder it is for them to "throw away" the time they put into exploring your product and making it part of their daily workflow.

You always want to postpone the most difficult/complex parts of the sales negotiation till the end of the sales cycle. That way you ensure the right amount of momentum as you move forward in the sales process and avoid too much upfront friction.

Here is what your response should be when someone asks for a discount without having tried your product:

"Thanks for inquiring about pricing options! Why don't you sign up for a trial and give the product a go? If you find out that it's a great fit I'll take care of you and make sure you get a price that makes you happy. Sound fair enough?"

This works every time. The reply you usually get will be:

"Great! Just signed up and giving the product a go. Thanks!"

What's the result you should expect?

9 out of 10 times the people that turn out to be a bad fit will self select during a trial and just leave. The prospects that are a good fit will love your product so much that they will not negotiate hard for a discount since they now really understand its value.

Even if they do it's fine to give great customers a good price because you know they are buying for all the right reasons and will probably stay with you for a long time.

We've done this thousands of times and it always works. I hope this startup sales negotiation tactic serves your business as much as it has ours :)

P.S. The same tactic doesn't just work for people that are "unreasonable" and potentially a bad fit. Even when uber nice prospects are worrying about pricing upfront we tend to ask them to first discover if the product is even a good fit for their needs. See below a recent twitter exchange as a good example for this:

Redesigning the United Flight Search Interface in Photoshop | Nathan Barry

Thu, 13 Feb 2014 13:40:53 -0800

Comments: "Redesigning the United Flight Search Interface in Photoshop | Nathan Barry"

URL: http://nathanbarry.com/redesigning-united-flight-search-ui-photoshop/

Design | February 13th, 2014

Flying out of Boise, Idaho you don’t have a lot of options. There are only direct flights to a few cities and every itinerary has its downsides. Because of that I spend a lot of time using United’s advanced flight search.

I had two goals in redesigning this screen:

To know how it could look and function. To teach designers how to use Photoshop.

In this post I’ve got a two-part video walking you through the redesign. These videos are the first half of the entire case study available in my upcoming course, Photoshop for Interface Design.

The videos

Part 1

Part 2

The finished redesign

These videos take you halfway through the case study. The next two are coming next week. To make sure you get those sign up for the email list at the end of this page.

In the mean time, here’s what the final design looks like:

Ready to learn more?

Photoshop for Interface Design is my new course on using Photoshop to design great software. Whether iOS apps, websites, or complex SaaS applications, I’ll teach you the techniques used by professional designers. You can skip right past the Photoshop tutorials for photographers (no red-eye removal tutorials here) and go right to learning about paths, layer styles, and the techniques you’ll use every day.

If you want to learn to use Photoshop—and value your time—this course is for you.

Learn to design your own interfaces in Photoshop

Leading up to the course launch you'll receive free tutorials, resources, and a special bonus at launch.

Thanks for your interest! Now check your email to confirm your subscription.

OCaml: what you gain - Thomas Leonard's blog

Thu, 13 Feb 2014 14:30:11 -0800

Comments: "OCaml: what you gain - Thomas Leonard's blog"

URL: http://roscidus.com/blog/blog/2014/02/13/ocaml-what-you-gain/

Way back in June, in Replacing Python: second round, I wrote:

The big surprise for me in these tests was how little you lose going from Python to OCaml.

Of course, I was mainly focused on making sure the things I needed were still available. With the port now complete (0install 2.6 has been released, and contains no Python code), here’s a summary of the main things you gain.

Table of Contents

This post also appeared on Hacker News and Reddit, where there are more comments.

( This post is part of a series in which I converted 0install from Python to OCaml, learning OCaml in the process. The full code is at GitHub/0install. )

A note on bias

I started these blog posts unemployed (taking a career break), with no particular connection to any of the languages, and motivated to make a good choice since I’d be using it a lot. I wasn’t biased towards OCaml; it wasn’t even on my list of candidates until a complete stranger suggested it on the mailing list. But I must now disclose that, since my last blog post, I’m now getting paid for writing OCaml.

Functional programming

Some people commented it was good to see more projects moving to functional programming. So, what’s it like doing functional programming after Python? To be honest, not much has changed. According to OCaml’s What is functional programming?, “In a functional language, functions are first-class citizens” and “The fact is that Perl is actually quite a good functional language”.

So, if you’ve ever used Python’s (built-in) map, reduce, filter or apply functions, ever written or used a decorator or ever passed a function as an argument to another function, you’re already doing functional programming as far as OCaml is concerned. By contrast, “pure functional programming” (as in Haskell) would be a major change.

OCaml does make partially applying functions easier, which is sometimes convenient, and it supports tail recursion. Tail recursion allows you to write loops in a functional style (without needing break, continue or mutable state). That can make it easier to reason about loops, but I couldn’t find any examples in 0install where this style was clearly better than a plain Python loop.

Type-checking

I’ve used statically-typed languages before (I used to program in Java for my day job). That can catch many errors that Python would miss, but OCaml’s type system is far more useful than Java’s. Here’s an example, where we want to display an icon for some program in the GUI:

1 2 get_icon program |> widget#set_icon (* Error! *)

Error: This expression has type Icon.t -> unit
 but an expression was expected of type Icon.t option -> 'a

Oops. The program might not have an icon (icons are optional). We’ll need to use a default one in that case:

1 2 3 get_icon program |> default generic_program_icon |> widget#set_icon

Downloading some data:

1 2 3 4 match download url with | `success data -> process data | `network_failure msg -> show_error_dialog msg (* Error! *)

Warning 8: this pattern-matching is not exhaustive.
Here is an example of a value that is not matched:
`aborted_by_user

Oops. The user might click the “Cancel” button - we need to handle that too:

1 2 3 4 match download url with | `success data -> process data | `network_failure msg -> show_error_dialog msg | `aborted_by_user -> abort ()

When registering an extra feed to an interface we want to download it first to check it exists:

1 2 3 let add_feed iface (feed_url:feed_url) = download_feed feed_url; (* Error! *) register_feed iface feed_url

Error: This expression has type feed_url
 but an expression was expected of type [< `remote_feed of url ]
 The second variant type does not allow tag(s) `local_feed

Oops. The user might specify a local file too:

1 2 3 4 5 let add_feed iface (feed_url:feed_url) = begin match feed_url with | `remote_feed _ as feed_url -> download_feed feed_url | `local_feed path -> check_file_exists path end; register_feed iface feed_url

Java makes you do all the work for static type checking, but manages to miss many of the benefits. No matter how much care you take with your Java types, there’s always a good chance you’re going to crash with a NullPointerException. By requiring correct handling of None (null) and ensuring pattern matching is exhaustive, OCaml’s type checking is far more useful. As with Haskell, when a piece of OCaml code compiles successfully, there’s a very good chance it will work first time.

And, of course, static checking makes refactoring much easier than in Python. For example, if you remove or rename something, the compiler will always find every place you need to update.

Data type definitions

OCaml makes it really easy to define new data types as you need them. The types are always easy to see, and you know that OCaml will enforce them (unlike comments in Python, which may be incorrect). Here’s a record type for the configuration settings for an interface (an optional stability level and a list of extra feeds):

1 2 3 4 type interface_config = { stability_policy : stability_level option; extra_feeds : Feed.feed_import list; }

And here’s a variant (enum / tagged union / sum) type for the result of a download:

1 2 3 4 type download_result = [ `aborted_by_user | `network_failure of string | `success of filepath ]

Polymorphic variants

The OCaml labels tutorial describes polymorphic variants as a way to use the same name (e.g. Open) for different things (e.g. opening a door vs opening a lock) and says:

“Because of the reduction in type safety, it is recommended that you don’t use these in your code”.

This is quite misleading (and I was quickly corrected when I repeated it). Their real purpose is to support subsets and supersets, which are useful all over the place. Some examples:

The “0install” command-line parser accepts a large number of options. The “0install run” subcommand accepts a subset of these. That subset can be further subdivided into common options (present in all commands, such as --verbose), options common to selection commands (e.g. --before=VERSION) and those specific to “0install run” (e.g. --wrapper=COMMAND).
The GUI code that handles dialog responses (OK, Cancel, etc) must handle the union of all the action button responses it added and the always-present window close icon.
The download code only handles the subset of feed URLs that represent remote resources.
Users can only register local and remote feeds to an interface, not distribution-provided virtual feeds.
Cached feeds contain only remote implementations, local feeds contain local and remote implementations, and distribution feeds contain only distribution implementations. All three types get combined together and passed to the solver.

Here’s an example, showing the run command dividing its options into sub-groups, with the compiler checking that every option will be handled in all cases:

1 2 3 4 5 6 7 let select_opts = ref [] in Support.Argparse.iter_options flags (function | #common_option as o -> Common_options.process_common_option options o | #select_option | `Refresh as o -> select_opts := o :: !select_opts | `Wrapper w -> run_opts.wrapper <- Some w | `MainExecutable m -> run_opts.main <- Some m );

Without polymorphic variants, OCaml’s exhaustive matching requirements mean you’d have to provide code to handle cases that (you think) can’t happen. That’s tedious and your program will crash if you get it wrong. Polymorphic variants mean you can prove to the compiler that only the correct subset needs to be handled at each point in the code. This is fantastic, and I can’t think of any other major language that does this (though I’m sure people will suggest some in the comments).

Immutability

In OCaml, all variables and record fields are immutable by default. This is far saner than Java (where the default is mutable and you must use final everywhere to override it). Immutable is a better default because:

Typically, you want most things to be immutable (in any language).
If you forget to mark something as mutable, the compiler will quickly let you know, whereas forgetting to mark something as immutable would be missed.

With mutable structures, you are always worrying about whether one piece of code will mutate a structure that another is relying on. For example, I originally made the XML element type mutable, but I found I was writing comments like this:

1 2 (** Note: this returns the actual internal XML; don't modify it *) val as_xml : selections -> element

After removing the mutable annotations from the element declaration, the compiler showed me each piece of code I needed to modify to make it work again. Then, I was able to remove those notes.

[ The main difficulty in this conversion was handling XML namespace prefixes. Previously, each element had a reference to its owning document, which held a shared (mutable) pool of prefix bindings. Now, each namespaced item holds its preferred prefix, and the output code builds up a set of bindings before writing out the tree. ]

There is one case where Python and Java do better than OCaml: OCaml strings are mutable! The convention is to treat them as immutable, though.

Abstraction

OCaml makes it very easy to hide a module’s implementation details from its users using abstract types.

I gave one example in the bugs post, where hiding the fact that a sorted tree is really the same type as an unsorted one prevents bugs due to mixing them up.

Here’s another example. In the Python code, we would parse a selections XML document into a Selections object, like this:

1 2 3 class Selections(object): def __init__(self, root_element): ...

I found all this parsing and serialising complicated things and so in the OCaml rewrite I decided to use the plain XML element type everywhere. That did simplify things, but it also removed some safety and clarity from the APIs. Consider the Selections.create function (which now does nothing unless the document is in an old format and needs to be upgraded):

1 2 3 4 let create root = ZI.check_tag "selections" root; if is_latest_format root then root else convert_to_latest root

It’s nice and simple, but it just returns an element. It would be easy to pass some other XML element to a function that only works on selection documents (or to pass a document that’s still in the old format). We can solve this simply by declaring an abstract type for selection documents in the interface file (selections.mli):

1 2 type t val create : element -> t

(note: it’s an OCaml convention for a module’s main type to be called t; other modules will refer to this type as Selections.t)

I think this gets the best of both worlds. Internally, a selections object is just the XML root element, which is simple and efficient, but code using it can’t mix up the types. And, of course, we can change the internal type later if needed without breaking anything.

This isn’t a particularly novel idea (you can do something similar in C). However, Python and Java would require you to write a wrapper object around the object you want to hide, and Python makes it easy for users of the API to access the internal representation even then. If you’re writing a library, OCaml (like C) makes it clear when you’re changing the module’s interface vs merely changing its implementation.

( There is another interesting feature, which I haven’t used yet: you can use the “private” modifier to say that users of the module can see the structure of the type but can’t create their own instances of it. For example, saying type t = private element would allow users to cast a selections value to an XML element, but not to treat any old XML as a selections value. )

I did experience one case where abstraction didn’t work as intended. In the SAT solver, I declared the type of a literal abstractly as type lit and, internally, I used type lit = int (an array index). That worked fine. Later, I changed the internal representation from an int to a record. Ideally, that would have no effect on users of the module, but OCaml allows testing abstract types for equality, which resulted in each comparison recursively exploring the whole SAT problem. It can also cause runtime crashes if it encounters a function in this traversal. Haskell’s type classes avoid this problem by letting you control which types can be compared and how the comparison should be done.

Speed

Python is well known for being slow, but much of what real programs do is simply calling C libraries. For example, when calculating a SHA256 digest, C does all the heavy lifting.

Despite this, I’ve found OCaml to be fairly consistently 10 times faster in macro benchmarks (measuring a complete run of 0install from start to finish). Also, although I’ve added a lot of code and dependencies since the initial benchmarks, it still runs almost as quickly. The 0release benchmark took 8ms with June’s minimal version, compared to 10ms with the final version.

When doing pure calculations (e.g. a tight loop adding integers), OCaml is typically more than 100x faster than Python.

Even so, OCaml is probably not a great choice for CPU-intensive programs. Like Python, it has a global lock, so you can’t have multiple threads all using the CPU at once. But if you’re writing small utilities that need to run quickly, it’s perfect.

No dependency cycles

Perhaps I’m making a virtue of a flaw here, but I like the fact that OCaml doesn’t allow cyclic dependencies between source files. I think this leads to cleaner code (back when I was writing Java, I wrote a script to extract all module dependencies and graph them so I could find and eliminate cycles).

What this means is that in any OCaml code-base, no matter how complex, there’s always at least one module that doesn’t depend on any of the others and which you can therefore read first. Then there’s a second module that only depends on the first one, etc. For example, here are the modules that make up 0install’s GTK plugin (note the lack of cycles):

Cycles can be a problem when converting existing code to OCaml, though. For example, the Python had a helpers.py module containing various high-level helper functions (e.g. get_selections_gui to run the GUI and return the user’s selections, and ensure_cached to make sure some selections are cached and download them if not). That doesn’t work in OCaml, because the helpers module depends on the GUI, but the GUI also depends on the helpers (the GUI sometimes needs to ensure things are cached). The result is that I had to move each helper function to the module it uses, but I don’t mind because the result is a clearer API.

Another example is the Config object. When I started the Python code back in 2005, I was very excited about using the idea of dependency injection for connecting together software modules (this is the basis of how 0install runs programs). Yet, for some reason I can’t explain, it didn’t occur to me to use a dependency injection style within the code. Instead, I made a load of singleton objects. Later, in an attempt to make things more testable, I moved all the singletons to a Config object and passed that around everywhere. I wasn’t proud of this design even at the time, but it was the simplest way forward. It looked like this:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 class Config(object): @property def fetcher(self): if not self._fetcher: from zeroinstall.injector import fetch self._fetcher = fetch.Fetcher(self) return self._fetcher @property def trust_mgr(self): if not self._trust_mgr: from zeroinstall.injector import trust self._trust_mgr = trust.TrustMgr(self) return self._trust_mgr ...

OCaml really didn’t like this design! config.py depends on all the other modules because it calls their constructors, while they all depend on it to get their dependencies.

Note that this design isn’t very safe: Fetcher’s constructor could ask for config.trust_mgr, and TrustMgr’s constructor could ask for config.fetcher. In Python, we have to remember not to do that, but in OCaml we’d like the type checker to prove it can’t happen.

In most places, I fixed this by passing to each constructor just the objects it actually needs, which is cleaner.

Another approach, which I used when lots of objects were needed, is that instead of requiring a config object, a class can take simply “an object with at least fetcher and trust_mgr methods”. Then we know statically that it will only call those methods, even though we actually give it the full config object.

The result of all this is that you can look at e.g. the fetch.mli interface file and see exactly which other modules it depends on, none of which will depend on it.

GUI code

Converting the GTK GUI to OCaml (using the LablGtk bindings), I replaced 5166 lines of Python (plus 1736 lines of GtkBuilder XML) with 4017 lines of OCaml (and no XML). I’m not sure why, but writing GTK code in OCaml just seems to be much easier than with Python.

I used GtkBuilder in the Python code in the hope that it would make it easier to modify the layouts, and to improve reliability (since the XML should always be valid, whereas Python code might not be). However, it actually made things harder because Glade (the editor) is constantly trying to force you to upgrade to the latest (incompatible) XML syntax, and I ended up having to run an old OS in a VM any time I wanted to edit things.

In the OCaml, the static type checking gives us similar confidence that the layout code won’t crash. Also, with GtkBuilder you name each widget in the XML and then search for these names in the code. If they don’t match, it will fail at runtime. Having everything in OCaml meant that couldn’t happen. [ Note: I later discovered that LablGtk doesn’t support GtkBuilder anyway. ]

Here’s an example of some OCaml GTK code and the corresponding Python code. This shows how to build and display a menu (simplified to have just one item):

OCaml 1 2 3 4 let menu = GMenu.menu () in let explain = GMenu.menu_item ~packing:menu#add ~label:"Explain this decision" () in explain#connect#activate ~callback:(fun () -> show_explanation impl) |> ignore; menu#popup ~button:(B.button bev) ~time:(B.time bev);

Python 1 2 3 4 5 6 7 8 9 10 11 global menu # Fix GC problem with PyGObject menu = gtk.Menu() item = gtk.MenuItem() item.set_label('Explain this decision') item.connect('activate', lambda item: self.show_explanation(impl)) item.show() menu.append(item) if sys.version_info[0] < 3: menu.popup(None, None, None, bev.button, bev.time) else: menu.popup(None, None, None, None, bev.button, bev.time)

Some points to note:

LablGtk allows you to specify many properties at once in the constructor call, whereas in PyGTK we need separate calls for each.
The Python API broke between Python 2 and Python 3, so we have to make sure to use the right one. It’s not sufficient to test the Python code using only one version of Python!
The Python bindings have always suffered from garbage collection bugs. If we don’t store menu in a global variable, it may garbage collect the menu while the user is still choosing - this makes the menu disappear suddenly from the screen!
Actually, I see that Python’s MenuItem takes a label argument, so maybe I could save a line. Or maybe that doesn’t work on some older version. It’s not worth the risk of changing it.

Update: I used the original layout for the OCaml above as I was comparing line counts, but it’s a bit wide for this narrow blog and some people are finding it hard to read. Here’s an expanded version which uses less special syntax:

1 2 3 4 5 6 7 8 let menu = GMenu.menu () in let explain = GMenu.menu_item ~packing:menu#add ~label:"Explain this decision" () in let callback () = show_explanation impl in let _signal_id = explain#connect#activate ~callback in menu#popup ~button:(B.button bev) ~time:(B.time bev)

Here, ~ indicates a named argument and # is a method call. So menu_item ~packing:menu#add ... is like menu_item(packing = menu.add, ...) in Python.

However, I did still have a few problems with the OCaml GTK bindings:

There’s no support for custom cell renderers. I used three of these in the Python version and had to find alternative UIs for each.
Various minor functions aren’t included for some reason. The ones I wanted but couldn’t find were Dialog.add_action_widget, Style.paint_arrow, MessageDialog.BUTTONS_NONE, Dialog.set_keep_above, icon_size_lookup and Selection_data.get_uris.
You usually don’t need the result of creating a label or attaching a signal so you need to use ignore, which can cause silent failures if you forgot an argument (it will ignore the partial function rather than the widget or signal result). Probably I should make ignore_signal and ignore_widget utility functions.
I had to work around a bug in the IconView support (reported, but with no response).

API stability

This is a community thing rather than a language issue, but OCaml and OCaml libraries seem to be very good at maintaining backwards compatibility at the source level. 0install supports the old OCaml 3.12 and libraries in Ubuntu 12.04 up to the latest OCaml 4.01 release without any problems. The only use of conditional compilation for compatibility is that we don’t define the |> operator on 4.01 because it’s already a built-in (this avoids a warning).

On the other hand, binary compatibility is very poor. You can replace the implementation of a module with a newer version as long as the public interface doesn’t change (good), but any change at all to the interface requires everything that depends on it to be recompiled, and then everything that depends on them, and so on.

For example, if the List module adds a new function then the signature of the List module changes. Now any program using the new version of the List module is incompatible with every library binary compiled against the old version. Even if nothing is even using the new function! This means that distributing OCaml libraries in binary form is effectively impossible.

Summary

OCaml’s main strengths are correctness and speed. Its type checking is very good at catching errors, and its “polymorphic variants” are a particularly useful feature, which I haven’t seen in other languages. Separate module interface files, abstract types, cycle-free dependencies, and data structures that are immutable by default help to make clean APIs.

Surprisingly, writing GTK GUI code in OCaml was easier than in Python. The resulting code was significantly shorter and, I suspect, will prove far more reliable. OCaml’s type checking is particularly welcome here, as GUI code is often difficult to unit-test.

The OCaml community is very good at maintaining API stability, allowing the same code to compile on old and new systems and (hopefully) minimising time spent updating it later.

The Couple That Pays Each Other to Put Kids to Bed - NBC News

Thu, 13 Feb 2014 14:30:11 -0800

Comments: "The Couple That Pays Each Other to Put Kids to Bed - NBC News"

URL: http://www.nbcnews.com/business/consumer/couple-pays-each-other-put-kids-bed-n13021

Like most couples, Daniel Reeves and Bethany Soule fight. Unlike most, they solve their spats with dollars.

When a chore comes up neither wants to do, they put their hands behind their back and think of how much they'd pay the other to do it. Then, at the same time, they show that number on their fingers.

It might be $20 to put the kids to bed, $4 to go fetch tacos in the rain, or $400 to make travel plans. Whoever has the lowest bid does the job and gets paid their bid to do it.

It's basic game theory and behavioral economics, just applied to a marriage.

Folks like Danny and Bethany, part of the burgeoning "quantified self" movement, use smartphones and gadgets to track their goals. From the steps they take, number of emails they answer, and hours they play with their kids, they believe a better you is just a data point away.

For this Portland, Ore., couple, just as in love as when they met nine years ago, their objective is total fairness and zero resentment, using cold hard cash.

"'But I did the dishes last time!' turns into, 'But I did the dishes last time and I got paid $40 for it, so that was kind of awesome!'" said Bethany, 32, who co-founded and runs a startup called Beeminder with her husband, Danny, 38. They have two kids, Faire and Cantor, ages 6.7 and 4.997, according to a program Danny wrote.

If that sounds geeky, that's just for starters.

At one point, they paid each other $24 an hour to watch the kids. At another, Danny paid Bethany about $30,000 in various costs to have their first child while he worked a full-time tech job and she took time off school. Split over what to name their kid, they submitted sealed bids worth several thousand dollars.

"If the money makes the other person feel valued ─ and like there's a payoff ─ great," said Paula Szuchman, co-author of Spousonomics, in an email. "For some couples a thank you and hug might suffice. For this couple they need something more tangible."

"You need to really appreciate that money is utility, you have to trust in the math so much."

In the couple's ledger, Bethany's balance is -$80,421.10, largely money she owes Danny from grad school. Sometimes she pays for more things out of pocket in "the real world" to bring down her debt in the private books. The plan is their startup's eventual success will wipe out the balances.

"Sometimes it will feel unfair in the short term, but you have to trust that it will be fair in the long term," said Danny.

They do use some tricks to make it easier, like separate bank accounts. To cut down on bookkeeping they also say a random number between one and 10 when they haggle. The chore still gets done, but only if their two numbers match is there a payout, which they multiply by 10 to make up for only one in 10 bids matching.

"Normal people have perfect good approximations of this," said Danny. "This is just nerd level turned up to 11."

For two people who met online and share advanced degrees in computer science, it's a great system.

"You need to really appreciate that money is utility," said Bethany, a concept borrowed from economic theory. "You have to trust in the math so much."

So what about sex?

There's no bartering in the bedroom, they say.

"I don't think we've ever disagreed," said Bethany, because there's never been a time when only one of them was in the mood.

Family therapist Terrence Real has no problem with the cash component. "They use it as a gauge to ask the critical question... 'what's it to ya?'" he said.

However, there can be dangers if kids are involved. "If I was their child and I learned that they were bidding over the task of putting me to sleep I might get upset," said Real.

Leah Nash

Daniel Reeves reads his children Cantor Soule-Reeves (rt) and Faire Soule-Reeves (lt) a bed time story at their home in Portland, OR.

Another wrinkle is that the couple hasn't factored for how Danny didn't give birth to two children. He's also earned more in his life, giving him a bigger pot to bet with. They're mulling over whether to give Bethany a lump-sum payment to fix the imbalance.

It sounds mercenary, even cruel. But there are ways within the system to show affection.

Recently Danny and Bethany haggled to see if he would drive her home after a long day of work or if she would have to ride her bike. She bet $15, and he intentionally "lost," so that he could do her the favor.

Communicating in a series of brief instant messages, she replied back with a smiley face emoticon, a symbol of 21st century love distilled down to two characters.

In this blog post, Bethany describes how their system works in more detail. Danny and Bethany's startup, Beeminder, can be found here.

First published February 13 2014, 1:22 AM

cr.yp.to: 2014.02.13: A subfield-logarithm attack against ideal lattices

Thu, 13 Feb 2014 15:04:11 -0800

Comments: " cr.yp.to: 2014.02.13: A subfield-logarithm attack against ideal lattices "

URL: http://blog.cr.yp.to/20140213-ideal.html

The cr.yp.to blog

2014.02.13: A subfield-logarithm attack against ideal lattices Computational algebraic number theory tackles lattice-based cryptography.

2014.02.05: Entropy Attacks! The conventional wisdom says that hash outputs can't be controlled; the conventional wisdom is simply wrong.

2014.02.13: A subfield-logarithm attack against ideal lattices

Background: post-quantum cryptography

Imagine that an attacker is recording all your network traffic. Yes, yes, I know this sounds paranoid, but the U.S. government said in October 2012 [mirror] that we have to watch out for Chinese-manufactured routers providing "Chinese intelligence services access to telecommunication networks" for "economic and foreign espionage by a foreign nation-state already known to be a major perpetrator of cyber espionage".

This recorded traffic includes your public keys, the public keys of everyone you're communicating with, and all the ciphertexts encrypted to those keys. Years from now the attacker will use Shor's algorithm to factor your RSA public keys and compute discrete logarithms of your ECC public keys. The attacker will then decrypt all the ciphertexts. Furthermore, if you're still using RSA and ECC for signatures at that point, the attacker will forge valid signatures, for example on operating-system updates.

Shor's algorithm needs a large general-purpose quantum computer. When I say "general-purpose" I mean that the computer can reliably carry out certain standard quantum operations in any specified order. Beware that one heavily advertised corner of quantum computing, namely D-Wave's quantum-annealing computer, doesn't even try to be general-purpose. D-Wave's computer can't run most quantum algorithms, in particular can't run Shor's algorithm, and has very little evidence of actually being useful. Large general-purpose quantum computers will be much more useful, notably because they can run Shor's algorithm.

It isn't easy to build a large general-purpose quantum computer, but many people publicly working on it seem to think that they'll be successful next decade. Maybe there will be serious obstacles that slow down or stop this success, but there's clearly a serious risk to RSA and ECC, and it isn't sensible risk management to pretend that the problem doesn't exist.

Fortunately, there are post-quantum alternatives to RSA and ECC. My favorites are hash-based signatures and code-based encryption, both of which were introduced in the 1970s and inspire confidence after extensive security analysis.

Lattice-based cryptography

Lattice-based cryptography is another important candidate for post-quantum public-key cryptography, and is the main topic of this blog post. The main practical reason to consider lattice-based encryption rather than code-based encryption is that lattice-based systems advertise much smaller public keys.

Actually, there are two types of lattice-based encryption. The first type uses special types of lattices called ideal lattices; the classic example is the NTRU cryptosystem introduced almost twenty years ago. The second type uses lattices without this special ideal structure. The second type is much slower and has much larger public keys, so from now on I'll consider only the first type. Sometimes people write lattice-based papers of the second type because they find unstructured lattices simpler than ideals, but then there are followup papers adding ideals to the picture (e.g., switching from "LWE" to "Ring-LWE"), because ideals are critical for the efficiency of lattice-based cryptography.

Lattice-based encryption has also attracted attention from theoreticians who talk about its "flexibility for realizing powerful tools like fully homomorphic encryption". The latest speed reports for fully homomorphic encryption are—let me use precise technical terminology here, since I'm a big fan of careful benchmarking—ludicrously slow, but without ideal lattices they would be utterly ludicrously slow. Lattices can also be used for constructing more important tools such as signatures.

The same theoreticians also say that lattice-based cryptography has "strong provable security guarantees". If this is taken literally then it is false advertising. The correct advertising is that a broad class of attacks against various lattice-based cryptosystems can be converted, with limited loss of speed and success probability, into attacks against certain standard lattice problems. But this begs the question of how hard those lattice problems are. (It also doesn't distinguish lattice-based cryptography from other areas of cryptography, but that's a topic for another blog post.)

The prototypical lattice problem is the shortest vector problem (SVP): finding the shortest nonzero vector in a lattice L, given as input a basis for L. There are many papers attacking this problem, and some papers attacking the variants of the problem that actually appear in lattice-based cryptography. The typical attack cost twenty years ago was exponential in n log n, where n is the lattice dimension. Newer attacks take time exponential in n, obviously an improvement if n is large enough. There have been various improvements in the base of these exponentials; subexponential speedups; polynomial-factor speedups; etc.

An attack against ideal lattices

Here's the main point of this blog post: an ideal-lattice attack strategy that, unlike traditional lattice attacks, exploits the multiplicative structure of ideals.

Setup. The prototypical ideal-lattice problem is as follows. Someone has a secret short nonzero element g of a ring R, and tells us a basis for the principal ideal gR. Our task is to find g. The ring R is public, the ring of integers of a standard degree-n number field.

Practically all of the literature on lattice-based cryptography takes specifically R = Z[ζN], the smallest subring of the complex numbers containing ζN = exp(2 π i/N); I'll also focus on this case, so I won't discuss the general theory of number fields. The number field in this case is Q(ζN), the smallest subfield of the complex numbers containing ζN; this field has degree n = #(Z/N)*. There is a ring isomorphism from Q[x]/ΦN(x) to the number field taking x to ζN, where ΦN is the Nth cyclotomic polynomial. The same map restricted to Z[x]/ΦN(x) is a ring isomorphism from Z[x]/ΦN(x) to R.

NTRU takes N to be prime; then n = N-1 and ΦN(x) = (xN-1)/(x-1) = 1+x+x2+...+xn. PASSSign, a recent lattice-based signature scheme, also takes a prime N, specifically 433 or 577 or 769 or 1153; in each case n factors as a product of powers of 2 and 3 to allow "very fast Fourier transform algorithms". In recent papers it is very common to consider only the cases N=2, N=4, N=8, N=16, etc.; then n = N/2 and ΦN(x) = xn+1. For example, SWIFFT takes N=128, so ΦN(x) = x64+1.

There are exactly n different ring morphisms from the number field into the complex numbers. Specifically, there are exactly n different roots of ΦN in the complex numbers (namely all powers ζNj for j in (Z/N)*), and for each root there is a unique ring morphism from the number field into the complex numbers taking ζN to that root. I'll write these n different maps as ψ1,ψ2,...,ψn; the order doesn't matter. Dirichlet's logarithm map takes each nonzero element v of the number field to the vector Log v = (log|ψ1(v)|,log|ψ2(v)|,...,log|ψn(v)|), where log is the natural logarithm.

Precomputation. The attack begins as follows. Write down many small elements s of R, and keep a list of principal ideals sR that factor as products of powers of small prime ideals. When there are more factorizations than small prime ideals, solve for the small prime ideals as products of (positive or negative) powers of these principal ideals. This is simple linear algebra on the exponents in the factorizations.

(There is a tiny algebraic obstruction to this, called the class group of R. A nontrivial class group means that a few tiny prime ideals won't be expressible as products of powers of the principal ideals; but all the remaining small prime ideals will be expressible as products of powers of the principal ideals and those few tiny prime ideals. One can easily trace the effect of this complication through the rest of the computation.)

Note that any excess factorizations end up writing R in various ways as products of powers of principal ideals, i.e., end up writing down units u as various products of powers of elements s of R. We can compute each Log u to any desired accuracy as a corresponding sum of multiples of Log s. Dirichlet's unit theorem tells us that Log R* is a lattice of dimension n/2-1 (since, out of the maps ψ1,ψ2,...,ψn above, 0 map to the reals and n don't), and also tells us the determinant of the lattice, so it's easy to recognize when we have enough units to generate this lattice.

Main computation: the logarithm attack. If the input ideal gR factors into small prime ideals then we immediately obtain an expression for gR as a product of powers of principal ideals, and thus for a generator of gR as a product of powers of elements of R. If the input ideal gR doesn't factor into small prime ideals then we search relatively small elements e of gR until the quotient ideal eR(gR)-1 factors into small prime ideals, again showing us a generator of gR as a product of powers of elements of R.

We're not done yet: we cannot expect this generator to be g or anything else small. The generator is gu for some unknown u in the unit group R*. So let's look for elements of the lattice Log R* close to Log gu; modulo roots of unity this means looking for u in R* with log|ψ1(u)| close to log|ψ1(gu)|, with log|ψ2(u)| close to log|ψ2(gu)|, etc. This is a closest-vector problem for a lattice of dimension n/2-1.

The desired generator g (and -g and other roots of unity times g) will be found in this way: by hypothesis g is short, so |ψ1(g)|,|ψ2(g)|,... are short, so log|ψ1(g)|,log|ψ2(g)| are small, so log|ψ1(u)| is close to log|ψ1(gu)| etc. The coefficients of g are easily interpolated from good approximations to ψ1(g),ψ2(g),..., which in turn are easily computed from good approximations to log ψ1(g),log ψ2(g),... (with any branch cuts), which in turn are easily computed from the known power-of-product expressions of u and gu. One can alternatively compute g by computing u and gu modulo a large prime, or modulo enough not-so-large primes.

Improvement: the subfield-logarithm attack. At this point the lattice dimension has dropped from the original n by a factor of 2. But we can do even better, depending on the exact choice of N.

Consider, for example, N=128, and consider the ring automorphism m of R that takes ζ128 to -ζ128; this is equivalent to the ring automorphism of Z[x]/(x64+1) that takes x to -x. Imagine that, along with the input ideal gR, we magically know the product g m(g). This makes the last step of the logarithm attack much easier: we know gu as a power product, so we know gu m(gu) as a power product, so dividing by the magic g m(g) we know u m(u) as a power product (as usual modulo roots of unity). This gives us many new linear constraints on Log u, effectively reducing the lattice dimension from 32 to just 16.

How do we find the magic g m(g)? Notice that the product g m(g) is in the subring S = Z[ζ64]. Starting from gR, compute gR m(gR) = gm(g)R; compute gm(g)S as the intersection of S with gm(g)R; and compute gm(g) by applying the algorithm recursively to S.

Generalization: Compute all relative norms of gR down to all proper subfields of the original number field. Use the same algorithm recursively to compute each relative norm of g. These relative norms give various linear constraints on Log u; the remaining lattice dimension depends on the exact subfield structure. A large number of subfields of small relative degree should make the attack much faster; if n has enough small prime factors then I would expect the lattice dimension to drop to something slightly sublinear in n, producing a slightly subexponential total attack cost.

For comparison, Gentry and Szydlo introduced a fast method to compute g from the product gg*, where g* is the complex conjugate of g; i.e., from the relative norm of g in the maximal real subfield of the original cyclotomic field. This can be viewed as reducing the security level n of the original field to the security level n/2 of the subfield. However, from the perspective of the logarithm attack, both of these fields already have the same n/2 security level and the reduction doesn't accomplish anything. The subfield-logarithm attack reduces the security level further, breaking the n/2 barrier.

Where does this algorithm come from?

The precomputation that I've described doesn't have anything new: it's one of the standard approaches to computing class groups and unit groups, the most fundamental tasks in computational algebraic number theory. There's one paper from 2009 on fully homomorphic encryption, by Smart and Vercauteren, that mentions but dismisses class-group computations, claiming incorrectly that the "best known algorithms" run in "exponential time in the degree of the field"; see below for more about the speed of these computations.

As for the logarithm attack, I think that this approach to finding generators is reasonably well known among computational algebraic number theorists. On the other hand, after considering the possibility of computing gu, Smart and Vercauteren dismiss gu as being useless since it "will be very large"; they don't seem to realize that one can use logarithms to move from gu to the target g.

I haven't seen the subfield-logarithm attack before. I think it's the first attack that can push the lattice dimension far below n/2. I came up with the idea in 2012 and have been discussing it with various people since then.

How fast is the precomputation?

This is certainly an important question. If the precomputation is too slow to carry out then we'll never even get to the lattice steps. But I find it clear that for large n the precomputation will be vastly less important than the reduced lattice dimension.

For number fields of fixed degree this precomputation is well known to take time subexponential in log|D|, where D is the discriminant. The argument is easy: there are many easy-to-find number-field elements of norm at most x, where x is not much larger than |D|; these norms have chance roughly 1/y of being y-smooth, where y is subexponential in log|D|, roughly exp(sqrt(0.5 log x log log x)); all of the steps in the class-group computation then take time polynomial in y, and thus subexponential in log|D|. The same argument appears in analyzing the number-field sieve, an integer-factorization algorithm that works in the same way and that obviously produces as a side effect the class group of the underlying number field.

As the degree grows, one expects more and more separation between the discriminant and the norms. I haven't seen a careful analysis of this effect in the class-group literature. There are various papers stating that class-group computation takes time at most nO(n) times something subexponential in log|D|; but this is an upper bound, not a lower bound. To slow down the computation so much would require norms exponential in n2, and as far as I can see this is very easy to beat.

In mid-2013 I chatted with an expert on class-group computations, Jean-François Biasse, and learned that for some big families of number fields he had an even better result (by fancier techniques), namely complexity exp((log D)1/3+o(1)) no matter how large n is. Soon afterwards he told me that for ΦN he expects subexponential complexity by at least two different techniques. Later in the year I discussed class-group computations with Steve Donnelly from the Magma computer-algebra group, and he told me that class-group computation for x64+1 is "not very hard".

Has the complete attack been proven? Implemented? Tested?

No, no, and no. Carefully analyzing and optimizing the complete attack will require tons of serious effort. Maybe the attack will turn out to be very slow or non-functional for some reason. But my best guess at this point is that it will work and will force serious changes in parameters for lattice-based cryptography.

It also seems that the power-of-2 cyclotomic fields used in most recent lattice papers are quite far from the safest number fields, even though these fields are constantly advertised as allowing "proofs of security". Here's what's particularly troublesome: The structures used in the "proofs of security", such as automorphisms, are also some of the structures exploited in this attack. This is not an isolated phenomenon: I see many examples where the single-minded pursuit of "proofs of security" adds dangerous structures to cryptographic systems and ends up compromising actual security. Any competent cryptographer will pay attention to the cryptanalytic algorithms and recommend that cryptographic users avoid these dangerous structures.

Whether or not the attack actually turns out to work, it's clear that at this point there has not been adequate security evaluation of ideal lattices. There is a vast body of work on computational algebraic number theory, including problems that are essentially identical to typical problems in lattice-based cryptography; this work has been given only the briefest consideration by lattice-based cryptographers and has been dismissed for frivolous reasons.

Wait a minute. Does solving this ideal-lattice problem actually break lattice-based cryptosystems?

Sometimes yes, sometimes no. Here are four examples to consider:

My understanding is that solving this problem directly breaks the Smart–Vercauteren system.
To break NTRU we want to solve a similar problem having two dimensions on top of a degree-n number field. I think this can be embedded into the same problem in a degree-2n number field.
I don't think the embedding trick will work in general for larger relative dimensions. But maybe it's possible to generalize everything from nonzero ideals to projective modules.
Many systems actually rely on CVP rather than SVP. The usual "embedding" trick for CVP doesn't respect the ideal-lattice structure unless one increases the relative dimension by 1.

People haven't actually been trying to break all these different lattice-based cryptosystems; people have been trying to break a few central lattice problems, most prominently SVP. If SVP is actually secure, and if ideal-lattice SVP is at least as secure as SVP, and if the cryptosystems are at least as secure as ideal-lattice SVP, then the cryptosystems are secure; but if ideal-lattice SVP isn't actually so secure then this logic collapses and we're left with cryptosystems whose security hasn't actually been evaluated.

Is there a defense against this attack?

Here's a concrete suggestion, which I'll call NTRU Prime, for eliminating the structures that I find worrisome in existing ideal-lattice-based encryption systems. This suggestion uses a number field of prime degree, so that the only subfield is Q; and uses an irreducible polynomial xp-x-1 with a very large Galois group, so that the number field is very far from having automorphisms. The best CVP dimension seems to be about half the degree; this is optimal for number fields without many real embeddings. (It's hard to create many real embeddings while keeping coefficients small, and if coefficients are large then there are other problems.) This suggestion also chooses its modulus q so that (Z/q)[x]/(xp-x-1) is a field; this simultaneously avoids (1) NTRU's traditional 2-adic structure and (2) the linear splittings used in most recent papers.

Standardize a prime p larger than 10. Standardize a positive integer t. Standardize a prime q larger than 72t+3 such that p and q are different and xp-x-1 is irreducible mod q. Define R = Z[x]/(xp-x-1).

Alice's public key is a random-looking element h of (R/q)*. Bob sends Alice a message as follows:

Take a random m in R with all coefficients in {-1,0,1}.
Take a random r in R with t coefficients equal to 1, t coefficients equal to -1, and the remaining coefficients all 0.
Send c=m+rh to Alice.
Use a hash of (m,r) as a secret key (for, e.g., AES-GCM) to encrypt and authenticate Bob's actual message to Alice.

Alice secretly computed the public key as follows:

Take a secret f in R with t coefficients equal to 1, t coefficients equal to -1, and the remaining coefficients all 0. Note that 1+3f is nonzero, and thus invertible, in the field R/q.
Take a secret g in R with floor(p/3) coefficients equal to 1, floor(p/3) coefficients equal to -1, and the remaining coefficients all 0. Note that g is invertible in R/q.
Compute the public key h=3g/(1+3f) in R/q.

Given c=m+rh, Alice recovers m and r as follows:

Compute (1+3f)c = (1+3f)m+3rg in R/q.
Lift to R with coefficients in {-(q-1)/2,...,(q-1)/2}.
Reduce modulo 3 to obtain m.
Compute r=(c-m)/h in R/q.

This works because the lifting produces exactly (1+3f)m+3rg in R: i.e., each coefficient of (1+3f)m+3rg is in {-(q-1)/2,...,(q-1)/2}. Indeed, multiplying m by f in Z[x] (with the usual lifts) produces a polynomial of degree at most 2p-2, with each coefficient bounded in absolute value by 2t. Each coefficient of mf in R is thus bounded in absolute value by 6t: note that reducing modulo xp-x-1 adds the coefficient of x2p-2 to the coefficients of xp-1 and xp-2, adds the coefficient of x2p-3 to the coefficients of xp-2 and xp-3, etc. Each coefficient of rg is similarly bounded in absolute value by 6t. Each coefficient of m+3fm+3rg is thus bounded in absolute value by 1+18t+18t = 36t+1, and q was chosen so that (q-1)/2 is at least ((72t+3)-1)/2 = 36t+1. Typically the coefficients of m+3fm+3rg are considerably smaller, but I prefer to avoid the mess of figuring out whether an attacker can trigger decryption failures.

Of course, I don't recommend actually using NTRU Prime unless and until it survives years of serious cryptanalytic attention, including quantitative evaluation of specific parameter choices. I could have screwed up something small, or something big, or there could be something even more dangerous about the entire concept of using ideal lattices in cryptography.

Version: This is version 2014.02.13 of the 20140213-ideal.html web page.

Migrating from MongoDB to Cassandra

Thu, 13 Feb 2014 15:44:50 -0800

Comments: " Migrating from MongoDB to Cassandra "

URL: http://www.fullcontact.com/blog/mongo-to-cassandra-migration/?

The Story

Long long ago (2011) in a Littleton founder’s basement not so far away, the decision was made to use MongoDB as the persistence engine for our Person API product. The idea of the Person API is pretty simple:

Given some input piece of data {email, phone, Twitter, Facebook}, find other data related to that query and produce a merged document of that data. This is essentially a curated, recursive federated search, where multiple databases are consulted, reconsulted and their results aggregated and stored into MongoDB. These databases consist of various internal services (including previously searched data) and APIs from providers across the web.

When we first started, we needed to move fast according to what our market wanted. As our customers queried us and we added data, our database continued to grow in size and operations per second.

We were a young startup and made a few crucial mistakes. MongoDB was not a mistake. It let us iterate rapidly and scaled reasonably well. Our mistake was when we decided to put off fixing MongoDB’s deployment, instead vertically scaling to maintain the Person API product while working on other technologies such as the FullContact Address Book and our ‘secret sauce’ behind the Person API and our deduplication technology (look for posts on this later this month).

Eventually MongoDB started to have issues with lock time percentage, even on the generous hardware it already had. MongoDB has what’s known as a shared-exclusive or readers-writer lock over each database. While a write operation is in progress, a read is unable to proceed until the write operation yields the lock. Even then, a queued write is given precedence over a read and can queue reads, leading to a latency spike. As you can imagine, the Person API writes a lot of data (sometimes over 200K for a single document), averaging a 50/50 or 60/40 distribution of read/writes and several million daily writes. We worked hard to eliminate multiple updates by no longer using partial updates and staging documents in Redis, but even this wasn’t enough and our lock percentage continued to climb into the 40-50%’s leaving us with unhappy customers.

We pushed Mongo onto the best database hardware AWS could provide: hi1.4xlarge’s. With these we had 2TB of SSD. This dropped our lock percentage to a small fraction of the 40-50% we’d been pushing, not to mention much more storage space.

And it was for a long time. We refactored products and launched others. One day we realized our ‘data burn’ was on the order of 20GB a day and we had less than 200GB remaining on our SSDs. Even MongoDB’s ‘repairDatabase’ to compact the data was unable to comfortably free enough data for more than a few days.

The Dirty Solution

The most painful part of this story to admit is the solution we came up with to limp along. Maybe you can guess from the code snippit below:

By the time we had cycles to spend, it was too late to shard effectively. It would have been terribly painful and unacceptably slowed our cluster for days. We could have done it in parallel, but if we were going to make a painful transition it was better to align our database technology to our eventual goals of extreme fault-tolerance, horizontal scalability, and operational simplicity. MongoDB’s HBase-esque model of config servers, shard routing daemons, and the shards themselves is needlessly complicated. Webscale, right?

To buy us time, we ‘sharded’ our MongoDB cluster. At the application layer. We had two MongoDB clusters of hi1.4xlarges, sent all new writes to the new cluster, and read from both. If a document was present in the new cluster we’d return that, otherwise the old cluster. Meanwhile, Sherlock (internal codename for our search system) was busily refreshing older data and writing it into the new cluster as well. This in mind, we projected this to only last 2-3 months.

Cassandra

We’re believers in the Netflix way of thinking. Build with resilience in the DNA of an application, with the public cloud in mind. We’ve been long Apache HBase users, but found it to be tough to maintain 100% availability on Amazon EC2 for a variety of reasons. Cassandra attracted us with a veritable siren song of operational simplicity. No HDFS or Zookeeper dependencies, multiple servers serve the same data without requiring regions to failover (a process of sometimes multiple minutes or more), and a similar data model to HBase.

It’s important to note that Cassandra was a year-long evaluation for us and aligned with our availability and fault-tolerance goals. Additionally, strong consistency isn’t something we needed for our use case. YMMV. That’s not to say things would be simple or expected, so we moved a less critical service over.

Trial Run – HBase -> Cassandra

As experience is everything when it comes to stable production deployments, we first moved a different part of our system, an outbound HTTP proxy cache if you will, to Cassandra. The existing system ran on Apache HBase, quite well for the most part. Periodically, this system is MapReduce’d into a coherent set of data for ingestion. This was a simple refactor and taught us some of the patterns we’d need to undertake our MongoDB conversion.

The general pattern of any online conversion looks a little like this:

We lean toward reading HBase HFiles directly when we can and avoiding HBase. It’s unnecessary overhead when the entire table is processed anyways. Our plan was to go from HFiles -> SSTables -> Cassandra Cluster.

One of the issues we first ran into was a deficiency of Cassandra’s Hadoop MapReduce tooling. By default, it’s not compiled for Cloudera CDH4. We ended up vendoring half of the classes and applying patches from the Cassandra JIRA before giving up on their BulkOutputFormat (BOF) and ColumnFamilyOutputFormat (CFOF). It was a nice concept to write pre-sorted SSTables and load them directly into Cassandra, but it didn’t work out due to a combination of Hadoop bugs and operational issues (Context#progress no-op, SSTableLoader just randomly breaking, having to manually delete the SSTables after load).

The other option was to use the ColumnFamilyOutputFormat, but we had ended up deciding to use CQL3 tables to make things simple from a tooling perspective (cqlsh and co.) and future-proofing perspective. CQL3 is really some pattern-sauce on top of traditional ‘Thrift’ tables using Composite Types. The Hadoop utilities are rooted in these more complex Thrift APIs and doesn’t easily support non-COMPACT CQL3 tables. I’d already written some basic utilities around being able to utilize CQL3 tables from the BulkOutputFormat, MutationUtils.java (these are pretty rough, you’ll get the idea if you need it), and found the CFOF to be just as awkward but without the promise of SSTables bulk-loaded. We didn’t try the CQL3 tools in recent releases.

Eventually we decided to just use Astyanax (Netflix’s Cassandra Client) directly from our MapReduce job reducers, which worked splendidly for writes. The first time we did this, we wrote as fast as we could to the Cassandra cluster. Topping out at just under 9300 writes/s on a 6-node cluster, we were appropriately impressed. Unfortunately, using all the CPU for the write path left no time for leveled compaction and the cluster spent several days trying to compact 128mb SSTables. Thanks to AWS, we spun up another cluster and repeated the job, this time with the Size-Tiered Compaction strategy (less IO, good for spinning disks) and a Guava RateLimiter to allow Cassandra to breathe and compact during the load process. A handy side-effect of ratelimiting, we were able to accurately project when our jobs would be finished.

The rest of this trial wasn’t without issues, some of which we didn’t discover until we’d moved ahead with moving MongoDB to Cassandra. In general, a simple move.

MongoDB Conversion

We’re big users of Hadoop MapReduce and tend to lean on it whenever we need to make large scale migrations, especially ones with lots of transformation. That fact along with our existing conversion project from before, we used 10gen’s mongo-hadoop project which has input and output formats for Hadoop. We immediately realized that the InputFormat which connected to a MongoDB cluster was ill-suited to our usage. We had 3TB of partially-overlapping data across 2 clusters. After calculating input splits for a few hours, it began pulling documents at an uncomfortably slow pace. It was slow enough, in fact, that we developed an alternative plan.

The plan was to export large BSON files to S3 and MapReduce over them the same way we’d converted our cache before. Files are MapReduce’s bread and butter, this would work much better than opening a lot of cursors against a backup-restored MongoDB server. The Mongo-Hadoop project has a BSONInputFormat for reading in BSON dump files. A match made in heaven? Perhaps.

MongoDB is capable of exporting a single BSON file comprising your entire collection with the mongodump utility. The mongodump utility doesn’t need MongoDB to be running, making it ideal to extract data from backups. As we’d kept backups with snapshotted EBS volumes, we were able to boot a new machine with backup-bearing volumes attached. We pointed mongodump at the data directory on the EBS volume and exported the data as BSON to another disk, which took about a day for each cluster. After this process, we had to run mongo-hadoop’s bson_splitter.py tool to generate split points the BSONInputFormat can use, otherwise the entire file must be read in by a single mapper. It goes without saying, a single mapper is very slow. Then you can push both files to S3 and kick off your Hadoop job.

# Approximate commands..
# mongodump --dbpath /mnt/restore/data/ --collection identities --out /mnt/bsondump/
<wait a day>
# python bson_splitter.py /mnt/bsondump/identities.bson
<wait 20 minutes>
# s3cmd --multipart-chunk-size-mb=1000 put /mnt/bsondump/identities.bson s3://bucket/path
# s3cmd put /mnt/bsondump/.identities.bson.splits s3://bucket/path

It’s important to use a newer s3cmd with multipart support. S3 uploads tend to fail for single large objects with single puts, or too many puts. 5mb (the default with s3cmd-1.1.0) was too small — we found an acceptable size at 1GB which partitioned the upload into ~1500 individual uploads resulting in a single large ~1.5TB S3 object. The other cluster was about half the size. Phew. The only wrinkle we had with this process was getting SBT (mongo-hadoop’s build tool) to upload our CDH4-compiled artifacts to our Artifactory for inclusion into our conversion project.

The mapper in this case is quite simple:

package com.fullcontact.hadoop.mongo;
import com.mongodb.hadoop.io.BSONWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import org.bson.BSONObject;
import java.io.IOException;
public class MongoConversionMapper extends Mapper<Object,BSONObject, Text, BSONWritable> {
 @Override
 protected void map(Object key, BSONObject value, Context context) throws IOException, InterruptedException {
 if (key != null && value != null) {
 context.write(new Text(key.toString()), new BSONWritable(value));
 }
 }
}

This emits each document as a different input group. The reducer too can be rather simple. Note: IntWritable key doesn’t match the Text key of the mapper, not shown is an intermediate step to SequenceFiles. We did this to more rapidly develop the reducer against on-disk data vs. re-pulling from S3 each time. Additionally, we wanted to write more jobs over this data for analytical purposes.

 @Override
 protected void reduce(IntWritable randomIdentifier, Iterable<BSONWritable> values, Context context) throws IOException, InterruptedException {
 for (BSONWritable bsonWritable : values) {
 ProfileIdentity identity = null;
 try {
 identity = buildProfile(bsonWritable);
 context.getCounter(Stats.PROFILE_SUCCESS).increment(1);
 } catch (Exception e) {
 log.warn("Error building profile for " + identity + ", ex: " + e.getClass().getSimpleName() + " => " + e.getMessage(), e);
 context.getCounter(Stats.PROFILE_FAILED).increment(1);
 continue;
 }
 try {
 writeCassandraRow(context, identity.get_id(), identity);
 context.getCounter(Stats.CASSANDRA_SUCCESS).increment(1);
 } catch (Exception e) {
 log.warn("Error writing Identity ("+randomIdentifier.toString()+") " + e.getClass().getSimpleName(), e);
 context.getCounter(Stats.CASSANDRA_FAILED).increment(1);
 }
 }
 }
 private void writeCassandraRow(Context context, String storageIdentifier, ProfileIdentity identity) throws IOException, InterruptedException, FullContactException {
 limiter.acquire();
 store.upsert(storageIdentifier , identity);
 }

We ran the conversion and converted all the data to our new data model, along the way cleaning up several artifacts of mistakes long past. The process was fast enough that we needed to rate limit the writes to Cassandra lest we end up with a endlessly-compacting cluster as we had in our first attempt. We then ran the same job against the second cluster of newer data and overwrote any overlapping data from the first load.

At long last, we’d moved MongoDB into Cassandra and started verifying data and ensuring our 9-node m1.xlarge deployment would hold up under our read load by doing dual reads and not returning the Cassandra result to API consumers. Unfortunately, our latency was terrible and we weren’t yet sure why.

Cassandra & Linux Tuning

We’d just started sending reads and realized that spinning rust might not be fast enough for us, or AWS’s rust was even more rusty than we’d imagined. Not everything added up though. With some napkin math, it was pretty obvious we should have been able to sustain much higher loads for the amount of spindles we’d deployed. m1.xlarges are given 4 ephemeral drives to work with. 1 of these we dedicate to the Cassandra write log, the other 3 are put into RAID0 for a data directory. On top of this, we run LUKS+dm-crypt to encrypt the filesystem, as we have the self-imposed requirement to encrypt all contact data. All that being said, 200 IOPS should have been a cakewalk for this system. What was wrong?

The first thing we really noticed is that we were reading 40+MB/s of disk for a relatively modest 500-1000KB/s of network traffic out. That was our first clue that something wasn’t tuned right, either in Cassandra or Linux. The disparity was especially troublesome, we were doing a lot of extra work. We first tried lots of different Cassandra tuning options, but finally came around to Linux.

It’s important to set your disk readahead appropriately, especially databases. We’d forgotten to check it, as the defaults are usually pretty sane. A look at blockdev –report blew us away.

RO RA SSZ BSZ StartSec Size Device
rw 256 512 4096 0 8589934592 /dev/xvda1
rw 128 512 4096 0 450934865920 /dev/xvdb
rw 128 512 4096 0 450934865920 /dev/xvdc
rw 128 512 4096 0 450934865920 /dev/xvdd
rw 128 512 4096 0 450934865920 /dev/xvde
rw 128 512 4096 0 1352803024896 /dev/md127
rw 6144 512 4096 0 1352801452032 /dev/dm-0

For every disk read, we were pulling in 3MB of data (RA is sectors, SSZ is sector size, 6144*512=3145728 bytes) into cache. Oops. Not only were we doing tons of extra work, but we were trashing our page cache too. The default for the device-mapper used by LUKS under Ubuntu 12.04LTS is incredibly sub-optimal for database usage, especially our usage of Cassandra (more small random reads vs. large rows). We turned this down to 128 sectors — 64KB. Below this we saw no benefits.

We also mounted our filesystems with noatime and nodiratime. They’re pretty easy to forget, but given how often SSTables are accessed it definitely adds up.

Immediately after our first readahead change, we saw promising first results:

And disk IO from StackDriver:

Loads flattened and Cassandra was able to perform. We played with some Cassandra tuning too. Given our large number of keys, we opted to bump our key cache to 500MB to help reduce the number of random IOs necessary to find a row. We also tuned down our bloom_filter_fp_chance on our rows to 0.00075 to keep false positive rates low and random IOs down. We considered increasing our read_repair_chance to keep data warm on replicas too, but deemed that to be more harmful than helpful. All throughout, our graphs were our truth.

Over the next few days we continued dual reads, verified data, and moved select customers over before others to receive feedback before flipping the switch. For the last two months, the FullContact Person API has been powered by Cassandra 1.2.10.

That’s a wrap

Moving to Cassandra was a large effort with many long-running steps and acclimation time. Taking up the banner of any new database system is not without it’s share of risks. One really needs to know their data and access patterns to appropriately choose a technology. For us, horizontal scalability and the ability to run our database in a self-repairing autoscaling group are the most important factors for our choice. Paired with Priam, we’ve been able to unsafely terminate Cassandra instances with no impact to production clients. And perhaps the greatest benefit, we’ve been able to continue serving our API clients with an ever-expanding database of contact information.

Our success with Cassandra has inspired other teams to adopt it for problems where traditional RDBMSes are an impedance mismatch. Cassandra will likely power features in our Address Book in the future.

The future is bright for Cassandra at FullContact. As for MongoDB, I can’t say I’ve ever looked back.

Image Credit: Jemimus via Flickr

Update: Check out the Hacker News discussion.

Keybase.io

Thu, 13 Feb 2014 15:54:50 -0800

Comments: "Keybase.io"

The Day We Fought Back by the numbers

Thu, 13 Feb 2014 15:58:50 -0800

Comments: "The Day We Fought Back by the numbers"

URL: https://thedaywefightback.org/the-results/

Moon | Typeset In The Future

Thu, 13 Feb 2014 18:30:52 -0800

Comments: "Moon | Typeset In The Future"

URL: http://typesetinthefuture.com/moon/

I’ll tell you exactly where we are now. We are in the FUTURE; and we are on the MOON.

After studying 2001: A Space Odyssey in intimate detail, Duncan Jones’s Moon was the logical choice for my second foray into sci-fi typography. As this opening shot illustrates, Moon is a bleak, lonely, and above all beautiful love-letter to classic sci-fi typography and design. It’s also one of my favorite sci-fi films of all time.

UPDATE: I forgot to mention it originally, but this article contains some pretty massive spoilers about the plot of Moon. In my defense, you’re here to read about typography, and typography is always essential to the plot. Nonetheless, if you haven’t seen Moon, go and watch it first. The future will still be here when you’re done.

The searching question above is the opening shot of an infomercial for Lunar Industries. Their moon-based mining of Helium-3 is providing a plentiful source of energy on Earth. (On the evidence of the advert, they sound like a great company to work for. I’m sure they have strong ethical values.)

Sounds like science fiction, right? I’m afraid that’s where you would be wrong, sir or madam. As of January 2014, NASA is accepting applications from companies who want to mine the moon. This film isn’t science fiction – it’s inevitable science fact.

Fittingly, the infomercial finishes with an animation of the words ‘sun / moon / earth / energy / future’…

…and a transition to the Lunar Industries logo:

As introductions go, it doesn’t get more Eurostile Bold Extended than that.

And that’s where I have bad news, I’m afraid. It’s not Eurostile. It’s not even Eurostile’s daddy, Microgramma. According to conceptual designer Gavin Rothery, it’s actually Microstyle.

You’re probably wondering how to tell the three apart, right? Don’t worry – it’s trivially easy, as the big number two below will illustrate. Eurostile is in blue; Microgramma is in red; Microstyle is in green:

I’m glad we’ve cleared that up. (For ease of blog post cross-referencing, I’m going to put my fingers in my ears and continue to call it Eurostile anyway.)

Of course, sci-fi typography isn’t just about the Eurostile, you know. Eurostile has a long-standing competitor; a rival, if you will. And the yang to Eurostile’s tenuous yin is none other than our rectilinear friend, Bank Gothic. Moon uses both fonts in quick succession for maximum futuristic effect.

A clever 3D layout of Bank Gothic Medium is used for much of the film’s opening credits, superimposed in-and-over each of the positioning scenes:

One of these scenes introduces us to the film’s central character. Look! It’s Sam Bell, running on a treadmill to keep fit during his three-year stay on the moon. But what’s that on his t-shirt?

Haha! It’s a meaningless, throwaway gag about running. Because ‘runners don’t quit’, do they? Unless this comedy 80s typography has a deeper, hidden meaning…

One of my favorite things about Moon is its constant foreshadowing of the movie’s central twist. Wake Me When It’s Quitting Time is a pretty damn blatant forward reference to what happens later in the film. (And I hate to say it, type-fans, but that’s not Helvetica on Sam’s t-shirt. It’s Arial. You can tell from the Q and the G.)

This opening sequence is also the first time we discover that the moon base is called ‘SARANG – 사랑’:

This is essentially a duplication – the word ‘sarang’ is an English rendition of the Korean word ‘사랑’, which means ‘love’ or ‘affection’. (‘Sarang’ is also etymologically and graphically close to ‘saram’ or ‘사람’, which, ironically, can mean either ‘person’ or ‘people’.) In either case, it’s an unfortunate choice of name for Sam’s permanent home on the Moon, many thousands of miles away from his beloved wife and daughter.

Next up is Moon‘s title card, an inverted homage to Apollo 8‘s famous Earthrise photograph:

…albeit with more Bank Gothic than the original:

(The gradient fill effect on the MOON logotype is a popular sci-fi trope for adding drama to typography. We’ll revisit this in future blog posts.)

Moon uses an interesting angular typeface for its location-establishing shot:

This typeface is OCR-A, which was designed in 1968 for use in optical character recognition systems. It’s actually an ISO standard for character recognition. Moreover, it looks like THE FUTURE, and so it makes a perfect choice for on-screen interstitial positioning shots. (Matthew Skala has very kindly made a modern implementation of OCR-A available for free on his web site.)

Back inside the base, Sam notices that one of the four HE3 harvesters has a full load ready for collection. The on-screen display is classic Bold Extended sci-fi, with bonus points for yellow-and-white text and barber’s-pole patterning:

Also notable in this screenshot is a juxtaposition between the precision of the harvester’s on-screen display, and Sam’s hand-written customization of the surrounding fascia. This human customization of a clinical surrounding is something we certainly didn’t see in 2001, but it’s a common theme in Moon. Sam’s boredom keeps finding creative and subversive outlets within the strict design confines of the moon base environment.

Sam’s pressurized suit has three fabric mission patches:

Although Sam’s patches follow the Apollo mission patch trend for Bold Extended typography, they are very different from those of the Apollo era. NASA astronauts always had creative input into the patches for their missions. In his fantastic autobiography Carrying The Fire, Command Module Pilot Michael Collins describes the design process for the Apollo 11 mission patch typography:

I also penciled APOLLO around the top of my circular design and ELEVEN around the bottom. Neil didn’t like the ELEVEN because it wouldn’t be understandable to foreigners, so after trying XI and 11, we settled on the latter and put APOLLO 11 around the top.

Sam’s patches are the exact opposite of those from the Apollo era – rectangular rather than round; corporate and branded rather than personal and decorative.

Sam records a message for Lunar Industries central back on Earth. He’s unable to speak to them live, due to an ongoing problem with the long-range comms:

The on-screen interface is once again all about the Bold Extended. One of the buttons to the right of the screen has an interesting label. I wonder what it does?

Sam isn’t happy about the fact he’s been on the base for nearly three years. “I’m talking to myself on a regular basis”, he says. (Be careful what you wish for, Sam.)

Sam’s attempts to customize and personalize his environment continue in the living quarters. He’s keeping count of his days on the Moon with a dry-wipe marker on the bathroom wall. By my reckoning, this is 146 days and counting – not quite the nearly-three-years mentioned in the plot.

UPDATE: Patrick Devine has commented on Hacker News that there are actually 156 smileys, not 146. Turns out I miscounted. Patrick also notes that three years is exactly 156 weeks. Good spot, sir!

It’s not clear whether the facial emotions represent Sam’s mental state day by day. Nonetheless, they are a neat parallel to the simple expressions seen on the front of GERTY, Sam’s AI companion in the base:

GERTY communicates his emotions to Sam via the GERTY Unit Primary Emotional Interface. He has a stock set of emoji faces to draw on, based very closely on the standard emoticons that any Internet user will be familiar with. I’ll reference a few of them throughout this article, along with their official emoji names.

Sam has drawn a window and pastoral scene on another metal wall, and surrounded it with family photos…

…and the functional Eurostile of a harvester status screen is similarly customized with family photos and postcards:

Just readable amongst the photos is a Post-it note detailing Sam’s cheeky plans for starting the day:

07.30: Have a wank 07.32: Clean up mess 07.33: Nice cup of tea and a bickie

Sam might be American, but the film’s director and designer are definitely British. (To explain for Americans: ‘bickie’ is British slang for ‘biscuit’; ‘biscuit’ is British slang for ‘cookie’; and you can work out ‘wank’ for yourselves.)

This Post-it is particularly ironic in light of the quotation we see as the camera pans down the wall of photos:

‘Abstinence is a good thing, but it should always be practised in moderation.’ – Anon

I’m delighted to report that the card is set in popular sci-fi font Swiss 911 Compressed.

Next up is a gratuitous shot of Sam having a shower. If this isn’t a ‘backside of the moon’ gag, I’ll be sorely disappointed.

I’ve had a closer look, but I can’t see any typography:

If there’s no typography, then I’m afraid we’re just not interested.

After his shower, Sam gets his hair cut by GERTY:

GERTY is using a futuristic vacuum-based device to suck up Sam’s hair and cut it to the perfect length. You can just make out a Lunar Industries logo on the device’s hi-gloss handle:

Here’s how the device looks in close-up:

HANG ON A MINUTE. What’s that written on the transparent plastic tube?

Crop. Zoom in. Move left. Zoom in again. Enhance.

That embossed text appears to say:

www.haircut.com 2″ EXTENDER

Is this the Moon production team saying that at some point in the future, haircut.com will be purchased by a futuristic hair-cutting company who make robotic trimming devices?

No, dear reader, it is not. Because that future is already here. TODAY.

It turns out that haircut.com is the present-day home of RoboCut Inc.:

…and that RoboCut Inc. are the makers of the RoboCut DIY:

That’s right – the product you see in Moon is a product you can buy yourself today, for only $59.99 (including free shipping).

Want to know how the RoboCut works? Here’s RoboCut founder and inventor, Dr. Alfred Natrasevschi, to explain:

Haircut completed, Sam expresses his annoyance that LunarSat (a.k.a. ‘long range comms’) still hasn’t been fixed. GERTY notes that it’s fairly low on the company’s priority list right now. His passive role in this scene is fascinating. If you watch Moon after 2001, you can’t help but be suspicious of GERTY, in the same way that Ripley can’t help but be suspicious of Bishop in Aliens after her experience with Ash in Alien. GERTY is a calm, relaxing AI voice with a glowing camera lens and corporate logo on the front. Moon certainly isn’t afraid to play on our expectations. After all, what could possibly go wrong?

Sam listens to a message from his wife and daughter. He can’t talk to them live due to the LunarSat mishap, but he’s delighted to see them nonetheless. Just look at the joy on his face:

No. Don’t look at his face; look at his clipboard. That’s where the typographic interestingness is most likely to be found.

Crop. Zoom in. Rotate ninety degrees. Zoom in again. Enhance.

Hold on a minute – it looks like this clipboard was signed on the 16th January 2008. But that’s not THE FUTURE – indeed, it’s not even after Moon was released!

Moreover, who’s the signatory? It looks remarkably like the name of the film’s line producer. (Damn you, high-definition film releases.)

Later on, Sam watches some TV. Specifically, he watches Bewitched – or as it’s also known in the future, ‘SUPERNATURAL COMEDY LADIES’:

As he makes a cup of tea, he spots a strange girl sat in his chair:

While he’s distracted by the plot, let’s take a look at two typographically-interesting posters in the background:

This poster, with its square iconography and cheery SELECTION EVENT – HEARTY FEEDING – PLEASE ENJOY – ORGANIC SUSTAINANCE MENU, makes you wonder if the Lunar Industries catering team learnt their trade at Aperture Science.

Even more interesting is a sign just visible beneath the pre-packed food boxes:

Organic Material – Soylent Storage

Soylent. That name sounds familiar. Surely this isn’t the same foodstuff made by the Soylent Corporation in 1973′s Soylent Green? Because if it is, that would be remarkably worrying. Soylent Green is People!

At least we can be confident that this Soylent isn’t made from people. After all, where would you get them from? There’s only one person on this moon base, and that’s Sam Bell.

Sam burns his hand in all the kerfuffle. GERTY fixes him up in the infirmary. Just visible behind Sam’s head is a screen with the comforting advice to “TRUST ROBOTIC ASSIST”:

ROBOTIC ASSIST is the official name for GERTY 3000, Sam’s AI companion in the base (voiced by Kevin Spacey doing his best HAL impression). And why not TRUST ROBOTIC ASSIST? I’m sure the 3000 series has a perfect operational record.

Sam goes to bed. Beside his bunk is issue 15 of Take Off Magazine:

Take Off was a 132-issue aviation partwork published in the 1980s by Eaglemoss. This issue features V-Bombers and Biz-Jets:

Sam has a sexy dream about his wife. It does not, however, contain any typography.

Sam is rudely awoken by his Karlsson Digibell Alarm Clock, which features a classic LCD font. Sam’s future version of the clock has a bonus feature compared to the one you can buy in shops today. That’s right – Sam’s version plays Chesney Hawkes songs.

Specifically, it wakes him up by playing Chesney’s number one hit The One And Only. “I am, the one and only”, croons Mr. Hawkes.

Yes, you heard me right – the filmmakers are foreshadowing upcoming events via the lyrics of a Chesney Hawkes song.

Over food, we see more of Sam’s Post-it reminders:

Water plants!! 11:30 Gaze at earth from window

…and a picture of the Earth, with Wish You Were Here on it. He really does want to go home, you know.

Sam also has an IKEA instruction booklet tucked behind his Lunar Industries sheet. The reasons for this are somewhat unclear.

Sam heads out to retrieve some more HE3 from Matthew, one of the harvesters. He sees a second vision of the mysterious girl, and crashes into the side of Matthew, damaging both himself and his rover. The rover’s emergency sign once again asks him to TRUST ROBOTIC ASSIST:

There’s a clear message coming through from these on-screen text displays: if things go wrong, ROBOTIC ASSIST should be TRUSTed to make things right. Let’s fade to black with that sensible advice in mind.

Fade up from black. Why hello, Sam! You’re looking remarkably fresh-faced for someone who’s just nearly killed himself in an accident. How’s the head? Fuzzy? Tell you what, why not let GERTY run some tests to check for brain damage:

This test is a Concentration-style matching pairs game. (Unlike Concentration, these cards all have Lunar Industries logos on the back.) Some of the card symbols are reminiscent of Zapf Dingbats – such as the HEAVY GREEK CROSS, and a rotated BLACK DIAMOND MINUS WHITE X – but I do wonder if I’m reading too much into things, and the design team haven’t just drawn some pretty symbols in Illustrator. (I had hoped that the asterisk card might be my own personal favorite, the HEAVY EIGHT TEARDROP-SPOKED PROPELLER ASTERISK, but alas it is not to be.)

“Let’s try another test” says GERTY, like some kind of benevolent GLaDOS. The analogy makes me trust him even less.

Shortly thereafter, we overhear GERTY saying something to Lunar Central about “the new Sam”. What’re you going on about, GERTY?

I’ll tell you exactly what he’s going on about. This is a brand new Sam. He’s a clone of the original Sam Bell, and he’s just been woken up by GERTY. This might get confusing rather quickly, so let’s lean on our good friend ‘typographic convention’, and use subscript to refer to him henceforth as Sam2. Trust me, it’ll make life a damn sight easier once you know where we’re going with this.

Sam2 spots that Matthew has stalled. He asks GERTY to unlock the doors so that he can go and fix it. GERTY says “I’ll pass on your message”. (This is Lunar Industries speak for “I’m sorry Sam, I’m afraid I can’t do that”.)

Sam2 has a dream about his wife. This time, it’s not sexy at all – it’s psychologically and typographically disturbing. There’s a scary Sam1 under the bedclothes, and he works for RANUL SEIRTSUDNI, on the 랑사-GNARAS base:

Lunar Central tells Sam2 to stay put. GERTY apologies for being under strict orders not to let him outside. He’s acting like some kind of… benevolent HAL 9000. This analogy actually endears him to me slightly. Sam2 tricks GERTY into letting him go outside anyway. He finds Sam1 in the rover, and brings him back to the base. GERTY tells him that the person he’s found is also Sam Bell.

GERTICON: OHMYCRIKEYFACE

A considerable amount of plot happens. It’s all fascinating, but if I’m honest, it’s rather light on typographic action. I feel I’d be doing us a disservice to document the whole lot, but I will report an important bit of GERTY dialog when he’s explaining events to Sam1:

“I’m here to keep you safe, Sam.”

That sounds a lot like the First Law Of Robotics to me. If GERTY is programmed to follow the Laws, his robot AI is about to have some interesting choices to make.

A message comes through from Central to say that the crew of the Eliza are on their way to ‘rescue’ Sam2:

They look like a friendly bunch, don’t they? The instructions next to their mugshots say:

Lunar Industries rescue crews have your best intentions at heart. Please try not to panic until they arrive. Remain on-station and make sure you obey their instructions no matter how strange they may seem. After all they’re here to help!

What could possibly go wrong with a crew called Rothery, Ward and Shaw?

Sam2 realizes he must have come from somewhere on the base, and starts hunting around. He and Sam1 have a fight. Sam1 looks in a topical multi-mirror. (Don’t worry, that’s definitely Sam1 in both mirrors. After all, it’s not like there are any more Sams around, right?)

GERTY and Sam1 have a Big Chat. When pressed for the truth about his wife, GERTY says “I can only account for what occurs on the base”.

GERTICON: NOTHINGTOSEEHEREFACE

After some cajoling, GERTY finally tells Sam1 the truth. He’s a clone, and his memories of his wife and daughter are memory implants.

GERTICON: SADTRUTHFACE

The Sams find some signal-blocking antennas outside the base. Sam1 goes back to look at his old video logs. GERTY enters a special password on a Windows Keyboard to unlock all of the video diaries:

Eagle-eyed viewers will notice that the password begins with KLGAN. Unfortunately, the security-conscious Duncan Jones cuts away from the keyboard just before we can hack into his email.

Sam1 learns that Lunar Industries seem to view the words ‘termination’ and ‘contract’ in much the same way as the Mafia. Sam2 finds yet more long-range-comms-blocking aerials. He prints out their coordinates on a handy receipt printer:

I very much doubt those are going to become significant later on.

Now: I’m sure you’ve already noticed that these are Selenographic coordinates, as used to refer to locations on the Earth’s Moon. Specifically, the co-ordinates are:

LAT 034°23′01.2″S, LONG 124°56′67.6″E LAT 121°09′56.2″S, LONG 045°34′56.4″E

We’ll gloss over the fact that one of these numbers has sixty-seven seconds in a minute. However, it’s a little hard to ignore the latitude value of 121°S, given that the maximum value that latitude can take is 90°S at the pole. (Maybe Sam mistyped it due to the stubby fingers you get on space gloves.)

Back at base, the Sams discover a hidden room. Look at that: it’s Sams all the way down!

Trays 0001 through 0006 have already been opened. Sam0005 and Sam0006 (for it is they) decide to open tray 0007:

It’s the small details of this scene that I find most disturbing. Every pre-packed Samn has a pre-packed Wake Me When It’s Quitting Time t-shirt. The person at Lunar Industries responsible for this whole macabre set-up not only had the temerity to subject Sam Bell to his own personal Groundhog Day; they also had the gall to leave cynical in-jokes for every iteration to endure.

Once they’re out of the Tunnel Of Clones, Sam1 asks GERTY why he helped with the password. “Helping you is what I do”, says GERTY.

See, Sam1? I told you you should TRUST ROBOTIC ASSIST – he’s definitely read the Laws. (Although he may have skim-read the bit about not killing people in a space-coffin.)

Sam1 drives the rover a very long way from the base, and re-establishes video contact with Earth. He calls his house. Turns out his wife is dead, but his now-15-year-old daughter has a chat with him anyway. She ends by shouting off-screen to her Dad to say that someone is asking about Mom. We hear the distinct voice of a certain Sam Bell in reply. Sam0 is still alive! (Sam1 closes the video call device quickly, and looks sad and pensive.)

The known-to-be-alive-ness of Sam0 gives an an alternative, cheerier way of looking at the pre-packed t-shirt dilemma for the less depressingly-morbid of you. Presumably Sam0 is complicit in the fact that thousands of Sams are in cold storage on the Moon. If he’s complicit in the scheme, could it be that these small personal details – even that seemingly-sinister t-shirt – have been chosen with Sam0‘s involvement, as homely comforts to soften the terror of Samn‘s reality?

I don’t think that’s the truth of it at all. But if helps you sleep at night, you’re more than welcome to run with it.

We’re back at the base. Sam1 continues to fall apart. Sam2 sees Sam1‘s video call with his daughter, and realizes that he and Sam1 will be killed when the Eliza crew arrive. He convinces GERTY to wake up a new clone. Let’s call him Sam3.

This section of the film gives us a rare close-up of the base’s OPERATIONAL NOTIFICATIONS board:

Amongst other things, we discover that Lunar Industries Ltd. is a Registered Company Of The United Kingdom No. 06346944. Delightfully, this is indeed the case. In addition to the scrolling Eurostile, there’s also a Big Countdown Clock (in a pseudo-LED font), reinforcing the time pressure of Eliza‘s imminent arrival. (It was conveniently fixed at 88:88 before the Eliza plot point came into effect.) Notable on both font displays is an illustrative dot-matrix resolution substantially lower than the actual resolution of the fonts displayed thereon.

Sam1, who is by now in a right state, wakes up to Chesney Hawkes. (See? I told you it was foreshadowing.) He discovers Sam3 in the infirmary.

Talking of foreshadowing: on the movie’s original one-sheet poster, there are three copies of Sam Rockwell’s name, as shadows behind the main one:

Strictly speaking, this is an accurate cast list. The film now stars Sam Rockwell0, Sam Rockwell1, Sam Rockwell2, and Sam Rockwell3.

To put it another way – the movie’s poster is four-shadowing the foreshadowing.

Maybe I should go and have a lie down for a bit, and come back when the conspiracy theories have subsided. It’s a shame sci-fi films don’t have intermissions these days. Let’s transplant the one from my 2001: A Space Odyssey post, and go and have a cup of tea while the Sams work out what to do next.

And we’re back. After much debate, it is decided that Sam1 will go back into the crashed rover; Sam2 will take the HE3 launcher back to Earth; and Sam3 will stay in the infirmary to be discovered by the crew of Eliza. He’s still unconscious, so he doesn’t know this yet, but it’s a decision made by a quorum of hims, so he can’t really complain.

Once he’s dropped Sam1 off, Sam2 loads up the HE3 launcher. GERTY suggests turning him on and off again so that he can’t provide any incriminating evidence to Eliza. “I’m here to keep you safe, Sam. I want to help you.”

Sam2 asks GERTY if he’ll be okay. “Of course. The new Sam and I will be back to our programming as soon as I’ve finished rebooting.”

GERTICON: YAYPROGRAMMINGFACE

Sam2 replies: “Gerty, we’re not programmed. We’re people, you understand?”

As GERTY turns round, we see that Sam1 has stuck a KICK ME Post-it on GERTY’s back side:

Sam2 turns GERTY off. After a short, poignant pause, he removes the Post-it from his back.

GERTICON: REBOOTINGFACE

Sam2 gets ready to launch back to Earth. At the very last minute, he realizes he can use the harvesters to knacker the signal-blocking aerials, and avoid Sam3 suffering a similar fate to his own. He grabs the piece of paper with Chekhov’s Coordinates on it:

…and types a long series of numbers into a keyboard with great precision despite wearing spacesuit gloves. The harvester coordinates update. Go Sam2!

This screen prompts a couple of interesting questions:

1) What’s an ‘OLD MAN OVERIDE’?

2) Is that an infinitely-looping BASIC program?

20 GOTO 10 10 GOTO 20

The Eliza arrives. Sam3 wakes up and asks GERTY what’s going on.

GERTICON: DONTASKMEJUSTREBOOTEDFACE

Eliza crew members find Sam1 in his rover. Sam2 has a psychedelic trip through space. A harvester crashes into an aerial. Long Range Comms finally start working. And if you look closely, you discover that the contents of the portrait-orientation comms monitor is actually being played from a DVD player plugged into a landscape monitor turned 90 degrees clockwise:

All of which is a timely reminder that this frankly awesome sci-fi film was made on a budget the size of a postcard. Which just makes its amazing styling and design all the more impressive. Indeed, Moon is one of my favorite examples of sci-fi storytelling through design – and, to my mind, a worthy successor to Kubrick’s masterpiece.

- @daveaddey

Postscript: Apart from the note about Microstyle, and a nod to the DVD players, I deliberately wrote this post without referring to Moon Conceptual Designer Gavin Rothery‘s superb ‘They Never Went To The Moon’ blog. I much prefer to write these posts as a response to just what I see on screen – it’s more fun that way. You should totally go and read his blog in its entirety, however, as he explains or expands upon much of what I’ve written above. The blog also includes images of many of the computer screens and stickers from around the base, along with Gavin’s original renders for the base design. Good work, sir!

Flappy Space Program by corpsmoderne

Thu, 13 Feb 2014 21:30:25 -0800

Comments: "Flappy Space Program by corpsmoderne"

URL: http://corpsmoderne.itch.io/flappy-space-program

The Economics of Star Trek — Editor's Picks — Medium

Thu, 13 Feb 2014 22:04:25 -0800

Comments: "The Economics of Star Trek — Editor's Picks — Medium"

URL: https://medium.com/editors-picks/29bab88d50

I promise this is about Star Trek. Sort of. Bear with me a moment.

I’ve been reading a lot about robots lately. When I read about robots, and the future, I can’t help but think about it in economic terms. And that inevitably turns my mind to the branch of economics called post scarcity economics. Traditional economics, of course, deals with the efficient allocation of inherently scarce materials. Post scarcity economics deals with the economics of economies that are no longer constrained by scarcity of materials — food, energy, shelter, etc.

The thing that never sits quite right with post scarcity economics, though, at least the very little that I’ve read, is that it’s always sort of an all or nothing affair: you either don’t have enough of anything or you have enough of everything. Thinking of this as a mental exercise is kind of fun, I think, but in reality it seems to me that getting from point A — a scarcity economy — to point B — post scarcity — is going to be a long, complicated journey as some things become more abundant in some places, while other things are still scarce.

What is needed is some sort of interim-, or proto-post scarcity economics.

More and more I find myself thinking we are, as a race, constrained by the economic models we have. We have capitalism, of course, the proverbial worst model except for every other one that dominates much of our planet right now. It’s definitely a scarcity-based system. Then we have the centrally planned systems of Communism and Marxism, not particularly effective, as it turns out. We have European-style socialist capitalism, but that’s still capitalism, and scarcity-based, albeit with a much more robust safety net than we have here in the US. Some Americans seem to think that a robust safety net somehow nullifies the distributed planning of capitalism. I’ll listen to them again when our schools are decent and our life span starts increasing again magically.

The key here, to me, is to start thinking about how economics would work when we decouple labor from reward. Does that make a system inherently communist? I don’t think it does. People work. They get paid. It is market driven, and not centrally planned. In reality,the market already basically dictates this, for who can claim that a Wall Street banker works more than a teacher? The only thing we really need to do is take this to a logical extreme: that people can still get paid doing zero work. This fear seems to be at the heart of most people who say that Europe is communist: if we give people so much welfare, some of them might stop working! Quelle Horreur!

It seems to me that with the rise of machines and robotics, advances in mining technology, energy technology (both fracking and green energy technologies), the obesity epidemic in the US, etc., that there are plenty of reasons to believe that we may be at the beginnings of a post scarcity economy. We have a surplus, no doubt. Of course, we still have legions of people in the world that are starving, and even people still here at home. But we actually have the capacity to feed them, to feed everyone, even now, even if we don’t have the will. It’s not a matter of scarcity; it’s a matter of the organization of labor and capital.

Take a mental journey for a moment with me: what if, one day, technology reaches the point that a small number of humans — say, 10 million — can produce all of the food, shelter and energy that the race needs. This doesn’t seem like insanely wishful thinking, given current trends. There’s no rational reason why the advances in robotics, factories, energy and agriculture could continue unabated for long periods of time. Of course I’m not saying they will, but rather, they could.

So, then, take that journey. What, then, of labor? In today’s terms, a ‘healthy’ economy now is one at or near full employment. A healthy economy now is one where everyone has a job. But in our mental exercise, those jobs are actually unrelated to a healthy economy, at least from strict economic terms. Everyone’s fed and housed and tons of people simply don’t need to work. Right now, we have them working making shit we don’t need. Is that any better than them not working?

I give you we’re in some fringe areas of economics here, but I have always wondered: is there any economic proof that we need full employment to reach full satisfaction of needs? To my knowledge, there isn’t. There’s a body of economics that goes into standards of living, and the increased standard of living. And here we get to our shitty world of unabated consumerism,Thorstein Veblen’s conspicuous consumption and George Battaille’s accursed share — the inevitable destiny of all economies to eventually produce more than they need, and, thus, waste it.

Seems to me that if we could think beyond capitalism and think of a new model, we could break out of this pointless cycle of more and more consumption of shit we don’t need and model things in another way.

Yes yes, of course. We all know that. The problem seems to me that the minute we leave capitalism behind, we only look at the past alternatives of communism, marxism and pure socialism and pooh pooh them. Few people seem to be able to look beyond capitalism without regressing to the other failed economic models of the early 20th century, as if they are the only alternate possibilities for man.

Yet there have been some other attempts to leave capitalism behind that attempt to also leave the baggage of communism, marxism and socialism behind. The most notable is participatory economics, or parecon. This is a worthwhile attempt, I think, but to me it doesn’t quite pass the smell test of being sufficiently un-communist, what with its workers councils and lack of any sort of ruling class. All very un-American, and in any case, a bit preoccupied with “workers” and “individual need” to really work in any post-scarcity economy where the very concept of a laborer is iffy. When you start thinking this way you start getting into the dodgy world of heterodox economics and, well, that’s a world of a lot of crackpots. Some good ideas, sure, but a lot of crackpots, and more to the point, it’s a world devoid of empirical research, which is a serious problem. Economics is really at its worst when it’s just making up theories. It’s a lot more noble when there’s some real data to back it up.

Parecon does have some awesome concepts, though, by the way. I don’t hate it completely. I especially like that people’s say over any issue is proportional to the amount that issue affects them. It also has some states’ rights-ish aura similar to “laws being made at the level closest to those affected.” It’s a worthy school of thought to consider when looking for a pure alternative to capitalism in a vacuum, though probably not very practical in reality for reasons similar to communism (despite not being centrally planned, it still very much hinges on some third party deciding the relative worth of each job — a messy business). More to the point, it doesn’t help us in thinking about our mental picture: a world where a small number of people can produce enough for everyone.

Then I got to thinking. Screw the dodgy world of heterodox economics. Let’s go full-on fantastical and look at sci-fi. There IS actually a model out there that deals fairly realistically with a post scarcity economy. Not only that, it actually takes into account the difficulties of migrating from a capitalist society to a post scarcity society incrementally. It’s not just a theory in a vaccum.

It’s called Star Trek.

Stay with me here.

Star Trek and Economics

The Previous Theories

When looking at the economics of Star Trek, there have been three broad approaches in the past:

Trying to shoehorn Star Trek’s economics into the model of parecon. This is problematic because of the obviously hierarchical society of Starfleet, with Admirals, captains, commanders, chancellors, governors and whatnot, and the clear existence of a relatively strong Federation president, who is democratically elected. Plus we never once see a labor meeting, and it’s pretty obvious personal freedom and enrichment are important to society.Calling the Federation Communist, based on comments from Kirk in Star Trek IV on not having any money in the future and Picard’s speech about the economics of the federation being significantly different than 21st century economics and people pursuing personal enrichment rather than the accumulation of wealth. The problem with this definition is it’s lazy — just because they don’t pursue the accumulation of wealth, it does not mean the Federation is communist. There is absolutely, obviously, still private property in the Federation: most obviously Joseph Sisko’s restaurant in New Orleans and Chateau Picard, evidencing that not just small possessions are allowed but that the land itself is still privately owned. One could argue that these aren’t really Sisko and Picard’s to own, but they are routinely referred to as “his” restaurant and vineyard so we gotta go with Occam’s Razor here and assume they do, in fact, own them.A sort-of guessing game based on the various mentions of Federation Credits and trying to glean the system from every single mention of money or payments within the series. This is always a pain in the ass, especially given the original series sometimes did things that were pretty out there according to later firmly established canon, and later firmly rejected by Roddenberry himself before his death. Additionally, many of the assumptions about Federation Credits seem iffy: are they really currency? Do they have to be? Are they scrip? Rations? We simply don’t know. And in any case, trying to define the entire economy of the Federation — and perhaps even learning something from it — should be more than a matter of resolving obscure trivia references (though of course it’s fun).

None of them seem correct. None of them seem realistic. And yes, let’s go for realistic here, why not?

Let’s take a different approach here.

What we know

Let’s start with the facts.

The Federation is clearly not a centrally planned economy, and therefore obviously not communist. Individual freedom of choice is very obvious. Everyone chooses their careers, and there are many mentions of this throughout the series — witness every single time someone waxes nostalgic about why they chose to enter Starfleet. Witness Bashir going on about why he wanted to be a doctor instead of a tennis player. Witness Wesley dropping out of Starfleet. Witness Vash being an archeologist and Kasidy Yates being a cargo ship captain.

Private ownership still exists — the biggest examples, to me, are Sisko’s restaurant and Chateau Picard, but many other examples abound from all the trinkets everyone owns in their quarters. Crusher’s family owns a (haunted) cottage on some old-Scottish settlement planet. The Maquis routinely refer to “our land,” which they presumably owned, and while an individual tribe may have collectively owned the land through a corporation, like the Alaska Native Land Claims Settlement Act, or through a co-op, they clearly “owned” the land, just like anyone else owned land, while the Federation was the superseding government that could give that territory away to another sovereign party, much like the ceding of the Sudetenland or Guam. Any alternative situation (the government owning the land and renting it to the settlers?) is never alluded to and in any case the words stated (“our land”) clearly indicate private ownership is still very much part of the cultural zeitgeist. Then we have JJ Abram’s Star Trek and it’s pretty unlikely that, what? The Federation owned that shack Kirk grew up in, that sweet Corvette or that roadhouse bar? Those items sure looked privately owned. Some spaceships were privately owned. Finally, let’s not forget Star Trek: Generations when Kirk says in the Nexus “This is my house. I sold it years ago.”

Next: The Federation is not true post scarcity economy: famines routinely still exist, transportation lines are vital in moving around goods within the Federation. Transportation is a whole grey area in most post-scarcity economic works, at least the few I’ve read. The Federation might have enough food, but at any time some planet may well be starving or in need of medicine that needs to come from somewhere else.

It seems pretty clear cut that jobs are optional. They explicitly state on many occasions that the Federation is based on a philosophy of self improvement and cultural enrichment, and in any case we sure do run into a lot of “artists” in the Federation. I particularly love those hippies in TOS. The Federation seems a bit like Williamsburg — a lot of artists who don’t need to work. Or maybe more like the UK at the height of its social programs supporting artists. Let a million JK Rowlings bloom. It’s a bit weird, to me, that we’ve never seen people who sit around and literally do nothing, but then why would we? And, of course, we’ve certainly seen more than a few societies that are all chilled out and not doing much (Risa, etc).

Next: The Federation doesn’t use money. This is basically absolute. Kirk says it in Star Trek IV. Picard says it several times. Quark mocks it to Rom. Roddenberry put it down as a hard and fast rule. No theory of Star Trek economics can be real while ignoring this fact. It has to be addressed. It is the basis of all confusion and, honestly, interest in figuring it out at all.

Money still exists, so do banks. Crusher buys fabric at Farpoint. DS9 makes mention of theBank of Bolias, on a Federation planet. Nog loans Jake latinum.

We also know there exists such a thing as the Federation Credit. This presumably causes some confusion since they are routinely referred to like money (Kirk mentions that the Federation has invested 122,200 credits in Spock), and things are purchased for credits (Uhura buys a tribble, Quark occasionally accepts them at his bar).

This would seem to be a giant contradiction to the lack of existence of money. We’ll get to that in a bit.

There is still a ruling class, or classes — it is not perfectly-egalitarian in a communist manner. We have admirals and presidents and governors and colony leaders. There are enlisted personnel in Starfleet and officers. Some are elected, some are appointed. Some Federation members were even hereditary nobilities.

There is still commerce (and even Vulcan commerce), trade, trading vessels, and, we can assume, corporations, in some form (though this may not be 100% definite — Dytallix is mined for the Federation. It isn’t 100% clear it is in the Federation).

Some thought exercises

Let’s do a couple thought exercises.

First: if you eat a meal at Sisko’s Creole Kitchen, do you pay? It seems almost definite that you don’t pay. If you paid, with anything, including Federation Credits, that would be money. You could barter, but it seems if the entire economy was a barter economy, we’d hear it. No, it seems almost certain that you go to eat at Sisko’s, you don’t pay, and Joseph Sisko doesn’t pay for his supplies, and his suppliers probably don’t pay for theirs.

Next: Can everyone have anything? Anything at all? Is the Federation a perfect post scarcity society? The answer seems almost certainly no. If you went to a replicator, or a dealer, or the Utopia Planatia Fleet Yards and asked for 10 million star ships, the answer would be no. More concretely, when the Borg attacked, and during the Dominion War, the Federation suffered from a serious starship shortage.

Next: Imagine there’s some level of welfare benefits in every country, including America. That’s easy. That’s true. Imagine that, as the economy became more efficient and wealthy, the society could afford to give more money in welfare benefits, and chooses to do so. Next, imagine that this kept happening until society could afford to give the equivalent of something like $10 million US dollars at current value to every man, woman and child. And imagine that, over the time that took to happen, society got its shit together on education, health, and the dignity of labor. Imagine if that self-same society frowned upon the conspicuous display of consumption and there was a large amount of societal pressure, though not laws, on people that evolved them into not being obsessed with wealth. Is any of that so crazy? Is it impossible?

I think that is basically what’s going on on Star Trek.

A Theory of Star Trek Economics

I believe the federation is a proto-post scarcity society evolved from democratic capitalism. It is, essentially, European socialist capitalism vastly expanded to the point where no one has to work unless they want to.

It is massively productive and efficient, allowing for the effective decoupling of labor and salary for the vast majority (but not all) of economic activity. The amount of welfare benefits available to all citizens is in excess of the needs of the citizens. Therefore, money is irrelevant to the lives of the citizenry, whether it exists or not. Resources are still accounted for and allocated in some manner, presumably by the amount of energy required to produce them (say Joules). And they are indeed credited to and debited from each citizen’s “account.” However, the average citizen doesn’t even notice it, though the government does, and again, it is not measured in currency units — definitely not Federation Credits. There is some level of scarcity — the Federation cannot manufacture a million starships, for example. This massive accounting is done by the Federation government in the background (witness the authority of the Federation President over planetary power supplies).

Because the welfare benefit is so large, and social pressure is so strong against conspicuous consumption, the average citizen never pays any attention to the amounts allocated to them, because it’s perpetually more than they need. But if they go crazy and try and purchase, say, 10 planets or 100 starships, the system simply says “no.”

Citizens have no financial need work, as their benefits are more than enough to provide a comfortable life, and there is, clearly, universal health care and education. The Federation has clearly taken the plunge to the other side of people’s fears about European socialist capitalism: yes, some people might not work. So What? Good for them. We think most still will.

However, if they so choose they can also get a job. Many people do so for personal enrichment, societal pressure or through a desire to promote social welfare. Are those jobs paid? I would assume that yes, those jobs are “paid,” in the sense that your energy allocation is increased in the system, though, again, your allocation is large enough that you wouldn’t even really notice it. Why do I say this? The big challenge here is how does society get someone to do the menial jobs that cannot be done in an automated manner. Why would anyone? There are really only two options: there is some small, incremental increase in your hypothetical maximum consumption, thus appealing to the subconscious in some primal way, or massive societal pressure has ennobled those jobs in a way that we don’t these days. I opt for the former since it grounds everything in market economics, albeit on a bordering-on-infinitesimal manner, and that stands to reason, since that’s how people talk in Star Trek. They talk about individual fulfillment, buying, selling, etc. No one was ever guilt-tripped into joining Starfleet, save by maybe their family.

There is almost zero mention of central planning. It’s a capitalistic society, it’s benefits are just through the roof. Also, market economics = crowdsourced. That is, it’s not centrally planned. It’s democratic. It’s the only mechanism we know of to allocate resources that isn’t centrally planned. The alternative is that all allocations are done algorithmically through a computer and the economy is completely decoupled from market forces, but that’s still basically central planning, and infinitely more complex than assuming there is still some semblance of market underpinning, much like we stayed on the gold standard for far longer than we needed to and we still have pennies even though we don’t need them. It’s a vestige of the past. It’s the constitutional monarchy.

Either way, presumably, you take whatever job you want, and your benefits allocations are adjusted accordingly. But by and large you just don’t care, because the base welfare allocation is more than enough. Some people might care, some people might still care about wealth, such asCarter Winston. More power to them. They can go try and be “rich” in some non-Federation-issued currency. But most people just don’t care. After all, if you were effectively “wealthy” why would you take a job to become wealthy? It pretty much becomes the least likely reason to take a job.

So, behind the scenes there is a massive internal accounting and calculation going on — the economics still happen. They just aren’t based on a currency unit, and people don’t acquire things based upon a currency value. People just acquire things from replicators, from restaurants such as Sisko’s or coffee shops like Cosimo’s, or, presumably, get larger things from dealerships or (more likely) factories. This could still be called “buying,” as a throwback.

Two points here: first, the accounting is done in energy units, so that there is no need for currency. And why not? Resource allocation is mainly about energy anyhow, doubly so if it’s only robots building most things. And secondly, if you never had money, never saw it, and it didn’t physically exist to measure things, you’d pretty much tell people, like a certain 20th century oceanographer, that you don’t have money in the 24th century, regardless of some automated accounting. This jibes with Federation people knowing what money is — because other societies have it — but saying they don’t use it. Because they don’t.

However, you could still buy and sell things. You could take a thing from a replicator and go to someone else and “buy” something else with it. Why couldn’t you? It’s a free society. It’s essentially barter. Kirk may well have sold his house for a year’s supply of Romulan ale.

Or Federation Credits.

It is tempting to argue here that the massive accounting system uses a unit called the Federation Credit, but i don’t believe that’s the case. If it were, the credit would be too much like money because a) accounting is done in it, b) it is issued by a governing body (like a fiat currency) and c) it is fungible, i.e. you can already buy things with it and if you could buy things with it AND a and b were true, it would pretty much be a currency. This would fly in the face of Roddenberry’s absolute diktat that the Federation has no currency.

I’m gonna make a bold new theory here. Federation Units are “Federation” the same way that American Cheese is American. It is simply descriptive. Currency was invented long before capitalism as a means to disintermediate trades: you wanted my grain, I didn’t want your cows, I wanted farmer Ted’s grapes. Rather than make every trade a 3, 4 or 5 way trade, we made a little certificate we all agreed was worth something to us and us only. This need would still occasionally crop up in the Federation, even without money. I believe the Federation Unit is aprivate currency, developed by third parties to facilitate complex trades or trades outside the Federation. I believe that the Federation Unit is not actually underwritten or issued by the Federation. I think it is more akin to the Calgary Dollar or the Chiemgauer. Or bitcoin. This would solve so many problems. It would make it unequivocally true that the Federation doesn’t use money. It would give people a unit to use as reference when they say things are expensive. It would be a thing citizen’s could acquire, if they wanted to, through barter originally, then allowing them to use them to purchase things (like Tribbles or Holosuites) from people who elected to take them, since taking them is optional (witness Quark’s vacillations on whether he accepts them or not). It would make a nice proxy for talking about investment levels, such as when Kirk said how much the Federation had invested in Spock.

Foreign Reserves

Additionally, I believe that the Federation acts like any current sovereign nation state and holds foreign reserves of currencies of other nations. It’s assumed that not all foreign trade is done through barter. The federation itself probably holds foreign reserves in foreign currency just as China holds US dollars and England keeps a reserve of Euros. Sisko at one point tells Quark he could have charged rent for the bar, but he chose not to. Presumably that would have been paid in latinum. Presumably the Federation would have just held onto it as foreign reserves. All evidence, in fact, points to the fact that the Federation operates as a nation and uses foreign reserves exactly as we do now. The Chinese government holds US Dollars but you don’t here a Chinese person say “we use dollars.” This is a bit confusing by the episode in which the Federation offers 1.5 million Federation Credits for use of the Barzanian wormhole, but it doesn’t have to be contradictory. Federation Credits had value to the Barzanians, so the Federation could simply procure them from the issuer with its foreign reserves of other currencies at market rate.

The Individual Can Have Money

An individual of the Federation can procure latinum by barter for goods, labor or, presumably Federation Credits, if they had them. I assume that there’s probably some black market value for Federation Credits just like any other currency, sovereign issued or not (you can buy aLewes Pound on eBay right now for $7.98). Perhaps its more legitimate and the Units are traded on a commodities exchange. It really doesn’t matter. As a Federation Citizen I can have gold pressed latinum, Federation Credits, Frangs, Darseks, Isiks, Leks, or Quatloos in my wallet. I can have a wallet. I can buy things with Self Sealing Stem Bolts if I want. But none of that is in conflict with the fact that the Federation has no unit of currency, has no money, and my society is predominantly concerned with societal good and self improvement.

Then there’s the matter of Quark’s bar. What’s up with that? He never seems to charge anyone for drinks, but is obsessed with money, and you can buy holosuites in latinum or Federation Credits, and you can bet on the Dabo table with Latinum. At first I thought there was a whole complex thing where Quark doesn’t charge Starfleet personnel because he made the mental calculation it was cheaper to give them drinks for free and keep accepting free rent from Sisko, but then I realized that doesn’t really work because he charges them for the Holosuites and Dabo tables. Then I realized: Quark’s is like any other casino. The drinks are free: they are a loss leader against the higher profits of the Dabo Table and Holosuites.

The Proto Post Scarcity Economy

The thing I love most about this theory is that it seems plausible for our future. Tom Paris said that a new world economy takes shape in the 22nd century. That might be a smidge optimistic but we already have a world economy, in one sense, so the new one could be something only incrementally different from this one. Money went the way of the dinosaur, he said, and Ft. Knox was turned into a museum. Most of us are already off the gold standard, and it’s certainly not inconceivable in another 180 years we don’t use paper money at all, and a single currency has dominated the planet — the Dollar is already close — and it slowly fades into the background.

From there, perhaps a cultural shift takes place as we realize that “everyone in a job” isn’t the same as a full economy, and we start to look for models beyond capitalism that aren’t all communist hoo-ha.

I sort of love that Star Trek forces us to think about a society that has no money but still operates with individual freedom and without central planning. I love that democracy is still in place. I love that people can still buy and sell things. It’s real. It’s a more realistic vision of post-capitalism than I have seen anywhere else. Scarcity still exists to some extant, but society produces more than enough to satisfy everyone’s basic needs. The frustrating thing is that we pretty much do that now, we just don’t allocate properly. And allocating properly cannot be done via central planning.

The only real “out there” requirement in all of this is a governmental layer higher than the nation, and indeed, higher than the planet. This doesn’t seem insane, I suppose, if we were to suddenly find ourselves not alone in the universe. And indeed we already have some measure of international government now. Moreover, the Federation clearly adheres to the “laws made as close to home as possible” routine, since as far as we can tell the Federation president really only has authority over Starfleet, Foreign Relations and power allocation and accounting. Virtually every other law we encounter in the Federation happens at the individual planet or colony level.

It’s interesting to me because these are things we’re going to have to reckon with, I believe, in my lifetime. If robots do all the dirty work, and the US is hugely rich, does every single person really need a job? Are we going to let all of that money pile up in the 0.1% ruling elite, or can it be distributed to everyone? Does wealth being distributed to the people in an equal manner mean communism absolutely? Of course it doesn’t. The US isn’t communist. The UK isn’t communist. Denmark isn’t communist. What happens when the surplus is more than enough?

Chrome Extension Developer FAQ for upcoming changes in Chrome 33 related to hosting extensions - The Chromium Projects

Fri, 14 Feb 2014 01:56:01 -0800

Comments: "Chrome Extension Developer FAQ for upcoming changes in Chrome 33 related to hosting extensions - The Chromium Projects"

URL: https://sites.google.com/a/chromium.org/dev/developers/extensions-deployment-faq

Servo Layout Engine: Parallelizing the Browser // by Paul Rouget

Fri, 14 Feb 2014 01:59:01 -0800

Comments: "Servo Layout Engine: Parallelizing the Browser // by Paul Rouget"

URL: http://paulrouget.com/e/servopres/

During FOSDEM, Josh Matthews talked about Servo. You can find his slides here: joshmatthews.net/fosdemservo

Video:

http://mirrors.dotsrc.org/fosdem/2014/UD2218A/Saturday/Servo_building_a_parallel_web_browser.webm

10 Feb 2014

CppCat, an Ambitious C++ Code Analyzer from Tula

Fri, 14 Feb 2014 03:44:48 -0800

Comments: "CppCat, an Ambitious C++ Code Analyzer from Tula"

URL: http://www.viva64.com/en/b/0232/

This article was originally published (in Russian) at the website siliconrus.com. It is an interview with Evgeniy Ryzhkov by an author and editor at siliconrus.com, Konstantin Panphilov. The article was translated and published at our blog by the editors' permission.

Today we had a conversation with Evgeniy Ryzhkov, CEO of the "Program Verification Systems" company developing software products in the area of software testing systems and static code analysis systems. The company currently offers two products, PVS-Studio and a recently released CppCat. Both are static analyzers for C++ code.

Picture 1 - Evgeniy Ryzhkov

In this interview, we have discussed specifics of debugging, problems of running an international business in Tula, issues of the pricing policy in their area, and cats.

Panphilov: Hello! Tell us please about your business in general - what are you doing, why, and who else takes part in it?

Ryzhkov: I work at the "Program Verification Systems" company which we founded together with my associate and business partner Andrey Karpov. We are situated in a provincial city of Tula which is famous for its arms industry, samovars, and traditional pryanik gingerbread. Since all these fields are already occupied, we've chosen an area which is not very usual for Russia. We develop programmers' tools. To be more exact, we create static analysis tools for C++ code.

Currently our team consists of six people, but, despite the small size, we manage to create pretty cool and complex stuff. I mean first of all the PVS-Studio analyzer and our new product CppCat, which is a greatly revised version of PVS-Studio with a different pricing and licensing policy.

Picture 2 - Andrey Karpov, cofounder

Ours is a most typical software company: we create software products and sell them to customers. In this sense, we are not quite "in the trend": we don't design mobile applications with millions of installs, or possess a website with hundreds of thousands of unique visitors. We run a, so to say, classic business which seems to be pretty rare nowadays.

All the work on promoting, advertising and selling our primary product, PVS-Studio, we do on our own - not to mention that we develop it on our own too, of course.

KP: What is the difference between CppCat and PVS-Studio?

ER: First of all, PVS-Studio is a pretty expensive and strongly specialized product. The cheapest license - the one for a team of up to 9 developers - currently costs €5250. Besides, this is a product with a long history (it was introduced in the beginning of 2011) and it has by now accumulated a lot of features which not all developers need. That's why, in the autumn 2013, we decided to redesign our product as if we were developing it from scratch.

It took us 3.5 months to develop and release such new product (based on the existing one, of course). We got it clear of all the stuff that we found inconvenient and complicated, and worked out a new pricing policy. We offer single-user licenses for the product at $250. The tool was launched soon after the New Year. And you know, we were much astonished ourselves at having managed it all in just 3.5 months. By the way, I have a nice story to tell you about how its name was conceived.

Picture 3 - A recreation room in the office. It's empty - right, no time for relaxation!

You know, inventing names is a very difficult task. Those who have ever had to do it know what I mean, and those who haven't perhaps won't believe me. Anyway, we needed a name that met a few requirements: 1) it should emphasize that the tool was designed for C++ programmers; 2) it should be vacant (that's why various names like cppcheck and codescan wouldn't do); 3) it should be amusing; 4) it could be illustrated by a nice logo.

Well, there is a general belief that programmers are fond of cats, and we used it as an inspiration source. If you visit the cppcat.com website, you'll see our logo hanging there all about (just scroll down a bit). So, it took us two weeks out of 3.5 months to make up the name and about a month to design the logos and test them on focus groups.

Picture 4 - This is how CppCat was being born

KP: You're offering PVS-Studio at such a high price - are there really any customers? Is it that Visual Studio's standard debugger is so awfully bad?

ER: Well, 85% of all the purchases are made from the US and Europe, while Russia makes only 10% of our sales. Static analysis tools are commonly pretty expensive throughout the world. You can't have thousands of customers, so you have to set high prices. It is not that Visual Studio's debugger is bad. Let me explain the way a customer makes a decision about using our or any other static analysis tool. He runs the trial version and studies the bugs it has found. If he finds really interesting and worthy issues, he estimates how much time it would have taken a programmer to find and fix those bugs manually. Then he estimates the cost of an hour of work. If he finds that the tool can help save time on bug search, he will buy it. And here we face a few issues with sales, and I'd like to tell you about them.

First, when studying and evaluating a tool, the programmer may fail to find interesting messages among the issues detected by the analyzer. It's not because the tool has not found errors. It's that when you get, say, 300 messages, you will be most attentive when studying the first 10. You will be less attentive when studying the next 20. And then you will stop right there. So it is very important that the most interesting messages should be shown in the beginning of the list.

Second, the cost of one hour of work I've already mentioned. If it is low (for example, in a project where students are engaged), the customer certainly doesn't need a static analyzer. But when you deal with highly-paid American programmers, who earn $50 per hour, a tool that can help to save at least some time would be useful.

Picture 5- One more photo of the office: a shaman's drum - a gift from Intel. A crucial part of a debugging ritual.

Third, static analysis tools allow users to detect bugs in the code right in the middle of writing it, before it gets to testers who, in turn, will need some time for testing it and forming a bug report - and all that bureaucratic stuff accompanying the whole procedure. No need to say when it slips over to the users.

However, those who are not familiar with the static analysis technology may mistakenly think of it as a wonderful miracle - so why are there still bugs in programs when we have such a powerful weapon? Of course, it's not all that cool and neat. It's just an ordinary technology with its own advantages and disadvantages. Some find it useful, and some others do not.

By the way, the specific character of our target audience sets some interesting limitations on the ways of product promotion.

KP: Yes, that's interesting too, but before that I'd like you to tell us how you managed to launch your business so quickly.

ER: Well, I'm afraid our start was not so quick. We founded our company in 2008 - it was then that we started working with static code analysis technologies. Our first product named Viva64 was designed for finding issues of software migration to 64-bit systems. By the way, detection of the issues in migration to 64-bit systems was my graduation thesis subject, so our product was evolving along with our scientific work. And my coworker Andrey Karpov was working on his own thesis on parallel programming. Naturally, we had jobs as programmers but wished to try something of our own. That was how Viva64 appeared. Unfortunately, we were not that experienced in selling our products as we are now. For a long time, the tool didn't bring us enough profit to found a self-managed business and drop working as outsourcers to earn our living.

Picture 6 - It is usually recommended to nail a horseshoe to the computer's chassis... Why is it on the wall then?

Besides, that tool appeared to be ahead of its time. We counted on a swift software migration to 64-bit systems to happen soon, but this process has appeared to be slowly dragging and it is still not over. We also found it a difficult task to tell programmers about 64-bit errors - they just wouldn't believe us. So our first step was a flop - just as it ought to. By 2011 we developed a general-purpose code analyzer PVS-Studio which happened to be more familiar to people, and we saw a sales boost as a result.

There was also some luck to it. Not in the sense that we didn't do anything and then it all just suddenly came to us. No, we did work hard - and one morning I checked my e-mail and saw a purchase-report from id Software by a user named John Carmack. I checked it several times and made sure it was not a fraud, and then I was very pleasantly amused. But it was when Carmack praised PVS-Studio from the stage at QuakeCon 2011 that I and all in our small team felt truly happy with what we were doing. And I want to clarify here why I am telling you about it. Not only to boast - though we felt proud indeed. It's just another thing which is very important: if you want to be mentioned and praised by some tough fellow of yours, you'll need to work quite a lot and that work may seem unnoticeable at first.

And what such a quick launch of CppCat is concerned, we managed it because we had worked on the product concept for a long time by then, and we knew all the drawbacks of PVS-Studio that we needed to fix.

KP: Wow, that's really a cool story about Carmack.

ER: By the way, we are currently selling both products, PVS-Studio and CppCat, so they live in parallel and help each other.

Picture 7 - Carmack is glad for Russian software products

KP: You're teasing Chromium, MTA, Doom 3, etc. at your site - doesn't it prove that bugs are infinite and it's impossible to find them all?

ER: Konstantin, you've touched upon my favorite subject - promotion of our products. Our target audience are tough and experienced programmers. Decision about purchasing products is made by a team leader (in large companies) or development director (in small teams). You may flood the whole Internet with bright banners "Our product is the best, buy it and forget all trouble!" — they won't fall for it.

Moreover, many programmers either use AdBlock or have worked out a habit not to notice banners and to show criticism toward advertising reviews. But a few years ago we found - or rather accidentally discovered - the most efficient way of promotion. We take an open source project - the more famous, the better - and check it. We pick a dozen of bugs from the list and discuss them in the article. Andrey is especially good at writing such articles - he has really become a master in that. What is the idea of these articles? We DO NOT mean to say that all those projects are written by novices and bunglers and make fun of them. We mean to say, "Look, even the tough Chromium developers sometimes make mistakes! What about you - do you think you are perfect? Get the tool and check your code."

People do like these articles, which inspire them to visit our site and download a trial version. The idea is simple. Programs are written by humans. And humans tend to make mistakes, you know that. Even the most qualified ones. The whole stuff is always in motion: projects are evolving, developers are writing new code and modifying old one. So mistakes are inevitable. The earlier they are diagnosed (either by our tools or by others), the cheaper it is to fix them.

Picture 8 - A program's screenshot (for the sake of formality)

By the way, we are very lucky with our target audience (i.e. programmers). I've become convinced of it during the years of work. Out of 100 reports, only a few are inadequate; most often, we get really reasonable bug-reports, stack-dumps, and even recommendations (which are often quite useful) on how to fix bugs in our tools. I just can't but compare our audience to that of other products (as told by our friends) and feel happy that we are so lucky with our users.

KP: By the way, what running CppCat on CppCat's code will result in?

ER: It's one of the most frequently asked questions. We regularly run our tools on themselves. They usually reveal bugs at the moment of writing the code, when analysis is invoked right after compilation. And here's where they all are caught. That's why you will never see us publishing an article with a title like, "We found 20 bugs in our analyzer". It's not because we don't make mistakes - we do, and quite a lot. But we get them fixed right away, so they don't even get into the bug tracker.

KP: That's great. Speaking of money, does your startup pay off? How many customers do you have, what's the turnover?

ER: From the viewpoint of business, we have always tried to be profitable as we started and maintained the company with our own money. To be more exact, we have never been at a loss because we just can't afford it. This is partly the reason why we are still a small startup, though we want to grow, of course. Speaking of the turnover, we saw 40 purchases of PVS-Studio during 2013. Not all of them were first-time purchases (many of our customers renew licenses at 80% of the original price for one year). Among those purchases were both ordinary, i.e. cheaper, licenses and more expensive ones (the Site License for a team of many developers); but there were also cheaper deals when we carried out an experiment with the pricing policy. So these figures make a good estimate of our business. Not great, of course, and anyone would crave for more. But this is how it is for now.

What CppCat is concerned, it's a bit too early to give any figures as the product was only launched in January 2014, and, as I already said, you don't expect millions of installs in our area.

KP: Frankly, did you really start it all on your own, without any investments?

ER: Yes. However, when we only started and Russia didn't see such an investment boom like now, we communicated with some investment funds. They all answered, "Welcome when you have an established business". When we got an established business, they answered, "Welcome when you reach SUCH-AND-SUCH income level". And when our income reached that level, we did not need them anymore. So we haven't contacted any investors since then. This is our experience of trying to establish relations with investors. I suspect there are many cases when it may be mutually beneficial, but not for us with our narrow target area.

KP: Everything looks bright and neat, but the world is not that way. There must be something you are not satisfied with. What is it?

ER: That's true, there is always something you don't like, along with positive aspects. We don't like failing to reach a larger audience of USA and Europe developers, or a larger number of article reads and trial-version downloads. Although we work and try hard, results are still not very promising. Sometimes I take a look at a diagram of Google Analytics for our fellow colleagues and can't help wondering, "How do they manage to attract so many thousands of visitors? What am I doing wrong?" I can't but envy, "I wish I had traffic like this!"

And please no comments like, "Sure you won't get over to the USA and Europe from Tula!"

KP: Why, you are right. It doesn't matter where you are situated. People get to your website, not your office. So it's OK.

ER: Yes, but we just can't understand where, for instance, USA programmers hang about. That is, people give us addresses of some websites in comments, we pick the first five and try to carry out some promotional activity there. But it seems not enough. So, we are currently not very good at internet-marketing. I mean you can't just create a fresh account at a forum and write there, "Hello guys, welcome here - analyzers for download".

KP: Well, I'm sure you'll make it one day.

ER: No doubts :-)

KP: Thank you very much!

Watch This Man Confront CEO of Mt. Gox Over Missing Bitcoins

Fri, 14 Feb 2014 04:08:48 -0800

Comments: "Watch This Man Confront CEO of Mt. Gox Over Missing Bitcoins"

URL: http://www.coindesk.com/watch-man-confront-ceo-mt-gox-missing-bitcoins/

Something about bitcoin inspires people to jump on airplanes and fly to the other side of the world to protect their wealth. Maybe that’s part of bitcoin’s ‘inherent value’.

Kolin Burges was a software developer in London until the end of last year, when his vast Litecoin holdings suddenly went lunar. Concerned at the prospect of a Chinese New Year crash he moved much of his wealth back to Japan and Mt. Gox, only to find it locked in there around the same week.

Inspired by last week’s three-day sit-in protest by Australia’s pseudonymous CoinSearcher, Burges bought a last-minute ticket (at the cost of £850) and flew 12 hours from London to Tokyo to see for himself what was happening. He said:

“They haven’t answered anybody’s support requests, they seem to have just taken people’s money – they’ve taken my money.”

After arriving in Japan on Wednesday, Bruges tried to schedule an appointment with the company’s executives, but to no avail. On Friday morning he set up camp outside the Mt. Gox offices in Tokyo’s Shibuya district – despite the freezing temperatures.

At 9.15am he confronted Mt. Gox’s CEO Mark Karpeles outside the exchange’s front door and demanded to withdraw his coins, but the response was: “We can’t right now.” Burges’ next question: “Do you still have everyone’s bitcoins?” didn’t get a response.

Burges admitted that a doorstop challenge would probably be futile, but said the wealth that he has stored in Mt. Gox’s servers is definitely worth it. ”If I lose that, my lifestyle will change. I just didn’t want to lose that amount of money without a fight,” he said.

Burges says he’ll stay in Tokyo until next week, camping out at Mt. Gox’s offices again if necessary.

When he announced his travel and protest plans on Reddit, the response was mixed. While some supported his effort, others said Mt. Gox had already given a credible explanation for the withdrawal problems and should be allowed time to fix them.

Japan reddit Tokyo

A new take on an old industry., Drop the ego and prove "I could make that so much better"

Fri, 14 Feb 2014 04:17:48 -0800

Comments: "A new take on an old industry., Drop the ego and prove "I could make that so much better""

URL: http://blog.lawnstarter.com/post/76623724133/drop-the-ego-and-prove-i-could-make-that-so-much

I mow lawns for a living, what do you do?

Well, if you’re working on a startup it probably sounds a helluva lot cooler than that. Whether it was my ego, or insecurity I came very close to never even starting LawnStarter simply due to the industry it was in.

After dropping out I was ready to work on something that sounded revolutionary. What I didn’t expect was to stumble upon an industry that needed a full system overhaul.

In the very beginning of working on LawnStarter I almost always told myself and other people that this was a segue to make some cash until my “real” startup began. I mean, come on, clearly I wasn’t going to make some lawn care company!

As weeks turned into months and progress became rather substantial I started to realize the only thing holding us back was my ego. I felt uncomfortable telling friends what we were doing (in my defense they gave me tons of shit), I undersold the opportunity, and worst of all I was honestly a bit embarrassed.

Looking back, it couldn’t be clearer that I had fallen into a mindset that is all too familiar in our community.

Why the hell not do a startup to try to change the lawn care industry? It’s a massive industry that could really use some help!

For some reason, I bought into this premium that our community puts on companies that produce gimmicky technology and raise massive rounds without having any real revenue streams. I got so caught up in this mentality that I was literally embarrassed to be working on a company that was making money in a less than sexy industry. Seems ass-backwards to me, and if I had simply ignored my ego for 10 minutes, I would have recognized how ridiculous I was for being embarrassed.

Over the past 5 months I’ve realized just how many opportunities there are to make really shitty industries better. People are willing to pay for a better experience, think Homejoy and Moveline (hopefully one day somebody puts Lawn Starter in that group). Even with money to be made and awesome companies to be built it seems to me like our egos and insecurities are holding us and progress back in dozens of not so attractive industries. Why not ride the wave of awesome customer experience and try to put your stamp on an industry that hasn’t changed in decades?

While everyone is trying to make the next facebook or snapchat - which is cool, I use both of these a lot - you could be trying to reinventing an industry nobody else wants to touch.

We all have a list of things or industries that “we could make so much better”. For me it was lawn care since high school. I’m finally putting my money where my mouth was and I look forward to seeing if I was right.

Me, I’ll be mowing lawns in Fairfax and dozens of other places. Where will you be?

-Steve

De La Soul to Make Entire Catalog Available for Free for 25 Hours | Music News | Rolling Stone

Fri, 14 Feb 2014 05:32:23 -0800

Comments: "De La Soul to Make Entire Catalog Available for Free for 25 Hours | Music News | Rolling Stone"

URL: http://www.rollingstone.com/music/news/de-la-soul-to-make-entire-catalog-available-for-free-20140213

De La Soul

Robbie Jeffers

February 13, 2014 4:35 PM ET

In honor of next month's 25th anniversary of their debut album 3 Feet High and Rising, De La Soul are making their entire catalog available for free download for 25 hours on the group's website. The download bonanza will begin on Friday, February 14th at 11 a.m. EST until Saturday at noon.

Check out 10 Unseen Photos Included in De La Soul's Album Downloads

"It's about allowing our fans who have been looking and trying to get a hold of our music to have access to it," De La Soul member Posdnuos tells Rolling Stone. "It's been too long where our fans haven't had access to everything. This is our way of showing them how much we love them."

The same things that made 3 Feet High and other De La albums so influential — its creative, if not fully licensed, use of a myriad of samples — has also prevented the group's work from appearing on many digital platforms. "It's been a trying journey," admits Posdnuos. "We've been blessed to be in the Library of Congress, but we can't even have our music on iTunes. We've been working very hard to get that solved." The rapper points to frequent personnel changes at record labels and hazy language in early contracts that have led to long delays in properly clearing the group's catalog.

The release of the group's catalog is the first of numerous upcoming projects. In a few weeks, they'll post new songs to their site, with You're Welcome, their first album since 2004's The Grind Date, expected to be released before summer. Next month will also see the release of Preemium Soul on the Rocks, a six-song EP with three beats each from DJ Premier and Pete Rock. The group is also planning a visit to Detroit to work on an unreleased beat from J Dilla, the prolific producer who passed away in 2006. "Dilla was the Tupac of producers," says Posdnuos. "He has so many unreleased things that no one has heard. His family knows how vital and important an ingredient his music was to our work."

Where does 3 Feet High and Rising rank on Rolling Stone's 500 Greatest Albums of All Time?

Asked about the status of You're Welcome, an album originally scheduled for release last year, Posdnuos says that it's "coming along amazingly," but points to self-criticism in its delay. "We have tons of music, but we're our own worst critics," admits the rapper. "Certain groups have too many 'yes men.' In our group, we have too many 'no men.' When we look back on some of the stuff we have, we're like, 'Yo, we need to just put this out.' The album is still called You're Welcome, but we also have this whole other album that we're working on that…Wooo, I wish I could talk about it."

Twenty-five years into their career, the group is ready, if somewhat cautiously, to adopt the more-is-more release schedule of its younger peers. "We're just getting in the mode of constantly giving people new music," says Posdnuos. "I'll be the first to say that not everyone can do it. You can put out a new mixtape every week, but it can dilute what you're putting out because you haven't had enough time to see what's going on with your life to write something from a different angle. With us, we've sat a long time without releasing an album. It's high time we start releasing a bunch of stuff because it's there."

To read the new issue of Rolling Stone online, plus the entire RS archive: Click Here

Mark Shuttleworth » Blog Archive » Losing graciously

Fri, 14 Feb 2014 05:32:23 -0800

Comments: "Mark Shuttleworth » Blog Archive » Losing graciously"

URL: http://www.markshuttleworth.com/archives/1316

Friday, February 14th, 2014

With Bdale Garbee’s casting vote this week, the Debian technical committee finally settled the question of init for both Debian and Ubuntu in favour of systemd.

I’d like to thank the committee for their thoughtful debate under pressure in the fishbowl; it set a high bar for analysis and experience-driven decision making since most members of the committee clearly took time to familiarise themselves with both options. I know the many people who work on Upstart appreciated the high praise for its code quality, rigorous testing and clarity of purpose expressed even by members who voted against it; from my perspective, it has been a pleasure to support the efforts of people who want to create truly great free software, and do it properly. Upstart has served Ubuntu extremely well – it gave us a great competitive advantage at a time when things became very dynamic in the kernel, it’s been very stable (it is after all the init used in both Ubuntu and RHEL 6 and has set a high standard for Canonical-lead software quality of which I am proud.

Nevertheless, the decision is for systemd, and given that Ubuntu is quite centrally a member of the Debian family, that’s a decision we support. I will ask members of the Ubuntu community to help to implement this decision efficiently, bringing systemd into both Debian and Ubuntu safely and expeditiously. It will no doubt take time to achieve the stability and coverage that we enjoy today and in 14.04 LTS with Upstart, but I will ask the Ubuntu tech board (many of whom do not work for Canonical) to review the position and map out appropriate transition plans. We’ll certainly complete work to make the new logind work without systemd as pid 1. I expect they will want to bring systemd into Ubuntu as an option for developers as soon as it is reliably available in Debian, and as our default as soon as it offers a credible quality of service to match the existing init.

Technologies of choice evolve, and our platform evolves both to lead (today our focus is on the cloud and on mobile, and we are quite clearly leading GNU/Linux on both fronts) and to embrace change imposed elsewhere. Init is contentious because it is required for both developers and system administrators to understand its quirks and capabilities. No wonder this was a difficult debate, the consequences for hundreds of thousands of people are very high. From my perspective the fact that good people were clearly split suggests that either option would work perfectly well. I trust the new stewards of pid 1 will take that responsibility as seriously as the Upstart team has done, and be as pleasant to work with. And… onward.

This entry was posted on Friday, February 14th, 2014 at 1:01 pm and is filed under ubuntu. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Comments are filtered through Akismet for spam detection. Please follow the Ubuntu Code of Conduct - your opinions are welcome but please keep them polite and constructive.

@benguild | Apple's Remote Desktop software is secretly bundled for free with every Mac!

Fri, 14 Feb 2014 05:44:23 -0800

Comments: "@benguild | Apple's Remote Desktop software is secretly bundled for free with every Mac!"

URL: http://benguild.com/2011/02/14/apples-remote-desktop-software-is-secretly-bundled-for/

Apple's Remote Desktop software is secretly bundled for free with every Mac!

14 Feb 2011 | Tags: #mac, #software

I’d always used Apple’s built-in VNC server for remote system administration via Chicken of the VNC, but today I learned there is a better option based on what Apple includes with their “sold separately” Remote Desktop software. Ironically, it’s essentially the same product (including Clipboard sharing and Image scaling) … just less bloated and hidden for free deep inside most Macintosh computers!

Go to this folder on your computer: /System/Library/CoreServices/
(You can get there quickly by pressing Command+Shift+G in the Finder and pasting that string in as the folder)

In that folder, you’ll see an app called “Screen Sharing” … create a shortcut in your Applications folder by holding Command+Option and dragging it there. Do not move the actual file, make sure you hold those keys down and it will “link” it as a duplicate alias.

This App works just like Remote Desktop, and can even save passwords in your system’s Keychain. Just enter a hostname and remember to enable “Screen Sharing” on the destination machine before starting!

I’m not sure why Apple didn’t just put this in the Utilities folder? Enjoy!

Linux Design Tools: High-end Design on a Low-end Budget

Fri, 14 Feb 2014 05:47:23 -0800

Comments: "Linux Design Tools: High-end Design on a Low-end Budget"

URL: http://www.sitepoint.com/linux-design-tools/

While the world’s best commercial graphic applications come with packed with features, they also come with a price tag many find hard to justify.

Though there are plenty of less expensive alternatives, the simple truth is: It’s hard to get cheaper than free.

Today we’re going to look some of the free, open source graphic apps available, and see if they are a viable replacement.

If you are currently unfamiliar with the abundance of free open source graphic apps now available, you may well be missing out.

The best open source graphic applications on this list are comparable in quality to their leading commercial equivalents.

Don’t be put off by the word ‘free’ either. This is a case where ‘free’ definitely doesn’t mean low quality.

I know what you’re thinking: “If I use free software, I’ll have far too much extra money!’

Relax. You can always donate to the hard-working developers behind these apps, and help them make them even better.

So, let’s get to the list, and see if open source can genuinely compete with their expensive commercial counterparts.

Sidenote: Although most of these applications were originally developed for Linux, they often have Windows and Mac versions.

1. Could You Use GIMP to Replace Photoshop?

Often the first application that comes to mind when you are thinking about free, open source graphics is GIMP (Gnu Image Manipulation Program).

While GIMP doesn’t have all the niche features of Photoshop, it certainly offers a lot. In fact, there are a good many professional designers who use GIMP and judging by their output, the results are stunning.

So, assuming you have some design skills, the limitations of the applications you use is secondary.

Image Credit: gimp.org

What Can GIMP Do?

GIMP is a raster graphics application that is useful for image editing and retouching.

You can also use it to create raster images from scratch, as well as simple animations.

Cage Transform in action

It supports layers and channels, has an advanced gradients function, includes paths and quick masks. There are rotation and transformation tools, as well as tools for scaling, cropping, resizing, file format conversion, and more.

GIMP can also be used for both Web and print design.

It even has a few tools that don’t have direct analogs in Photoshop — for instance, the Cage Transform tool that allows to warp just parts of objects.

Where Does GIMP Fall Short?

The main areas where GIMP falls behind are interface and feature list.

Any designer raised on the Photoshop interface, might not always find menus options where they predict them to be.

It’s true, the GIMP interface is different from Photoshop, and there will be an inevitable learning curve.

However, if the UI differences are a show-stopper for you, there’s always Gimpshop. Gimpshop is GIMP retro-fitted a very Photoshop-like interface.

Some may argue that reproducing Photoshop is not something to aspire to. You be the judge.

In terms of features, it’s hard to pinpoint exactly which Photoshop features are missing, as most missing functionality can be added via GIMP’s rich plugins ecosystem.

In theory, this is a better model, as you won’t have lots of unused functionality obscuring the features you need.

As an example, GIMP doesn’t ship with RAW image processing abilities, but this can be added via the UFRaw plugin.

GIMP does lack the granular text options of Photoshop, along with its adjustment layers. CMYK support is also somewhat limited by default, though additional plugins can plug this gap if required.

While the lack of these features may well be a show-stopper for some, I suspect many designers would not even notice their absence.

Interoperability is often cited as the biggest problem for dedicated GIMP users. Clients frequently want to provide you with PSD files, and while GIMP supports PSD in theory, the practical results aren’t always ideal.

This file format issue is often cited as a key hurdle for new adopters, and is likely the major reason GIMP is not as widely-adopted as it might be.

Price Comparison Between GIMP and Photoshop

While there are some obvious caveats to adopting GIMP, these hurdles often become less important when you compare costs. As I mentioned, GIMP is free – it’s hard to get cheaper than that.

On the other hand, Photoshop is no small investment. Currently, you can get Photoshop CS6 as a standalone application for $699-999 or for $1,299-2,559, bundled with other Creative Suite 6 products.

With Adobe’s new rent-only policy – i.e. their Creative Cloud the situation is similar.

If you are lucky to catch a promotion, you can get Photoshop for under $10 a month – otherwise the regular price is $19.99-29.99 a month.

If you decide to go for the complete Creative Cloud, the price goes up to $49.99 a month (on a yearly contract), which is $600 a year.

That is a lot money for most of us — especially if you are not using the software every day!

2. Could you use Inkscape to replace Illustrator?

Adobe Illustrator is another leading commercial vector graphic application that has a viable free open source competitor in the form of Inkscape.

SCREENSHOT, Image Credit: inkscape.org

What Does Inkscape Bring to the Table?

Inkscape uses the W3C standard Scalable Vector Graphics (SVG) file format as it’s default. As you might know, SVG is a format that is becoming more relevant to web developers with every passing day.

It can also work with other popular vector formats, such as .ai, .eps, or .pdf, while most other vector graphics programs can handle its SVG files.

Inkscape ships with markers, clones, alpha blending, path operations, bitmap tracing, and other standard SVG features.

For a full list of features, check this.

Perhaps the most interesting section here is the list of features Inkscape has that are missing in Illustrator. According to inkscape.org, the following Inkscape abilities are lacking in Illustrator:

Edit SVG source directly
Clones, tiled clones, edit clones on canvas
Keys to move/rotate/scale by screen pixels
Shapes as objects
Edit gradients with handles on-canvas
Edit nodes with keyboard
One-click paint bucket fill

While this list may need to be updated, I think it’s useful enough to include regardless.

What Illustrator Functionality Is Missing in Inkscape?

If you decided to switch to Inkscape, you should also be aware of its limitations.

Firstly, file interoperability can be an issue, especially with older .ai files.

Inkscape.org also publishes a list of Illustrator features that are currently missing in Inkscape:

These include:

Gradient meshes
Multiple strokes and fills for one object
Color management for print (ICCProfiles, etc.)
PMS color management
Natively work with graphs based ondata
Free transform and perspective transform (available via extension)
Blends (available via extension)

Again, this list may be slightly outdated, but gives you a sense of what you might be losing if you decide to throw your lot in with Inkscape.

Now, I wouldn’t class myself as a ‘power-user’ of either Illustrator or Inkscape. But as an occasional user I can attest that my skills (or the lack of them) are certainly a much more serious barrier, than any software limitations.

From my point of view, there is no critical feature or tool that Inkscape lacks. For the majority of projects — even demanding professional use — I believe Inkscape is a very feasible alternative.

Price Comparison Between Inkscape and Adobe Illustrator

The price comparison between Inkscape and Adobe Illustrator is very similar to the earlier Photoshop comparision – free vs $599 for Illustrator CS6, or $19.99 a month for the standalone Creative Cloud version, or $49.99 as part of CC bundle.

Switching to Inkscape could save you $240-600 a year.

3. Could You Use Scribus Instead of InDesign?

If you are into desktop publishing, then you might be interested in Scribus as an alternative to Quark Xpress, Adobe InDesign, or Microsoft Publisher.

Scribus is not an exact match of either program, so you shouldn’t expect it to have all the features you know from these programs.

Scribus: SCREENSHOT, Image Credit:
http://www.tomsguide.com/us/download/Scribus,0302-6428-150189.html

What can Scribus Do?

In simple terms, Scribus offers solid “CMYK color, separations, Spot Colors, ICC color management and versatile PDF creation.” It also can work with most common raster and vector image formats, as well as many text formats.

Features that Scribus Lacks

Similarly to GIMP and Inkscape, the Scribus interface is very different to that of InDesign. Until you become accustomed to it, it’s likely to take you some time to perform even the simplest tasks.

Now, I am no Scribus expert, and to be frank, my encounters with InDesign are pretty sketchy, but experienced desktop publishers tell me they couldn’t use Scribus for critical publishing tasks.

Aside from this, many common functions you will find in InDesign, are currently absent from Scribus. For instance, in Scribus you can’t even dock, move or hide individual palettes.

But what’s the main buzz-killer?

Currently Scribus can’t open InDesign’s default format. While Scribus’s PDF export facility covers most print situations, the lack of ‘INDD’ support will be a roadblock for some.

Again the savings are significant – free versus $19.99/49.99 a month for InDesign.

However, since Scribus isn’t anywhere near as competent a replacement to InDesign, as GIMP and Inkscape are to Photoshop and Illustrator respectively, I think the price comparison is far less relevant.

This comparison is truly an ‘apples to oranges’ situation.

4. Other High Quality Linux Graphics Tools

The list of open source useful applications that are compete competently with commercial equivalents is too long to cover comprehensively here.

However special mention has to go to the amazingly powerful Blender 3D, used for 3D design, video editing and special effects.

SCREENSHOT, Image Credit: blender.org

While arguably not be a complete substitute for Premiere, 3DS Max or Maya, Blender has become a rich and powerful platform — evidenced by Blender-powered, short films and games such as Big Buck Bunny, Sintel and Tears of Steel.

Big Buck Bunny: Created entirely in Blender.

Krita and MyPaint are two other graphic programs in addition to GIMP you can use. Paint.net for Windows is also a very viable Photoshop alternative.

Xara for vector graphics and Pencil for cartoons are also good. When you add FontForge (for fonts creation), Dia (for diagrams), or Draw from the LibreOffice package, you’ll see that you can do pretty well without a spending metaphorical dime.

These applications are just a sample of what’s available in the Linux world. Some are included in Linux distributions, while others need to be downloaded and installed separately.

If you don’t have the time to do it, you can get a graphics-focused Linux distribution, such as DreamStudio.

Dreamstudio includes all the applications listed above, plus some other graphics, sound, and video applications, like Darktable. Similar in function to Adobe Lightroom, Darktable helps you manage your digital negatives, view them through a zoomable light-table, as well as develop and enhance raw images .

Ready for Primetime?

Whether these applications can become your default editors is a question only you can answer. However, even if you decide they aren’t quite ready to completely replace your commercial suite yet, they may still may have a useful role in your toolkit.

Frequently I find it more efficient to use an open source application for many of the simple graphics tasks you might perform in a typical day — cropping, scaling and simple color tweaks.

Faster start-up times mean I’m often exporting from my open source app before it’s heavier commercial kin has had time to load.

And that can make a big difference to your productivity in a busy day.

As with many other decision in life, the choice of open source versus proprietary software isn’t only about the money.

Ada is a fulltime freelancer. She enjoys design and writing and tries to keep pace with the all the latest and greatest new developments in technology

GOG.com

Fri, 14 Feb 2014 06:03:23 -0800

Comments: "GOG.com"

Multiple Simultaneous Ajax Requests (with one callback) in jQuery | CSS-Tricks

Fri, 14 Feb 2014 06:30:38 -0800

Comments: "Multiple Simultaneous Ajax Requests (with one callback) in jQuery | CSS-Tricks"

URL: http://css-tricks.com/multiple-simultaneous-ajax-requests-one-callback-jquery/

Published February 13, 2014 by Chris Coyier

Let's say there is a feature on your website that only gets used 5% of the time. That feature requires some HTML, CSS, and JavaScript to work. So you decide that instead of having that HTML, CSS, and JavaScript on the page directly, you're going to Ajax that stuff in when the feature is about to be used.

We'll need to make three Ajax requests. Since we don't want to show anything to the user until the feature is ready to go (plus they all kinda rely on each other to work right) we need to wait for all three of them to be complete before proceeding.

What's the best way to do that?

Ajax calls in jQuery provide callbacks:

$.ajax({
 statusCode: {
 url: "/feature",
 success: function() {
 // Ajax success
 }
 }
});

Or the "Deferred" way, this time using a shorthand $.get() method:

$.get("/feature/").done(function() {
 // Ajax success
});

But we have three Ajax requests we're needing to perform, and we want to wait for all three of them to finish before doing anything, so it could get pretty gnarly in callback land:

// Get the HTML
$.get("/feature/", function(html) {
 // Get the CSS
 $.get("/assets/feature.css", function(css) {
 
 // Get the JavaScript
 $.getScript("/assets/feature.js", function() {
 // All is ready now, so...
 // Add CSS to page
 $("<style />").html(css).appendTo("head");
 // Add HTML to page
 $("body").append(html);
 });
 });
});

This successfully waits until everything is ready before adding anything to the page. So by the time the user sees anything, it's good to go. Perhaps that makes some of you feel nauseated, but I've done things that way before. At least it makes sense and works. The problem? It's slow.

One request ... wait to be done ... another request ... wait to be done ... another request ... wait to be done ... go.

It would be faster if we could do:

All three requests in parallel ... wait for all three to be done ... go.

We can use a bit of Deferred / Promises action to help here. I'm sure this is some JavaScript 101 stuff to some of you but this kind of thing eluded me for a long time and more complex Promises stuff still does.

In our simple use case, we can use jQuery's $.when() method, which takes a list of these "Deferred" objects (All jQuery Ajax methods return Deferred objects) and then provides a single callback.

$.when(
 // Deferred object (probably Ajax request),
 // Deferred object (probably Ajax request),
 // Deferred object (probably Ajax request)
}.then(function() {
 // All have been resolved (or rejected), do your thing
});

So our callback-hell can be rewritten like:

$.when(
 // Get the HTML
 $.get("/feature/", function(html) {
 globalStore.html = html;
 }),
 // Get the CSS
 $.get("/assets/feature.css", function(css) {
 globalStore.css = css;
 }),
 // Get the JS
 $.getScript("/assets/feature.js")
).then(function() {
 // All is ready now, so...
 // Add CSS to page
 $("<style />").html(globalStore.css).appendTo("head");
 // Add HTML to page
 $("body").append(globalStore.html);
});

Another use case: mustard cutting

My use-case example above is a 5% feature. Keep the page lighter for the 95% of users who don't use the feature, and have it be a relatively quick add-on for those that do.

Another situation might be a cut-the-mustard situation where you add in additional features or content to a page in certain situations, as you decide. Perhaps do a matchMedia test on some media queries and determine the device's screen and capabilities are such that you're going to include some extra modules. Cool, do it up with some parallel Ajax calls!

10 French startups disrupting the way you experience fashion online

Fri, 14 Feb 2014 06:40:38 -0800

Comments: "10 French startups disrupting the way you experience fashion online"

URL: http://www.rudebaguette.com/2014/02/14/10-french-startups-fashion-marketplace/

It may sound like a stretch to say that French people are the reference for fashion. But as we say, “il n’y a pas de fumée sans feu” (there’s no smoke without fire).

In addition to having a very prolific fashion sense, the French are also very creative when it comes to the shopping experience. The landscape of French fashion startups is flourishing those days: 6 of the 10 companies presented in this article closed a round last spring/summer season and the results of their investments are released now. Let’s open the catwalk.

1. Flink, the Flipboard of fashion blogs.

The last born of “fashion inspiration” apps Flink defines itself as the “Flipboard of fashion blogs”. Thomas France and his co-founders developed the idea while working on another e-commerce app, Shopelia: user-feedback got them the idea on focusing on the fashion vertical. Its mobile-first approach and great UX seduced fashion lovers and got them to 50,000 downloads in just a few days. Their next step is to release a new, more social version in two months.

From Flink’s Instagram account

2. Carnet de Mode, crowdfunding meets social shopping.

Carnet de Mode brings together fashion-lovers and young creative talents: the former can participate in the funding of a collection, the latter are able to develop it and get access to their first customers. A real mix between crowdfunding and social shopping which seems to be a good call for fashion – a highly emotional sector that federates its passionnates. Elaia Partners agreed and invested in the startup in May 2013 ($1 million – the first e-commerce platform in Elaia’s portfolio) arguing Carnet de Mode was truly “revolutionizing the online shopping experience”.

3. InstantLuxe, buying luxury online has never been safer.

The fashion luxury marketplace places its expertise in luxury leather goods, watches and accessories. They play the intermediary and ensure a safe transaction between buyers and sellers with no risk of counterfeits. They achieved a €2 millions round of investment in June 2013 with Iris Capital and ISAI (who already invested €650 000 in 2010) which enables them to focus on the German market and mobile commerce.

4. Monshowroom, the first marketplace for on-going collections.

When Séverine Grégoire and Chloé Ramade created Monshowroom.com in 2006, the platform was basically the only way to access current collections online, an opportunity for people living outside of big cities, or simply online-fashion lovers. Today, the platform has become a reference with 270 brands available. Alven Capital and Crédit Agricole Private Equity funds invested in the company in 2009 and Casino bought their share in 2012, in order to create synergies with the e-commerce platform “CDiscount”, as sources say (FR).

Monshowroom’s Instagram

5. Dymant, invite-only eCommerce, the true eLuxury experience.

Dymant changed the e-commerce industry by focusing on the story-telling and esthetical atmosphere experience more than on the products. Quality VS SEO, to put it shortly. The marketplace offers invite-only customers the opportunity to access high-end and exclusive French “artisanat” (craftsmanship). Partech Ventures and IDInvest invested $1 million last year in order to enable the startup to address other geographical markets.

6. Vestiaire Collective, a second-hand luxury marketplace.

Co-founded by Sébastien Fabre and Sophie Hersan, Vestiaire Collective is a marketplace for second-hand luxury fashion products. Created in Paris in 2009, (because, as Sebastien Fabre told us: “In order to launch a fashion marketplace, we had to be in Paris”), they now are developing abroad: London in 2012, Germany and the U.S in 2014. This internationalization follows their series C round with Condé Nast ($15 millions), an interesting choice of partner, motivated by the expertise of the group in online community-based businesses and the U.S market.

Vestiaire Collective has innovated in the marketplace sector by adding a friction in the process – which could be seen as a bad example: in order to ensure a faithful community of customers, Vestiaire Collective has an editorial committee that selects which articles can be sold online, that specialists then check before they arrive in the customer’s hands. A great service that ensures happiness on both sides of the sale. They just opened a “Vintage” and “Lifestyle” section: take a look at it (even you male readers, you represent 15% of Vestiaire Collective’s clientele!)

Vestiaire Collective’s Instagram

7. VideDressing, social shopping for previously owned outfits.

If your wallet is slightly smaller than Chanel and Dior-material, then VideDressing might be the right fit for you. The principle is the same as other social shopping platform: Meryl Job and Renaud Guillerm, co-founders, managed to build a strong a faithful community first, and then monetized it by facilitating trades inside this community. They now claim a 5-times growth factor between 2011 and 2012 and realized two rounds in 2013 in order to penetrate the German market – the last one with Global Founders Capital for $5 Million.

8. Selectionnist, shopping through fashion magazines.

If you have ever been frustrated by not remembering the reference of an outfit you’ve seen in an old edition of Elle Magazine while waiting at the doctor’s, then you’ll love Selectionnist.

Selectionnist, founded by female serial-entrepreneurs Tatiana Jama and Lara Rouyres (Living Social, Dealissime) enables its visitors to search for things spotted by magazine or by brand. The beautifully-built website might also inspire some other shopping wishlists. The website just launched yesterday, time to try it out!

Selectionnist’s Instagram

9. Wheretoget.it, finds the fashion items you lost of sight.

Extreme situation, you have spotted an outfit somewhere, anywhere, and you must have it. But of course, you don’t have the reference. Hopefully, Wheretoget.it is here to help.

The platform (which is French, unlike the URL extension « .it » might indicate) gathers a community of passionate fashion-lovers who are able to spot where you can find your desperately lost article, providing that you have a picture. If not, there is a great chance they will find something similar, or even better. The force of the “multitude” applied to fashion. Rumor is that Romain Moyne, founder, is currently working on a Series A round of investment[fr].

10. Rad, flash sales dedicated to the Cool.

Rad’s value proposition is quite similar to other fashion discovery platforms: the cooler the pictures, the more you want to buy. How did Rad work out so well then? Rad’s particularity holds in its editorial line: it’s just cool, then you can only get it for a small amount of time.

They closed a €2.5 Million Series A round of funding in June 2013, declaring: “We are raising to accelerate our growth — hire key people, build a logistics platform, speed up acquisition through marketing, develop our production capacities”.

Rad’s Instagram

Schneier on Security: The Insecurity of Secret IT Systems

Fri, 14 Feb 2014 06:50:38 -0800

Comments: "Schneier on Security: The Insecurity of Secret IT Systems"

URL: https://www.schneier.com/blog/archives/2014/02/the_insecurity_2.html

A blog covering security and security technology.

« GOPHERSET: NSA Exploit of the Day | Main | My Talk on the NSA »

February 14, 2014

The Insecurity of Secret IT Systems

We now know a lot about the security of the Rapiscan 522 B x-ray system used to scan carry-on baggage in airports worldwide. Billy Rios, director of threat intelligence at Qualys, got himself one and analyzed it. And he presented his results at the Kaspersky Security Analyst Summit this week.

It’s worse than you might have expected:

It runs on the outdated Windows 98 operating system, stores user credentials in plain text, and includes a feature called Threat Image Projection used to train screeners by injecting .bmp images of contraband, such as a gun or knife, into a passenger carry-on in order to test the screener's reaction during training sessions. The weak logins could allow a bad guy to project phony images on the X-ray display.

While this is all surprising, it shouldn’t be. These are the same sort of problems we saw in proprietary electronic voting machines, or computerized medical equipment, or computers in automobiles. Basically, whenever an IT system is designed and used in secret – either actual secret or simply away from public scrutiny – the results are pretty awful.

I used to decry secret security systems as "security by obscurity." I now say it more strongly: "obscurity means insecurity."

Security is a process. For software, that process is iterative. It involves defenders trying to build a secure system, attackers -- criminals, hackers, and researchers -- defeating the security, and defenders improving their system. This is how all mass-market software improves its security. It’s the best system we have. And for systems that are kept out of the hands of the public, that process stalls. The result looks like the Rapiscan 522 B x-ray system.

Smart security engineers open their systems to public scrutiny, because that’s how they improve. The truly awful engineers will not only hide their bad designs behind secrecy, but try to belittle any negative security results. Get ready for Rapiscan to claim that the researchers had old software, and the new software has fixed all these problems. Or that they’re only theoretical. Or that the researchers themselves are the problem. We’ve seen it all before.

Tags: air travel, disclosure, economics of security, obscurity, secrecy, security engineering

Posted on February 14, 2014 at 6:50 AM • 28 Comments

To receive these entries once a month by e-mail, sign up for the Crypto-Gram Newsletter.

In Europe they use quantum key cryptography in their voting systems:

http://www.idquantique.com/news-and-events/...

In Europe, the struggle to create a public realm out of the monarchy's private government extends back to populist movements in the Middle Ages such as the Ranters and Diggers and Bretheren of the Free Spirit; they seem more likely to view their government as something that really belongs to them, with the potential to work for them.

In the United States, our struggle to create a public government really begins with the 14th Amendment. Between then and the civil rights era is when we obtained universal suffrage. The franchise was highly exclusive in the Revolutionary era -- so much so that "WE, the People" probably only represents the will of 5-7% of the population at the time. Since then, the conservative battle cry has been "smaller government" and "privatization." Notwithstanding that we had private government once before -- when we were owned by the Britain -- we have this myth of obtaining self rule by fighting tyranny, when, in fact, the road to self rule has been a much more complicated struggle. But the myth prevails over history.

The 522B is ancient - at least 5-10 years old. http://aerodetection.com/rapiscan-522b/ says the units it has are 8-10 years old.

I'd be much more interested in the Rapiscan 620DV which appears to be the model used at major airports in Europe.

I still remember the first time I ran smack into security-by-obscurity. My boss at the time put me in charge of the most secure system we had at work. It was kept behind a heavy locked door and I was admitted only after a strong lecture on how important it was to keep it as secure as possible.

Even behind a closed door, he felt it necessary to whisper: "and the password is 'secret', which of course we can't tell anybody and they would never guess..."

The idea of continual improvement driven by the hostile nature of the operating environment seems very similar to evolution in the natural world. I don't mean analagous to, I mean another form of evolution, survival of the fittest. Obscurity, obfuscation, and political lobbying are all attempts to exclude a product from the security evolutionary process but even at the very highest levels - government printed currency, NSA information - it is impossible to isolate anything from security evolutionary forces.
It is better to embrace this process and to continually iteratively evolve and grow stronger and wiser during the process than to try and hold the driving forces of evolution at bay. The driving threat forces of security evolution themselves evolve, adapt and get stronger over time. You can hold them at bay temporarily through obscurity and obfuscation, but it then become only a matter of for how long? e.g. Sony Playstation 3
The Playstation 3 is an interesting example in that it was finally cracked due to sloppy cryptography implementation ( what was meant to be a random number generator was implemented as a constant ). I say interesting because peer / open review would have exposed the obvious flaw and it would have been fixed. In this case it was the obscurity and obfuscation that ultimately was responsible for the security being broken.
https://www.schneier.com/blog/archives/2011/01/sony_ps3_securi.html

OK, so if Kaspersky or Symantec or TrendMicro or Norton doesn't hand over all their source code I shouldn't buy the product... right?

OK, go ahead and post all your passwords and SSN's and your credit card info.

@beatty Well ... yes, although not just because you don't have source code. Virus scanners typically only search for known exploit code (and perhaps a few variations on it). They're fundamentally reactive and in my view not worth the effort.

@vincent You jest. The difference between secret paswords and secret algorithms has been explained to death already.

@vincent: not really. That is yours, your privacy for your own usage/protection, but when somebody offer for public usage some kind of security product, it should be available for public scrutiny on potential security threats/weaknesses and invasion of privacy (like recently Samsung on smart TV agreed to provide led indicator when camera is on. I hope that activation is hardware, not software).

The meta-problem here is that many (in my experience most) software "engineers" are incompetent and do not qualify as engineers. The result is that the typical software system sucks badly.

This machine is just a standard example. However though Windows (no matter what version) was suitable as an embedded OS has no business working on software or surrounding systems.

@beatty -- you shouldn't buy their products if you don't trust them. I don't and the systems my family uses haven't had an problems. But they don't hide what they're doing and they have some pretty vigorous competition.

@vincent -- I hope you don't think there is anything especially "secure" about your SSN. But hiding secret information used to access a system is different than hiding or obscuring information about the the system itself, which is what this post is about.

No, this just another cherry-picked instance of failed product development that is being used for self-serving purposes. It's easy, isn't it, to swoop in after the fact and point out everything that went wrong. Do you even know what the original threat model was? For all you know this failure was the RESULT of anal security engineers that suffocated product development until the project collapsed. You don't know. You pick up on all this pop news junk and fling it anyway you want.

Oh, it gets better. I keep thinking that if you control the software, you can probably control where the X-Ray beam is at. And observe that TSA agents walk back and forth through the scanner all the time.

So you keep it ON bouncing back and forth scanning at roughly chest level when not actually scanning the full body. When you see something metallic (say, a TSA badge), you immediately drop the emitter down to crotch level for the next 10 seconds...

"For all you know this failure was the RESULT of anal security engineers"

Win98 and plaintext passwords are not the result of anal security engineers. Unless, of course you mean actual anal security engineers, in which case it is not surprising, as they only know stuff about how to use a variety of rubber corks.

I somewhat disagree that engineers try to hide their bad designs through obscurity. I don't believe they even think about security.

There seems to be this idea of "Well, why would anyone attack that?" that is prevalent far too often. It's why we have empty passwords on internet-facing SCADA stuff, hopelessly outdated operating systems that can't be updated on embedded systems, etc. Nobody thinks like attackers. Closed systems are perfectly acceptable to people who don't think they will ever be a target.

It would be like me inventing my own door lock, and because no one has seen one before, I can assume I'm protected. (Of course this could also be said - Since no one has attempted to break into it yet, I hope I'm protected)

Rather than the alternative

Buying a door lock that has been proven in the real world. Paying attention to security bulletins so that if an exploit is found, I can replace it with a fixed version.

@vincent - Either way, I'm not giving you my key.

@Jason. All you said is valid when you are random target making you more protected than next target in the phishing scheme. Just to bring some relax mood: "Two young ladies were in the jungle and spotted lion. One start running, another asked is she really could run faster than lion. Nope, she reply. Just faster than you..." If you are NOT random target, all depends on the actor's resources available to break your security (psychical or informational): local thugs, organized crime, LEA local or state, LEA federal, foreign agents, etc.

@vas pup:

Well, now we're just getting deeper into security concepts. I don't think the threat of a targeted attack is a reason to abandon tried and tested methods. I think it's a reason to bring in additional expertise, add some additional layers of protection. (Instead of just a good door lock, add a surveillance system, alarm system, maybe a stronger door, bars on windows). The security system you use, no matter what you're protecting has a cost that must be weighed against the risk, and to be effective almost certainly will be layered.

@ Bruce,

The truly awful engineers will not only hide their bad designs behind secrecy, but try to belittle any negative security results

That statment is a little unfair, because when it comes to hardware the closer you are to the metal, generaly the more competent you are as an "engineer".

The problem generaly starts and ends with managment, because,

1, Like quality, security has to be there fully functional from project day 0.

2, Security processes, training etc "cost".

You have to be an "old engineer" to remember the days befor quality processes were considered part and parcel of the job. And unfortunatly the area quality processes are least frequently found is "software engineering". Just take any modern software methodology and find the bits that are actually about "Quality Assurance"...

The simple answer is all you will find is an illusion or mirage paying lip service to any real quality process. It's also the reason grizzled old vetrans of software coding will tell you that most software development methodologies are at best "make work" and that you will get better results where team members share a common non adveserial goal and thus trust each other.

And when you look back at the development of QA systems it was the teams who bought into it and trusted the others that the most benifit was seen.

The reason QA actually got going was two fold,

1, Managment saw the financial benifit before the factory door.

2, Those who saw benifit used QA as a part of purchase decision.

Neither of these conditions is true currently for "security" thus managment treat it as "a non productive inefficiency" and thus "managment mantra" says it should be ruthlessly expunged from the work process "to increase productivity"

The way to get security into the design process as a norm is by making having it the most profitable path to walk, that way as with QA "managment mantra" will change.

Untill that time blaiming other people for "keeping their jobs" is a little unfair.

I think the article is dead wrong about the threat projection system being a big issue.

The purpose of this system is to keep the screener alert. In a normal airport, a contraband item like a bomb, gun, etc. might occur at most once a day. Rare contraband like a bomb is probably less than once in a lifetime. Hence it would be natural for a screener to simply 'pass' all luggage, even if they are being diligent. Adding these "false positives" gives the screener something to do, and increases security by "impedence matching" the task at hand to the psychology of the operator.

It's true that an attacker could have the system inject innocuous items, or perhaps have it inject items at a very high rate. I suspect that either of these new behaviors would be quickly noticed.

Actually, in any airport, contraband like this would occur at most 24hrs/airport-lockdown-time per day.

Looks like I was wrong. The other article gives more details about the system, and it is pretty crappy.

It's one thing to superimpose false images that are removed after alarming on them. It's another entirely to allow some other person to choose the time when the false image will be shown, and to replace rather than modify the image.

Hacking is illegal. Selling crappy secured soft- / hardware isn't (wearing my black and white glasses now).

"Upon seeing a weapon on the screen, operators are supposed to push a button to notify supervisors of the find. But if the image is a fake one that was superimposed, a message appears onscreen telling them so and advising them to search the bag anyway to be sure. If a fake image of a clean bag is superimposed on screen instead, the operator would never press the button, and therefore never be instructed to hand-search the bag."

If the training software assumes that the .bmp images have simulated contraband, one would think that the training software would do something if the operator doesn't press the button when a .bmp is displayed. Or does the attacker who introduces a "clean" .bmp file also modify the software?

Thank you, Bruce... "Obscurity means insecurity" is exactly what I've always meant, when I said "closed source by definition is insecure".... only open source can be secure (which doesn't guarantee that it is, only that it's at least possible).

What really worries me is that we haven't really learned a lot.

The 1983 movie wargames could happen today. Maybe not in the US (although I doubt that), but there are more countries in the world. The problem with security by obscurity is that you just don't know whether there is a WOPR that has a backdoor with the login "Joshua".

How secure are these nucleair platform systems? Just look at the stoxnet virus. Is "the west" capable of protecting itself against this kind of things? I don't think so. (looking at this news item)

And is the JSF/F-35 capable of dropping a nuke? It also contains 20 mln lines of C++ code.

I think this is way more worrying than any "terrorist attack".

I don't know. Maybe it's just BS that I am talking about. I am not a security expert. But I do know that you can't trust computers. Not yesterday, today or tomorrow.

Bruce,

Saw you at SAS, thanks for speaking!

Did you notice on the way out that all the machines in the Punta Cana airport were the make and model Billy and Terry evaluated?

Mike

Your comment about engineers (smart vs awful) was unfortunate. Well-established companies such as Diebold produce software with a workforce that is salaried and university-educated. The software produced usually conforms to management's priorities. If QA isn't isn't part of the software process, the software produced will tell the tale. The company with good management and a weak engineering staff is a rare beast. Unicorn rare. If Diebold has crappy software, then Diebold is to blame, not some mythical bumbler.

To say nothing of the procurement process.

True of Diebold and Rapiscan!

Schneier.com is a personal website. Opinions expressed are not necessarily those of Co3 Systems, Inc..

BBC News - Linking to free web content is legal, says EU Court

Fri, 14 Feb 2014 08:30:18 -0800

Comments: "BBC News - Linking to free web content is legal, says EU Court"

URL: http://www.bbc.co.uk/news/technology-26187730

14 February 2014 Last updated at 09:50 ET

Websites can link to freely available content without the permission of the copyright holder, the European Court of Justice says.

The court's decision came after a dispute in Sweden between journalists and a web company that had posted links on its site to online news articles.

A Swedish court had asked the EU court to consider whether this broke copyright law.

The "position would be different" for links that bypass a paywall.

The journalists worked for the Swedish newspaper Goteborgs-Posten and had articles published on the paper's website.

The company Retriever Sverige runs a website that provides links to articles published by other websites.

Broken the internet

The journalists argued in the original case that users of Retriever Sverige's website would not know that they had been sent to another website by clicking on the links and therefore had made their articles available without authorisation. They said that because of this they were due compensation.

The case was unsuccessful in the Swedish courts but there was an appeal and the appeal court asked the EU Court of Justice to consider whether copyright law had been broken.

The court had to consider whether by providing links Retriever Sverige had taken part in an "act of communication to the public". Under EU copyright law, authors have the exclusive right to authorise or prohibit any communication to the public of their works.

The court ruled that the law had not been broken because the articles in question were on Goteborgs-Posten's website and therefore already "freely available".

In a statement it said: "The owner of a website may, without the authorisation of the copyright holders, redirect internet users, via hyperlinks, to protected works available on a freely accessible basis on another site."

But they ruled that the "position would be different" if a link led users to material that had purposely been restricted from being freely available - for example if it had been posted on a site that operates a paywall.

"If the decision had gone the other way it would have broken the internet," said Susan Hall, technology lawyer at Clarke Willmott.

"The way we communicate online is predicated on sharing material, whether that's links to Robert Peston on Bank of England interest rates, decisions of the European court or pictures of otters who look like Benedict Cumberbatch," she said.

Goteborgs-Posten told the BBC that it was making no comment on the ruling.

IP Address Details - ipinfo.io

Fri, 14 Feb 2014 08:30:18 -0800

Comments: "IP Address Details - ipinfo.io"

solo.im - a single founder peer group

Fri, 14 Feb 2014 08:39:18 -0800

Comments: "solo.im - a single founder peer group"

How I want to write node: stream all the things!

Fri, 14 Feb 2014 08:45:18 -0800

Comments: "How I want to write node: stream all the things!"

URL: http://caolanmcmahon.com/posts/how_i_want_to_write_node_stream_all_the_things_new/

I wrote the async library back when Node first removed Promises from core (yes, that really was the case). Back then, I preferred to do with plain callbacks what was usually done by including third-party Promise, Future or Continuable libraries instead.

I find a certain elegance in describing complex patterns using simple parts, particularly when the parts are made composable through a common interface. In Node, and JavaScript in general, you're likely to use a combination of Promises, callbacks, Streams, Event Emitters and even ES6 Generators. To me, however, these all represent values in the future. What if we could reduce entire programs to transformations over a Stream, and have just one API to rule them all?

Well, here is my proposal, for your consideration and feedback:
Highland, a high-level streams library

var _ = require('highland');
var doubled = _([1, 2, 3, 4]).map(function (x) {
 return x * 2;
});
var data = filenames.map(readFile).parallel(4);
data.errors(function (err, rethrow) {
 
});
data.pipe(output);
var output = fs.createWriteStream('output');
var docs = db.createReadStream();
_(docs).filter(isBlogpost).pipe(output);
docs.pipe(_().filter(isBlogpost)).pipe(output);
var clicks = _('click', btn).map(1);
var counter = clicks.scan(0, _.add);
counter.each(function (n) {
 $('#count').text(n);
});

This is not a new idea, but I believe it is a new combination of features, which is important. Those of you that work with Streams in Node may be aware of the excellent modules by Dominic Tarr, similarly if you're from the browser you might use FRP libraries such as RxJS. These are all great, but they hint at a deeper abstraction, one which would allow us to write entire programs using Streams. In an attempt to achieve this, Highland implements:

Back-pressure support - Data sources are regulated so that slow consumers are not overwhelmed
Laziness - So we can use the reading of Streams to sequence the execution of code - this means we can choose to read from files in parallel or series, or stop reading after an error, for example
Asynchronous operations - So we're able to handle async data sources and async transformations
Error and data channels - So we can manage error propagation from sync and async code
Compatible with Node Streams - So we can pipe Node Streams to and from Highland Streams and play nicely with the Node ecosystem

If you find this idea intriguing and want to explore it further, then check out the Highland website. I'd love to hear your experiences.

Highland on GitHub

Crazy ants take on fire ants and win | Ars Technica

Fri, 14 Feb 2014 08:55:18 -0800

Comments: "Crazy ants take on fire ants and win | Ars Technica"

URL: http://arstechnica.com/science/2014/02/crazy-ants-take-on-fire-ants-and-win/

Fire ant, meet crazy ant.

If you’ve ever been attacked by red imported fire ants, you can likely attest that these tiny insects have a nasty defense mechanism. When threatened, fire ants inject or dab their enemy with a potent neurotoxic venom that quickly dispatches most other ants and can sometimes even send humans to the hospital.

This deadly assault is a pretty effective deterrent to most other ant species, which tend to keep their distance. But the fire ants have now run up against another invasive species that's willing to take them on: the tawny crazy ant. This unassuming little ant has the surprising habit of aggressively barreling right into a colony of fire ants, seemingly undeterred by their toxic venom.

In this week’s issue of Science, researchers from the University of Texas at Austin report that crazy ants are actually able to detoxify fire ants’ venom, helping them displace this usually dominant species.

A crazy ant’s headlong charge into a mass of fire ants almost always results in it being smeared with venom. But once covered in the neurotoxin, these ants perform a systematic and ritualized behavior. First, an ant stands on its hind legs and curls up its abdomen, touching its mandibles to a small gland at the tip of its body called the acidopore. The ant then runs its front legs through its mandibles and grooms itself fastidiously.

It looks like the ant is covering itself with a secretion from its own body as a response to the fire ant's venom. To determine whether this behavior actually counteracts the venom’s effect, the researchers ran a controlled test. They used nail polish to seal the acidopore of one group of crazy ants, and simply sham-treated a second control group. After coming into contact with fire ant venom, the crazy ants with the sealed acidopores—which could not secrete any chemical defenses—had a survival rate of just 48 percent, whereas 98 percent of the control group survived. Clearly, something originating from the acidopore was increasing the survival of ants covered in venom.

By testing crazy ant secretions, the researchers found that the life-saving substance was actually formic acid from the crazy ants’ own venom. So far, it's unclear exactly how this chemical detoxifies fire ant venom. It’s possible that formic acid denatures the enzymes that enable neurotoxins to enter cells.

While this rare ability confers a huge advantage for crazy ant survival, its biggest implications are ecological. Ever since fire ants were imported into the southern US in the 1930s, they have been the dominant ant species in most grassland ecosystems. But crazy ants—introduced only about 12 years ago—are now taking over, thanks in part to their ability to detoxify fire ant venom. When the two species fight over food or space, crazy ants come out on top 93 percent of the time.

Digging into these two species’ past sheds light on this asymmetry. Tawny crazy ants and red imported fire ants share an evolutionary history since their native ranges overlap in parts of South America. Their arms race began there, with fire ants evolving venom to defend themselves and crazy ants evolving a detoxification mechanism as a counter-defense. Now the chemical warfare has been re-engaged here on a second continent, playing out across the Gulf Coast. And for a second time in the past century, a new invasive ant species is dominating and drastically transforming ecological communities.

Related PSA: crazy ants are attracted to electronics. So if you're a technology lover living in the southeastern US, watch your computers and appliances because these tiny invaders are headed your way.

Science, 2014. DOI: 10.1126/science.1245833 (About DOIs).

Listing image by University of Texas

The Coinbase Blog — Introducing “BitHack”: Hackathon by Coinbase

Fri, 14 Feb 2014 09:41:31 -0800

Comments: "The Coinbase Blog — Introducing “BitHack”: Hackathon by Coinbase"

URL: http://blog.coinbase.com/post/76553987867/introducing-bithack-hackathon-by-coinbase

We’re excited to announce the launch of Bithackathon.com – an online hackathon to inspire developers across platforms and continents to build solutions with bitcoin.

We will judge app entries based on creativity, usability, and execution. The prize? Bitcoins!

1st prize: $10,000 worth of bitcoin
2nd prize: $5,000 worth of bitcoin
3rd prize: $3,000 worth of bitcoin

Criteria

We are looking for apps that excel in three areas:

1) Creativity:

Originality of idea
Innovation

2) Usability:

3) Execution:

We invite any and all developers around the world to participate in the competition. Check out www.bithackathon.com for more information – spread the word! Look forward to seeing you there.

Cryptic Crossword: Amateur Crypto and Reverse Engineering

Fri, 14 Feb 2014 09:49:31 -0800

Comments: "Cryptic Crossword: Amateur Crypto and Reverse Engineering"

URL: http://www.muppetlabs.com/~breadbox/txt/acre.html

Rendered Prose Diffs · GitHub

Fri, 14 Feb 2014 10:37:31 -0800

Comments: "Rendered Prose Diffs · GitHub"

URL: https://github.com/blog/1784-rendered-prose-diffs

Today we are making it easier to review and collaborate on prose documents. Commits and pull requests including prose files now feature source and rendered views.

Click the "rendered" button to see the changes as they'll appear in the rendered document. Rendered prose view is handy when you're adding, removing, and editing text:

Or working with more complex structures like tables:

Non-text changes appear with a low-key dotted underline. Hover over the text to see what has changed:

Building great software is about more than code. Whether you're writing docs, planning development, or blogging what you've learned, better prose makes for better products. Go forth and write together!

Need help or found a bug? Contact us.

Phaser

Fri, 14 Feb 2014 11:12:35 -0800

Comments: "Phaser"

URL: http://phaserapp.com

Stephen Law: How the US Treasury imposes sanctions on me and every other "Stephen Law" on the planet - my letter to OFAC

Fri, 14 Feb 2014 12:26:16 -0800

Comments: "Stephen Law: How the US Treasury imposes sanctions on me and every other "Stephen Law" on the planet - my letter to OFAC"

URL: http://stephenlaw.blogspot.com/2014/02/how-us-treasury-imposes-sanctions-on-me.html

Right, here's another thing I am getting off my chest - email letter to OFAC (edited slightly from version sent).

Dear OFAC

This correspondence is copied to my UK Member of Parliament The Right Hon. Andrew Smith. Please copy him into your reply.

My name is "Stephen Law". The name "Stephen Law" appear on OFAC's "specially designated nationals" list:

Here is the actual OFAC listing for "Stephen Law", alias of "Steven Law"

https://ofac.data-list-search.com/Entities/ByName/steven-law

This person is Burmese and is suspected by US Treasury of drug trafficking. He is the son of Lo Hsing Han (dubbed by US Treasury as "The Godfather of Heroin") and has a Singaporean wife. His addresses, as listed by you, are all in Burma and Singapore. None are in the UK.

I have discovered that, as a result of this listing, US Customs block shipments of goods to me here in the UK. Also when people try to wire me money from abroad (not just from the US, but from anywhere), for e.g. occasional travel expenses for academic conference attendance, the payment is interrupted and various checks are made before the funds are released. This became so bad during one period (a series of payments every single one of which triggered a block) that I had to switch to a different bank account. At no point was I told why this was happening (i.e. that you, OFAC, are responsible). The banks concerned believe they must keep this information from me (I was told this by my bank branch). Hence it took me many months to figure out what the source of the problem was: OFAC/US Treasury.

It appears any "Stephen Law" anywhere in the world will suffer this same treatment, as indeed will anyone who merely happens to have the same name or alias as one of your "specially designated nationals". This has proved frustrating, time-consuming and also costly to me personally. E.g. I have paid US$77 postage for goods it turns out I can never receive because they are returned by US customs to the US vendor because my name is listed. As a result of the OFAC listing, I cannot now order goods from - or receive gifts from friends and relatives in - the United States.

Can you inform me: given I am very obviously NOT the Burmese Stephen Law:

(i) how I can avoid having all goods shipped to me from the US to my UK address being blocked and returned to sender by US customs?

(ii) how I can avoid my own bank repeatedly asking me who I am (and requesting information including my DOB, which they already possess) before unblocking any payment from abroad?

My bank knows who I am, and they know I am not the Burmese "Stephen Law" on the specially designated nationals list, but still I have to go through this same rigmarole every single time money is wired to me. How do I avoid this please?

Yours faithfully

Stephen Law

PS Ofac-caused delays to payments to me can run into weeks. On one occasion I ran up overdraft charges as a result of not receiving funds blocked by OFAC.

PPS I was interviewed by Foreign Policy magazine about all this a short while ago.Also interviewed by News Hour on BBC World Service.

Fri, 14 Feb 2014 12:32:16 -0800

Comments: ""

URL: http://www.darpa.mil/opencatalog/

Aptima Inc. Network Query by Example Analytics 2014-07 https://github.com/Aptima/pattern-matching.git stats Hadoop MapReduce-over-Hive based implementation of network query by example utilizing attributed network pattern matching. ALv2 Boeing/Pitt
Publications SMILE-WIDE: A scalable Bayesian network library Analytics 2014-07 https://github.com/SmileWide/main.git stats SMILE-WIDE is a scalable Bayesian network library. Initially, it is a version of the SMILE library, as in SMILE With Integrated Distributed Execution. The general approach has been to provide an API similar to the existing API SMILE developers use to build "local," single-threaded applications. However, we provide "vectorized" operations that hide a Hadoop-distributed implementation. Apart from invoking a few idioms like generic Hadoop command line argument parsing, these appear to the developer as if they were executed locally. ALv2 Carnegie Mellon University
Publications Support Distribution Machines Analytics 2014-07 https://github.com/dougalsutherland/py-sdm.git stats Python implementation of the nonparametric divergence estimators described by Barnabas Poczos, Liang Xiong, Jeff Schneider (2011). Nonparametric divergence estimation with applications to machine learning on distributions. Uncertainty in Artificial Intelligence. ( http://autonlab.org/autonweb/20287.html ) and also their use in support vector machines, as described by Dougal J. Sutherland, Liang Xiong, Barnabas Poczos, Jeff Schneider (2012). Kernels on Sample Sets via Nonparametric Divergence Estimates. ( http://arxiv.org/abs/1202.0302 ). BSD Continuum Analytics Blaze Infrastructure 2014-07 https://github.com/ContinuumIO/blaze.git stats Blaze is the next-generation of NumPy. It is designed as a foundational set of abstractions on which to build out-of-core and distributed algorithms over a wide variety of data sources and to extend the structure of NumPy itself. Blaze allows easy composition of low level computation kernels (C, Fortran, Numba) to form complex data transformations on large datasets. In Blaze, computations are described in a high-level language (Python) but executed on a low-level runtime (outside of Python), enabling the easy mapping of high-level expertise to data without sacrificing low-level performance. Blaze aims to bring Python and NumPy into the massively-multicore arena, allowing it to leverage many CPU and GPU cores across computers, virtual machines and cloud services. BSD Continuum Analytics Numba Infrastructure 2014-07 https://github.com/numba/numba.git stats Numba is an Open Source NumPy-aware optimizing compiler for Python sponsored by Continuum Analytics, Inc. It uses the LLVM compiler infrastructure to compile Python syntax to machine code.

It is aware of NumPy arrays as typed memory regions and so can speed-up code using NumPy arrays. Other, less well-typed code is translated to Python C-API calls effectively removing the "interpreter" but not removing the dynamic indirection.

Numba is also not a tracing just in time (JIT) compiler. It compiles your code before it runs either using run-time type information or type information you provide in the decorator.

Numba is a mechanism for producing machine code from Python syntax and typed data structures such as those that exist in NumPy.

BSD Continuum Analytics Bokeh Visualization 2014-07 https://github.com/ContinuumIO/bokeh.git stats Bokeh (pronounced bo-Kay or bo-Kuh) is a Python interactive visualization library for large datasets that natively uses the latest web technologies. Its goal is to provide elegant, concise construction of novel graphics in the style of Protovis/D3, while delivering high-performance interactivity over large data to thin clients. BSD Continuum Analytics and Indiana University
Publications Abstract Rendering Visualization 2014-07 https://github.com/JosephCottam/AbstractRendering.git stats Information visualization rests on the idea that a meaningful relationship can be drawn between pixels and data. This is most often mediated by geometric entities (such as circles, squares and text) but always involves pixels eventually to display. In most systems, the pixels are tucked away under levels of abstraction in the rendering system. Abstract Rendering takes the opposite approach: expose the pixels and gain powerful pixel-level control. This pixel-level power is a complement to many existing visualization techniques. It is an elaboration on rendering, not an analytic or projection step, so it can be used as an epilogue to many existing techniques. In standard rendering, geometric objects are projected to an image and represented on that image's discrete pixels. The source space is an abstract canvas that contains logically continuous geometric primitives and the target space is an image that contains discrete colors. Abstract Rendering fits between these two states. It introduces a discretization of the data at the pixel-level, but not necessarily all the way to colors. This enables many pixel-level concerns to be efficiently and concisely captured. BSD Continuum Analytics CDX Visualization 2014-07 https://github.com/ContinuumIO/cdx.git stats Software to visualize the structure of large or complex datasets / produce guides that help users or algorithms gauge the quality of various kinds of graphs & plots. BSD Continuum Analytics and Indiana University
Publications Stencil Visualization 2014-07 https://github.com/JosephCottam/Stencil.git stats Stencil is a grammar-based approach to visualization specification at a higher-level. BSD Data Tactics Corporation Vowpal Wabbit Analytics 2014-07 https://github.com/JohnLangford/vowpal_wabbit.git stats The Vowpal Wabbit (VW) project is a fast out-of-core learning system sponsored by Microsoft Research and (previously) Yahoo! Research. Support is available through the mailing list. There are two ways to have a fast learning algorithm: (a) start with a slow algorithm and speed it up, or (b) build an intrinsically fast learning algorithm. This project is about approach (b), and it's reached a state where it may be useful to others as a platform for research and experimentation. There are several optimization algorithms available with the baseline being sparse gradient descent (GD) on a loss function (several are available). The code should be easily usable. Its only external dependence is on the boost library, which is often installed by default. BSD Data Tactics Corporation Circuit Infrastructure 2014-07 https://code.google.com/p/gocircuit/source/checkout Go Circuit reduces the human development and sustenance costs of complex massively-scaled systems nearly to the level of their single-process counterparts. It is a combination of proven ideas from the Erlang ecosystem of distributed embedded devices and Go's ecosystem of Internet application development. Go Circuit extends the reach of Go's linguistic environment to multi-host/multi-process applications. ALv2 Georgia Tech / GTRI
Publications libNMF: a high-performance library for nonnegative matrix factorization and hierarchical clustering Analytics 2014-07 Pending LibNMF is a high-performance, parallel library for nonnegative matrix factorization on both dense and sparse matrices written in C++. Implementations of several different NMF algorithms are provided, including multiplicative updating, hierarchical alternating least squares, nonnegative least squares with block principal pivoting, and a new rank2 algorithm. The library provides an implementation of hierarchical clustering based on the rank2 NMF algorithm. ALv2 IBM Research
Publications SKYLARK: Randomized Numerical Linear Algebra and ML Analytics 2014-07 2014-05-15 SKYLARK implements Numerical Linear Algebra (NLA) kernels based on sketching for distributed computing platforms. Sketching reduces dimensionality through randomization, and includes Johnson-Lindenstrauss random projection (JL); a faster version of JL based on fast transform techniques; sparse techniques that can be applied in time proportional to the number of nonzero matrix entries; and methods for approximating kernel functions and Gram matrices arising in nonlinear statistical modeling problems. We have a library of such sketching techniques, built using MPI in C++ and callable from Python, and are applying the library to regression, low-rank approximation, and kernel-based machine learning tasks, among other problems. ALv2 Institute for Creative Technologies / USC Immersive Body-Based Interactions Visualization 2014-07 http://code.google.com/p/svnmimir/source/checkout stats Provides innovative interaction techniques to address human-computer interaction challenges posed by Big Data. Examples include:
* Wiggle Interaction Technique: user induced motion to speed visual search.
* Immersive Tablet Based Viewers: low cost 3D virtual reality fly-through's of data sets.
* Multi-touch interfaces: browsing/querying multi-attribute and geospatial data, hosted by SOLR.
* Tablet based visualization controller: eye-free rapid interaction with visualizations. ALv2 Johns Hopkins University
Publications igraph Analytics 2014-07 https://github.com/igraph/xdata-igraph.git stats igraph provides a fast generation of large graphs, fast approximate computation of local graph invariants, fast parallelizable graph embedding. API and Web-service for batch processing graphs across formats. GPLv2 Trifacta (Stanford, University of Washington, Kitware, Inc. Team) Vega Visualization 2014-07 https://github.com/trifacta/vega.git stats Vega is a visualization grammar, a declarative format for creating and saving visualization designs. With Vega you can describe data visualizations in a JSON format, and generate interactive views using either HTML5 Canvas or SVG. BSD Kitware, Inc. Tangelo Visualization 2014-07 https://github.com/Kitware/tangelo.git stats Tangelo provides a flexible HTML5 web server architecture that cleanly separates your web applications (pure Javascript, HTML, and CSS) and web services (pure Python). This software is bundled with some great tools to get you started. ALv2 Harvard and Kitware, Inc.
Publications LineUp Visualization 2014-07 https://github.com/Caleydo/org.caleydo.vis.lineup.demos.git stats LineUp is a novel and scalable visualization technique that uses bar charts. This interactive technique supports the ranking of items based on multiple heterogeneous attributes with different scales and semantics. It enables users to interactively combine attributes and flexibly refine parameters to explore the effect of changes in the attribute combination. This process can be employed to derive actionable insights as to which attributes of an item need to be modified in order for its rank to change. Additionally, through integration of slope graphs, LineUp can also be used to compare multiple alternative rankings on the same set of items, for example, over time or across different attribute combinations. We evaluate the effectiveness of the proposed multi-attribute visualization technique in a qualitative study. The study shows that users are able to successfully solve complex ranking tasks in a short period of time. BSD Harvard and Kitware, Inc.
Publications LineUp Web Visualization 2014-07 2014-06 LineUpWeb is the web version of the novel and scalable visualization technique. This interactive technique supports the ranking of items based on multiple heterogeneous attributes with different scales and semantics. It enables users to interactively combine attributes and flexibly refine parameters to explore the effect of changes in the attribute combination. BSD Stanford, University of Washington, Kitware, Inc. Lyra Visualization 2014-07 2014-02 Lyra is an interactive environment that makes custom visualization design accessible to a broader audience. With Lyra, designers map data to the properties of graphical marks to author expressive visualization designs without writing code. Marks can be moved, rotated and resized using handles; relatively positioned using connectors; and parameterized by data fields using property drop zones. Lyra also provides a data pipeline interface for iterative, visual specification of data transformations and layout algorithms. Visualizations created with Lyra are represented as specifications in Vega, a declarative visualization grammar that enables sharing and reuse. BSD Phronesis stat_agg Analytics 2014-07 https://github.com/kaneplusplus/stat_agg.git stats stat_agg is a Python package that provides statistical aggregators that maximize ensemble prediction accuracy by weighting individual learners in an optimal way. When used with the laputa package, learners may be distributed across a cluster of machines. The package also provides fault-tolerance when one or more learners becomes unavailable. ALv2 Phronesis flexmem Infrastructure 2014-07 https://github.com/kaneplusplus/flexmem.git stats Flexmem is a general, transparent tool for out-of-core (OOC) computing in the R programming environment. It is launched as a command line utility, taking an application as an argument. All memory allocations larger than a specified threshold are memory-mapped to a binary file. When data are not needed, they are stored on disk. It is both process- and thread-safe. ALv2 Phronesis laputa Infrastructure 2014-07 https://github.com/kaneplusplus/laputa.git stats Laputa is a Python package that provides an elastic, parallel computing foundation for the stat_agg (statistical aggregates) package. ALv2 Phronesis bigmemory Infrastructure 2014-07 http://cran.r-project.org/web/packages/bigmemory/index.html Bigmemory is an R package to create, store, access, and manipulate massive matrices. Matrices are allocated to shared memory and may use memory-mapped files. Packages biganalytics, bigtabulate, synchronicity, and bigalgebra provide advanced functionality. ALv2 Phronesis bigalgebra Infrastructure 2014-07 https://r-forge.r-project.org/scm/viewvc.php/?root=bigmemory Bigalgebra is an R package that provides arithmetic functions for R matrix and big.matrix objects. ALv2 MDA Information Systems, Inc., Jet Propulsion Laboratory, USC/Information Sciences Institute OODT Infrastructure 2014-07 https://svn.apache.org/repos/asf/oodt/ stats APACHE OODT enables transparent access to distributed resources, data discovery and query optimization, and distributed processing and virtual archives. OODT provides software architecture that enables models for information representation, solutions to knowledge capture problems, unification of technology, data, and metadata. ALv2 MDA Information Systems, Inc.,Jet Propulsion Laboratory, USC/Information Sciences Institute Wings Infrastructure 2014-07 https://github.com/varunratnakar/wings.git stats WINGS provides a semantic workflow system that assists scientists with the design of computational experiments. A unique feature of WINGS is that its workflow representations incorporate semantic constraints about datasets and workflow components, and are used to create and validate workflows and to generate metadata for new data products. WINGS submits workflows to execution frameworks such as Pegasus and OODT to run workflows at large scale in distributed resources. ALv2 MIT-LL
Publications Query By Example (Graph QuBE) Analytics 2014-07 2014-02-15 Query-by-Example (Graph QuBE) on dynamic transaction graphs. ALv2 MIT-LL
Publications Julia Analytics 2014-07 https://github.com/JuliaLang/julia.git stats Julia is a high-level, high-performance dynamic programming language for technical computing, with syntax that is familiar to users of other technical computing environments. It provides a sophisticated compiler, distributed parallel execution, numerical accuracy, and an extensive mathematical function library. MIT,GPL,LGPL,BSD MIT-LL
Publications Topic Analytics 2014-07 Pending Probabilistic Latent Semantic Analysis (pLSA) Topic Modeling. ALv2 MIT-LL
Publications SciDB Infrastructure 2014-07 https://github.com/wujiang/SciDB-mirror.git stats Scientific Database for large-scale numerical data. GPLv3 MIT-LL
Publications Information Extractor Analytics 2014-07 Pending Trainable named entity extractor (NER) and relation extractor. ALv2 Next Century Corporation Ozone Widget Framework Visualization 2014-07 https://github.com/ozoneplatform/owf.git stats Ozone Widget Framework provides a customizable open-source web application that assembles the tools you need to accomplish any task and enables those tools to communicate with each other. It is a technology-agnostic composition framework for data and visualizations in a common browser-based display and interaction environment that lowers the barrier to entry for the development of big data visualizations and enables efficient exploration of large data sets. ALv2 Next Century Corporation Neon Visualization Environment Visualization 2014-07 https://github.com/NextCenturyCorporation/neon.git stats Neon is a framework that gives a datastore agnostic way for visualizations to query data and perform simple operations on that data such as filtering, aggregation, and transforms. It is divided into two parts, neon-server and neon-client. Neon-server provides a set of RESTful web services to select a datastore and perform queries and other operations on the data. Neon-client is a javascript API that provides a way to easily integrate neon-server capabilities into a visualization, and also aids in 'widgetizing' a visualization, allowing it to be integrated into a common OWF based ecosystem. ALv2 Oculus Info Inc.
Publications ApertureJS Visualization 2014-07 https://github.com/oculusinfo/aperturejs.git stats ApertureJS is an open, adaptable and extensible JavaScript visualization framework with supporting REST services, designed to produce visualizations for analysts and decision makers in any common web browser. Aperture utilizes a novel layer based approach to visualization assembly, and a data mapping API that simplifies the process of adaptable transformation of data and analytic results into visual forms and properties. Aperture vizlets can be easily embedded with full interoperability in frameworks such as the Ozone Widget Framework (OWF). MIT Oculus Info Inc.
Publications Influent Visualization 2014-07 https://github.com/oculusinfo/influent.git stats Influent is an HTML5 tool for visually and interactively following transaction flow, rapidly revealing actors and behaviors of potential concern that might otherwise go unnoticed. Summary visualization of transactional patterns and actor characteristics, interactive link expansion and dynamic entity clustering enable Influent to operate effectively at scale with big data sources in any modern web browser. Influent has been used to explore data sets with millions of entities and hundreds of millions of transactions. MIT Oculus Info Inc.
Publications Aperture Tile-Based Visual Analytics Visualization 2014-07 https://github.com/oculusinfo/aperture-tiles.git stats New tools for raw data characterization of 'big data' are required to suggest initial hypotheses for testing. The widespread use and adoption of web-based maps has provided a familiar set of interactions for exploring abstract large data spaces. Building on these techniques, we developed tile based visual analytics that provide browser-based interactive visualization of billions of data points. MIT Oculus Info Inc.
Publications Oculus Ensemble Clustering Analytics 2014-07 https://github.com/oculusinfo/ensemble-clustering.git stats Oculus Ensemble Clustering is a flexible multi-threaded clustering library for rapidly constructing tailored clustering solutions that leverage the different semantic aspects of heterogeneous data. The library can be used on a single machine using multi-threading or distributed computing using Spark. MIT Raytheon BBN Content and Context-based Graph Analysis: PINT, Patterns in Near-Real Time Analytics 2014-07 https://github.com/plamenbbn/XDATA.git stats Patterns in Near-Real Time will take any corpus as input and quantify the strength of the query match to a SME-based process model, represent process model as a Directed Acyclic Graph (DAG), and then search and score potential matches. ALv2 Raytheon BBN Content and Context-based Graph Analysis: NILS, Network Inference of Link Strength Analytics 2014-07 https://github.com/plamenbbn/XDATA.git stats Network Inference of Link Strength will take any text corpus as input and quantify the strength of connections between any pair of entities. Link strength probabilities are computed via shortest path. ALv2 Royal Caliber
Publications GPU based Graphlab style Gather-Apply-Scatter (GAS) platform for quickly implementing and running graph algorithms Analytics 2014-07 https://github.com/RoyalCaliber/vertexAPI2.git stats Allows users to express graph algorithms as a series of Gather-Apply-Scatter (GAS) steps similar to GraphLab. Runs these vertex programs using a single or multiple GPUs - demonstrates a large speedup over GraphLab. ALv2 Scientific Systems Company, Inc., MIT, and University of Louisville BayesDB Analytics 2014-07 https://github.com/mit-probabilistic-computing-project/BayesDB.git stats BayesDB is an open-source implementation of a predictive database table. It provides predictive extensions to SQL that enable users to query the implications of their data --- predict missing entries, identify predictive relationships between columns, and examine synthetic populations --- based on a Bayesian machine learning system in the backend. ALv2 Scientific Systems Company, Inc., MIT, and University of Louisville Crosscat Analytics 2014-07 https://github.com/mit-probabilistic-computing-project/crosscat.git stats CrossCat is a domain-general, Bayesian method for analyzing high-dimensional data tables. CrossCat estimates the full joint distribution over the variables in the table from the data via approximate inference in a hierarchical, nonparametric Bayesian model, and provides efficient samplers for every conditional distribution. CrossCat combines strengths of nonparametric mixture modeling and Bayesian network structure learning: it can model any joint distribution given enough data by positing latent variables, but also discovers independencies between the observable variables. ALv2 Sotera Defense Solutions, Inc.
Publications Zephyr Infrastructure 2014-07 http://github.com/Sotera/zephyr stats Zephyr is a big data, platform agnostic ETL API, with Hadoop MapReduce, Storm, and other big data bindings. ALv2 Sotera Defense Solutions, Inc.
Publications Page Rank Analytics 2014-07 https://github.com/Sotera/page-rank.git stats Sotera Page Rank is a Giraph/Hadoop implementation of a distributed version of the Page Rank algorithm. ALv2 Sotera Defense Solutions, Inc.
Publications Louvain Modularity Analytics 2014-07 https://github.com/Sotera/distributed-louvain-modularity.git stats Giraph/Hadoop implementation of a distributed version of the Louvain community detection algorithm. ALv2 Sotera Defense Solutions, Inc.
Publications Spark MicroPath Analytics 2014-07 https://github.com/Sotera/aggregate-micro-paths.git The Spark implementation of the micropath analytic. ALv2 Sotera Defense Solutions, Inc.
Publications ARIMA Analytics 2014-07 https://github.com/Sotera/rhipe-arima stats Hive and RHIPE implementation of an ARIMA analytic. ALv2 Sotera Defense Solutions, Inc.
Publications Leaf Compression Analytics 2014-07 https://github.com/Sotera/leaf-compression.git stats Recursive algorithm to remove nodes from a network where degree centrality is 1. ALv2 Sotera Defense Solutions, Inc.
Publications Correlation Approximation Analytics 2014-07 https://github.com/Sotera/correlation-approximation stats Spark implementation of an algorithm to find highly correlated vectors using an approximation algorithm. ALv2 Stanford University - Boyd
Publications QCML (Quadratic Cone Modeling Language) Analytics 2014-07 https://github.com/cvxgrp/qcml.git stats Seamless transition from prototyping to code generation. Enable ease and expressiveness of convex optimization across scales with little change in code. ALv2 Stanford University - Boyd
Publications PDOS (Primal-dual operator splitting) Analytics 2014-07 https://github.com/cvxgrp/pdos.git stats Concise algorithm for solving convex problems; solves problems passed from QCML. ALv2 Stanford University - Boyd
Publications SCS (Self-dual Cone Solver) Analytics 2014-07 https://github.com/cvxgrp/scs.git stats Implementation of a solver for general cone programs, including linear, second-order, semidefinite and exponential cones, based on an operator splitting method applied to a self-dual homogeneous embedding. The method and software supports both direct factorization, with factorization caching, and an indirect method, that requires only the operator associated with the problem data and its adjoint. The implementation includes interfaces to CVX, CVXPY, matlab, as well as test routines. This code is described in detail in an associated paper, at http://www.stanford.edu/~boyd/papers/pdos.html (which also links to the code). ALv2 Stanford University - Boyd
Publications ECOS: An SOCP Solver for Embedded Systems Analytics 2014-07 https://github.com/ifa-ethz/ecos.git stats ECOS is a lightweight primal-dual homogeneous interior-point solver for SOCPs, for use in embedded systems as well as a base solver for use in large scale distributed solvers. It is described in the paper at http://www.stanford.edu/~boyd/papers/ecos.html. ALv2 Stanford University - Boyd
Publications Proximal Operators Analytics 2014-07 https://github.com/cvxgrp/proximal.git stats This library contains sample implementations of various proximal operators in Matlab. These implementations are intended to be pedagogical, not the most performant. This code is associated with the paper Proximal Algorithms by Neal Parikh and Stephen Boyd. ALv2 Stanford University - Hanrahan
Publications imMens Visualization 2014-07 https://github.com/StanfordHCI/imMens.git stats imMens is a web-based system for interactive visualization of large databases. imMens uses binned aggregation to produce summary visualizations that avoid the shortcomings of standard sampling-based approaches. Through data decomposition methods (to limit data transfer) and GPU computation via WebGL (for parallel query processing), imMens enables real-time (50fps) visual querying of billion+ element databases. BSD Stanford University - Hanrahan
Publications trelliscope Visualization 2014-07 https://github.com/hafen/trelliscope.git stats Trellis Display, developed in the 90s, also divides the data. A visualization method is applied to each subset and shown on one panel of a multi-panel trellis display. This framework is a very powerful mechanism for all data, large and small. Trelliscope, a layer that uses datadr, extends Trellis to large complex data. An interactive viewer is available for viewing subsets of very large displays, and the software provides the capability to sample subsets of panels from rigorous sampling plans. Sampling is often necessary because in most applications, there are too many subsets to look at them all. BSD Stanford University - Hanrahan
Publications RHIPE: R and Hadoop Integrated Programming Environment Infrastructure 2014-07 https://github.com/saptarshiguha/RHIPE.git stats In Divide and Recombine (D&R), big data are divided into subsets in one or more ways, forming divisions. Analytic methods, numeric-categorical methods of machine learning and statistics plus visualization methods, are applied to each of the subsets of a division. Then the subset outputs for each method are recombined. D&R methods of division and recombination seek to make the statistical accuracy of recombinations as large as possible, ideally close to that of the hypothetical direct, all-data application of the methods. The D&R computational environment starts with RHIPE, a merger of R and Hadoop. RHIPE allows an analyst to carry out D&R analysis of big data wholly from within R, and use any of the thousands of methods available in R. RHIPE communicates with Hadoop to carry out the big, parallel computations. ALv2 Stanford University - Hanrahan
Publications Riposte Analytics 2014-07 https://github.com/jtalbot/riposte.git stats Riposte is a fast interpreter and JIT for R. The Riposte VM has 2 cooperative subVMs for R scripting (like Java) and for R vector computation (like APL). Our scripting code has been 2-4x faster in Riposte than in R's recent bytecode interpreter. Vector-heavy code is 5-10x faster. Speeding up R can greatly increases the analyst's efficiency. BSD Stanford University - Olukotun
Publications Delite Infrastructure 2014-07 https://github.com/stanford-ppl/Delite.git stats Delite is a compiler framework and runtime for parallel embedded domain-specific languages (DSLs). BSD Stanford University - Olukotun
Publications SNAP Infrastructure 2014-07 https://github.com/snap-stanford/snap stats Stanford Network Analysis Platform (SNAP) is a general purpose network analysis and graph mining library. It is written in C++ and easily scales to massive networks with hundreds of millions of nodes, and billions of edges. It efficiently manipulates large graphs, calculates structural properties, generates regular and random graphs, and supports attributes on nodes and edges. BSD SYSTAP, LLC bigdata Infrastructure 2014-07 https://bigdata.svn.sourceforge.net/svnroot/bigdata/ stats Bigdata enables massively parallel graph processing on GPUs and many core CPUs. The approach is based on the decomposition of a graph algorithm as a vertex program. The initial implementation supports an API based on the GraphLab 2.1 Gather Apply Scatter (GAS) API. Execution is available on GPUs, Intel Xenon Phi (aka MIC), and multi-core GPUs. GPLv2 SYSTAP, LLC mpgraph Analytics 2014-07 http://svn.code.sf.net/p/mpgraph/code/ stats Mpgraph enables massively parallel graph processing on GPUs and many core CPUs. The approach is based on the decomposition of a graph algorithm as a vertex program. The initial implementation supports an API based on the GraphLab 2.1 Gather Apply Scatter (GAS) API. Execution is available on GPUs, Intel Xenon Phi (aka MIC), and multi-core GPUs. ALv2 UC Davis Gunrock Analytics 2014-07 https://github.com/gunrock/gunrock.git stats Gunrock is a CUDA library for graph primitives that refactors, integrates, and generalizes best-of-class GPU implementations of breadth-first search, connected components, and betweenness centrality into a unified code base useful for future development of high-performance GPU graph primitives. ALv2 Draper Laboratory
Publications Analytic Activity Logger Infrastructure 2014-07 https://github.com/draperlab/xdatalogger.git stats Analytic Activity Logger is an API that creates a common message passing interface to allow heterogeneous software components to communicate with an activity logging engine. Recording a user's analytic activities enables estimation of operational context and workflow. Combined with psychophysiology sensing, analytic activity logging further enables estimation of the user's arousal, cognitive load, and engagement with the tool. ALv2 University of California, Berkeley
Publications BDAS Infrastructure 2014-07 N/A BDAS, the Berkeley Data Analytics Stack, is an open source software stack that integrates software components being built by the AMPLab to make sense of Big Data. ALv2, BSD University of California, Berkeley
Publications Spark Infrastructure 2014-07 https://github.com/mesos/spark.git stats Apache Spark is an open source cluster computing system that aims to make data analytics both fast to run and fast to write. To run programs faster, Spark offers a general execution model that can optimize arbitrary operator graphs, and supports in-memory computing, which lets it query data faster than disk-based engines like Hadoop. To make programming faster, Spark provides clean, concise APIs in Python, Scala and Java. You can also use Spark interactively from the Scala and Python shells to rapidly query big datasets. ALv2 University of California, Berkeley
Publications Shark Infrastructure 2014-07 https://github.com/amplab/shark.git stats Shark is a large-scale data warehouse system for Spark that is designed to be compatible with Apache Hive. It can execute Hive QL queries up to 100 times faster than Hive without any modification to the existing data or queries. Shark supports Hive's query language, metastore, serialization formats, and user-defined functions, providing seamless integration with existing Hive deployments and a familiar, more powerful option for new ones. ALv2 University of California, Berkeley
Publications BlinkDB Infrastructure 2014-07 https://github.com/sameeragarwal/blinkdb.git stats BlinkDB is a massively parallel, approximate query engine for running interactive SQL queries on large volumes of data. It allows users to trade-off query accuracy for response time, enabling interactive queries over massive data by running queries on data samples and presenting results annotated with meaningful error bars. To achieve this, BlinkDB uses two key ideas: (1) An adaptive optimization framework that builds and maintains a set of multi-dimensional samples from original data over time, and (2) A dynamic sample selection strategy that selects an appropriately sized sample based on a query's accuracy and/or response time requirements. We have evaluated BlinkDB on the well-known TPC-H benchmarks, a real-world analytic workload derived from Conviva Inc. and are in the process of deploying it at Facebook Inc. ALv2 University of California, Berkeley
Publications Mesos Infrastructure 2014-07 https://git-wip-us.apache.org/repos/asf/mesos.git stats Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It can run Hadoop, MPI, Hypertable, Spark, and other applications on a dynamically shared pool of nodes. ALv2 University of California, Berkeley
Publications Tachyon Infrastructure 2014-07 https://github.com/amplab/tachyon.git stats Tachyon is a fault tolerant distributed file system enabling reliable file sharing at memory-speed across cluster frameworks, such as Spark and MapReduce. It achieves high performance by leveraging lineage information and using memory aggressively. Tachyon caches working set files in memory, and enables different jobs/queries and frameworks to access cached files at memory speed. Thus, Tachyon avoids going to disk to load datasets that are frequently read. BSD University of Southern California
Publications goffish Infrastructure 2014-07 https://github.com/usc-cloud/goffish.git stats The GoFFish project offers a distributed framework for storing timeseries graphs and composing graph analytics. It takes a clean-slate approach that leverages best practices and patterns from scalable data analytics such as Hadoop, HDFS, Hive, and Giraph, but with an emphasis on performing native analytics on graph (rather than tuple) data structures. This offers an more intuitive storage, access and programming model for graph datasets while also ensuring performance optimized for efficient analysis over large graphs (millions-billions of vertices) and many instances of them (thousands-millions of graph instances). ALv2

Instapainting Turns Your Photos Into Hand-Painted Oil Paintings On The Cheap | TechCrunch

Fri, 14 Feb 2014 13:16:35 -0800

Comments: "Instapainting Turns Your Photos Into Hand-Painted Oil Paintings On The Cheap | TechCrunch"

URL: http://techcrunch.com/2014/02/14/instapainting-turns-your-photos-into-hand-painted-oil-paintings-on-the-cheap/

Surprise! It’s Valentine’s Day, the stealthiest of all the holidays. Sneaks up on you, doesn’t it?

If you’re trying to get a gift today, you… might be a bit short on options. Will you go with the gas station teddy bear? The twice-crushed box of chocolates? A bouquet of acceptable-looking roses for $200?

If your nearly-forgotten flame would be content with the promise of a pretty cool gift in a few weeks, though, you might be set. Instapainting, a YC-backed company launching this morning, turns any photo into a hand-painted piece on canvas for under $100 bucks.

If you’ve ever tried to have something like this done before, you probably know: this exists. A few companies have been doing the whole photo-into-art thing for years. Where Instapainting thinks they have them beat, however, is in pricing and speed.

Instapainting’s smallest option (a 12″x12″ canvas) starts the pricing at $53 (including shipping), with the largest option (29.33″x22″) going for $130 . A quick search turns up a number of others in this space — OilPaintingExpress, OilPaintings.com, and myDavinci to name a few. The next wallet-friendliest option I could find was OilPaintingExpress, where a 12″x12″ work starts at $119. Most of them start the pricing at $200-$300 dollars.

Instapainting’s website is also a bit more… modern, for lack of a better word. Setting up your order takes all of a few seconds; upload your photo, crop it to the region you like, pick a canvas size, and you’re set. Built on top of tools like FilePicker and Stripe, the whole ordering flow is slick and simple.

So how do they keep the prices down? A few ways:

Your original photo is printed onto canvas first, and this printed piece is used as the base/foundation of the hand painted piece. In other words: they’re painting on top of the photo. The artist still has to know how to properly mix colors and how to recreate lights/shadows/etc. in oil, but it’s a whole lot quicker than starting on blank canvas. Many a professional artist might balk at the idea — but unless your friends start scratching at the paint to see what’s underneath, they probably won’t be able to tell.
As you might’ve guessed, much of the work is done overseas. Instapainting’s founders source their painters (primarily in China) one-by-one, mostly through their myriad online profiles. After quietly starting to roll the service out around a month ago, Instapainting says they have just shy of 100 painters producing pieces.
They ship your art rolled in a tube, leaving it to the customer to frame it or stretch it onto canvas. The company tells me they’re working on a quick-assembling canvas frame that they can pack into the shipping tubes, but that’s still a few months out.

But what about shoddy work? Cheaper rarely means better, after all. To keep quality up, Instapainting puts two layers of protection into the mix: first, each painting is checked by a second set of eyes before it heads out to the customer. Second, they guarantee their work; if you don’t dig the oil-painted version they send you, they’ll remake it or give you a full refund.

Meanwhile, the company is also dabbling with the idea of being a marketplace for artists looking to have their work recreated by hand. Artists upload the digital version of their painting or photograph, and Instapainting recreates their work and shares the revenue. It’s not quite the same as buying an original piece by the original artist, of course — but when your main concern is how it looks hanging above your couch, it’s a nice alternative to buying a standard print.

We’re planning on putting the just-launched service through the proper paces, so be on the lookout for a full review in the coming weeks.

Bitcore

Fri, 14 Feb 2014 13:46:44 -0800

Comments: " Bitcore "

Venezuelans Blocked on Twitter Amid Opposition’s Protests - Businessweek

Fri, 14 Feb 2014 15:30:39 -0800

Comments: "Venezuelans Blocked on Twitter Amid Opposition’s Protests - Businessweek"

URL: http://www.businessweek.com/news/2014-02-14/twitter-says-venezuela-blocks-its-images-amid-protest-crackdown

Twitter Inc. (TWTR:US) said the Venezuelan government blocked users’ online images as opposition groups marched through Caracas for a third day, demonstrating against record shortages and the world’s fastest inflation.

Nu Wexler, a Twitter spokesman, confirmed in an e-mail that the government was behind the disruption. President Nicolas Maduro banned protests Feb. 12 and has asked his supporters to counter with a “march against fascism” tomorrow, in a week of social unrest that has left at least three Venezuelans dead.

Related:

Boyâ€™s Life Hanging on 8-Hour Trip Shows Why Venezuelans Protest
Opinion: Let's Watch Venezuela Destroy Itself

In the absence of information from the government or local television outlets, Venezuelans have turned to foreign reporters and social media for news. Twitter users had been posting their photos of demonstrations that started in provincial towns earlier this month, providing an alternative to state-controlled media. It’s unclear if photos are blocked for users of all Internet providers in Venezuela, Wexler said.

“We won’t cease protests until all our comrades are free,” Gaby Arellano, a leader of the Andes University student union and member of the opposition party Voluntad Popular, said by telephone today. “We will continue fighting for our democratic rights.”

Maduro ordered police on Feb. 12 to detain opposition leaders for inciting violence after clashes between opposition-affiliated students and armed pro-government socialist collectives left 66 people injured and 118 under detention, according to the Interior Ministry.

Cantv Denial

State-run phone company Cia. Anonima Nacional Telefonos de Venezuela SA, or Cantv, denied blocking the images of San Francisco-based Twitter. Twitter’s servers are outside of Venezuela, and other countries experienced the same issue, the company said in an e-mailed statement today.

A spokesman for the Information Ministry, who can’t be named because of internal policy, said he had no problem seeing pictures on his Twitter account, which he uses to follow friends and family.

Students have defied Maduro’s ban on protests, blocking Caracas’ major avenues today on the way to the Eastern Cemetery to bury student Roberto Redman, one of the three killed in Feb. 12 clashes.

Only 10 of the 118 people detained have been charged to-date, mostly for instigating violence, according to the prosecutor general’s office. Human rights group Venezuelan Penal Forum put the number of detained at 148.

‘No Idea’

Alex Matute, a Caracas-based web developer, said his brother Angel, a 24-year-old student, was among the 30 protesters who were held by the National Guard without being charged and without access to lawyers or family for two days, before being driven to courts.

“We have no idea what they plan to do with him,” Alex Matute said by telephone today. “The police won’t tell us anything.”

A National Guard spokesman, who can’t be named because of internal policy, and Interior Ministry spokesman Marco Hernandez declined to comment, citing national-security risks.

Maduro has accused international outlets of bias. He took Colombian station NTN24 off the air after the protests and in a national address yesterday criticized Agence France Presse for “manipulating information.”

‘Massive Attack’

In November, Maduro asked authorities to investigate a disappearance of 6,000 of his 1.4 million Twitter followers, calling it a “massive attack” by the “international right wing.”

Billy Vaisberg, who runs an online directory of Venezuelan Twitter users called TwVen.com, said he had received several reports today from people who couldn’t see images on their feeds. In a post on its Spanish-language account, @twitter_es, Twitter advised Venezuelan users to subscribe to its text-message service to get updates.

“We are having a media blackout,” Josefina Blanco, a freelance science journalist and social media user, said in an e-mail from Caracas. Only because of Twitter, NTN24 and radio station RCR 750, “we can know what is really going on in our streets,” she said.

Inflation more than doubled in Venezuela in the past year to 56.3 percent in January, according to the central bank. At the same time, the central bank’s scarcity index rose to a record 28 percent, meaning that more than one in four basic goods was out of stock at any given time.

Bonds Fall

Amid the threat of new social unrest, the South American country’s benchmark dollar bond due in 2027 fell 0.67 cent to 64.85 cents on the dollar, near a 30-month low. The yield on the bond rose 16 basis points to 15.53 percent, at 3:40 p.m. in New York.

“The opposition has made a huge progress in the past week, as a couple of student protests have reignited the movement,” David Smilde, a senior fellow at the Washington Office on Latin America, said by telephone from Caracas yesterday. “If in the coming months the economy gets substantially worse and the protests continue, Maduro will be in tough position.”

To contact the reporters on this story: Patricia Laya in Mexico City at playa2@bloomberg.net; Sarah Frier in San Francisco at sfrier1@bloomberg.net; Anatoly Kurmanaev in Caracas at akurmanaev1@bloomberg.net

To contact the editors responsible for this story: Andre Soliani at asoliani@bloomberg.net; Nick Turner at nturner7@bloomberg.net

Sublimall

Fri, 14 Feb 2014 16:29:02 -0800

Comments: "Sublimall"

95% of Climate Models Agree: The Observations Must be Wrong « Roy Spencer, PhD

Fri, 14 Feb 2014 17:29:48 -0800

Comments: "95% of Climate Models Agree: The Observations Must be Wrong « Roy Spencer, PhD"

URL: http://www.drroyspencer.com/2014/02/95-of-climate-models-agree-the-observations-must-be-wrong/

I’m seeing a lot of wrangling over the recent (15+ year) pause in global average warming…when did it start, is it a full pause, shouldn’t we be taking the longer view, etc.

These are all interesting exercises, but they miss the most important point: the climate models that governments base policy decisions on have failed miserably.

I’ve updated our comparison of 90 climate models versus observations for global average surface temperatures through 2013, and we still see that >95% of the models have over-forecast the warming trend since 1979, whether we use their own surface temperature dataset (HadCRUT4), or our satellite dataset of lower tropospheric temperatures (UAH):

Whether humans are the cause of 100% of the observed warming or not, the conclusion is that global warming isn’t as bad as was predicted. That should have major policy implications…assuming policy is still informed by facts more than emotions and political aspirations.

And if humans are the cause of only, say, 50% of the warming (e.g. our published paper), then there is even less reason to force expensive and prosperity-destroying energy policies down our throats.

I am growing weary of the variety of emotional, misleading, and policy-useless statements like “most warming since the 1950s is human caused” or “97% of climate scientists agree humans are contributing to warming”, neither of which leads to the conclusion we need to substantially increase energy prices and freeze and starve more poor people to death for the greater good.

Yet, that is the direction we are heading.

And even if the extra energy is being stored in the deep ocean (if you have faith in long-term measured warming trends of thousandths or hundredths of a degree), I say “great!”. Because that extra heat is in the form of a tiny temperature change spread throughout an unimaginably large heat sink, which can never have an appreciable effect on future surface climate.

If the deep ocean ends up averaging 4.1 deg. C, rather than 4.0 deg. C, it won’t really matter.

Why I Dropped Out Of YC | wikichen

Fri, 14 Feb 2014 17:52:28 -0800

Comments: "Why I Dropped Out Of YC | wikichen"

URL: http://wikichen.is/writing/why-i-dropped-out-of-yc/

I bought my first Moleskine by sheer coincidence while shopping for craft supplies that a design course had required; I had avoided buying a notebook for the first two years at university by resourcefully (for which some might argue was born out of necessity from the lazy frugality of your average college kid) removing stacks of printer paper from the student center under a well-rehearsed, convincing ruse that I was somehow heroically fixing a “paper jam”.

By my third year I was making enough money from being a resident advisor and scrapping by with a delightful hodgepodge of research jobs that I could finally renounce my thieving ways and afford some decent stationeries, one of which was a proper notebook on which I could keep notes. I’m going to be taking actual notes in an actual notebook like a Shakespearean boss, I thought as I browsed the notebook section in the art supplies store. I picked the most conventional looking one from the pile: a large Moleskine Classic Notebook with a hard cover and plain, white pages. I knew not what a Moleskine was then or of its more hipster connotations (I swear) or why it cost an arm and leg for one, but I was pleased to have something I could, at long last, take notes on that’s not single sheet and A4-sized.

Naturally, those notes turned out to be nothing but the wild swirling caricatures of a sleep-deprived procrastinator nodding off every few minutes as the professor droned on. But over time that first Moleskine served as an outlet for my creative self, a blank canvas I painted with my thoughts and ideas and dreams. It captured the moments of blinding clarity and paroxysms of fleeting epiphany that so often defined checkpoints of my personal growth. More importantly, the notebook embodied me at my best and dutifully provided a record from which I could reflect and study.

I filled up that very same Moleskine at the end of last year, around the time when I decided to join the startup as she made the transition to the Bay Area after being accepted into Y Combinator. The fortuitous signs of a full notebook greeting a new opportunity in a new year got the best of my superstitious tendencies, so I ordered a brand-spanking new one off Amazon. In the five or so weeks since Y Combinator commenced in full swing, I haven’t written anything in the Moleskine. In fact, it’s still sitting on my shelf, wrapped in its original packaging.

Earlier this week we met with our advisor and YC partner Aaron over what seemed to be a divergence in the direction of the company and the underlying disagreement on how to best solve problems. In the hour we talked, what surfaced instead was an irreconcilable working relationship rooted in fundamental differences in personalities and misaligned expectations of roles and responsibilities. There were no hard feelings, just hard lessons learned.

Maybe I signed up for the wrong thing or for the wrong reason, as well-intentioned as I was from the start. Maybe it was more about Y Combinator, the subject of numerous pages in my Moleskine, and that was wrong. Maybe in the wake of a failed startup I deluded myself into thinking great teams are predicated on great friendships, a truism that no longer holds absolute truth.

It was also the acceptance that the startup would never be something I could call my own, not just in stake but in equal say, that cemented my decision to walk away. A sense of ownership can be a powerful motivator for a startup; in this case the lack thereof tore it apart. But if you asked me for the one reason why it didn’t work out, the truth is that it just wasn’t fun anymore, and when you’re grinding away on an early stage venture with no pay, the enjoyment you derive from working with your partners is all you have.

Egos clashed and convictions shaken, we came to the consesnsus that it’d be best if I part ways with the startup. To some extent I was relieved, freed from an incompatible environment that brought no party joy, and it hardly mattered that, to those like me who held the institution on a pedestal, this is Y Combinator we’re talking about. Aaron put it this way, “Stop thinking about YC. YC is an artificial construct, a forcing function that augments a startup’s chances to succeed. But if you can’t work together, then none of it matters. If someone tells me I can do YC but I’ll have to wear hot pants the whole time, I wouldn’t do it.” And I wholeheartedly agree with him: life’s too short to wear hot pants.

It’s been a few days since my departure, and I’m as uncertain as ever regarding my next steps, largely given that I’m still digesting what I’ve learned from this experience, and partly because I’ve grown jaded and disillusioned with Silicon Valley at large. What’s certain is that I won’t be touching startups for a very long time, and I might even stay away from Hacker News for a while. If anything, when the time comes and that startup itch manifests itself again, I’ve taken to heart that it will be of my own creation, with a team I can entrust equal ownership, and on my own terms, Y Combinator or not.

Taking the new Moleskine from the shelf and removing it from its plastic wrap, I turned to the first blank page and picked up my pen.

Here we go again.