Hacker News 100

Chris's Wiki :: blog/linux/SystemdWhyItWon

Tue, 11 Feb 2014 19:29:20 -0800

Comments: " Chris's Wiki :: blog/linux/SystemdWhyItWon "

URL: http://utcc.utoronto.ca/~cks/space/blog/linux/SystemdWhyItWon

Recently, an article by Rich Felker called Broken by design: systemd has been making the rounds. I have a number of things to say about this article but today I want to talk about one specific issue it brings up, which is systemd's novelty (or lack thereof) and why it is succeeding. To start with, here is the relevant quote from Felker's article:

None of the things systemd "does right" are at all revolutionary. They've been done many times before. DJB's daemontools, runit, and Supervisor, among others, have solved the "legacy init is broken" problem over and over again (though each with some of their own flaws). Their failure to displace legacy sysvinit in major distributions had nothing to do with whether they solved the problem, and everything to do with marketing. [...]

This is wrong on several levels. To start with and as usual, social problems are the real problems. In specific, none of these alternate init systems did the hard work to actually become a replacement init system for anything much. Anyone can write an init system, especially a partial one (I did once, long ago). Getting it adopted by people is the hard part and none of these alternatives tackled that effectively (if they did so at all, and some of them certainly didn't). And as Felker admits, each of these theoretical alternatives have flaws of their own.

(Note that this is not a criticism of those alternate init systems. I don't think any of them have really been developed with replacing SysV init in Linux distributions or elsewhere as a goal. DJB daemontools certainly wasn't; I believe that DJB's attitude towards it, as towards more or less everything he's developed, can be summed up as 'I showed you the way, what you do with it is up to you'.)

The reason systemd has succeeded in becoming an SysV init replacement is simple: it did the work. Not only did it put together a lot of good ideas regardless of their novelty or lack thereof but its developers put in the time and effort to convince people that it was a good idea, the right answer, a good solution to problems and so on. Then they dealt with lots and lots of practical concerns, backwards compatibility, corner cases, endless arguments, and so on and so forth. I want to specifically mention here that one of the things the systemd people did was write extensive documentation on systemd's design, how to configure and operate it, and what sorts of neat things you can do with it. While this documentation is not perfect, most init systems are an order of magnitude less well documented.

(I am sure that in some quarters it's popular to believe that Lennart Poettering bulldozed the Fedora technical people into adopting his new thing. I do not think that the Fedora technical people are that easily overrun (or that impressed by Poettering, especially after PulseAudio), and for that matter at least some of the Debian technical people feel that systemd is the best option despite having looked deeply at the alternatives (cf).)

You can call this marketing if you want, although I don't think that that's a useful label for what is really happening. I call this 'trying' versus 'not trying'. If you don't try hard and work hard to become a replacement init system, it should be no surprise when you don't.

(In particular, note that SysV init is not a particularly bad init system so it should be no surprise when it is not particularly easy to displace.)

Beyond that I have some degree of experience with one of these alternate init systems, specifically DJB daemontools, and I've looked at the documentation for the other two. Speaking as a system administrator, systemd solves my problems better. The authors of systemd have looked at problems that are not solved by SysV init and come up with real solutions to them. Many of these problems are not solved by any of the alternatives that Felker put forward. In specific, often the alternatives assume (or require) cooperative daemon processes in order to fully realize their benefits; systemd is deliberately designed so that it does not and can fully manage even existing obstreperous Unix daemons with their willful backgrounding and other inconvenient behaviors.

(I don't know the field of Linux and Unix init-like systems well enough to say whether or not features like socket activation and clever use of control groups are genuinely novel in systemd or simply the first time I've become aware of them. They do feel novel.)

Since that may not be clear, let me be plain: systemd is a better init system than the alternatives. It does more to solve real problems and it does it better. That alone is a good reason for it to win in the practical world, the one where people care about getting stuff done. That systemd is not necessarily novel or the first to come up with the ideas that it embodies is irrelevant to this. Implementation matters more than ideas.

(Arguably it's an advantage that systemd feels no urge to reinvent different wheels when perfectly decent ones exist.)

PS: Please note that the reason that Unix itself succeeded is not its ideas alone, it is that Unix implemented them very well. A number of Unix's ideas are both great and novel, but a bad implementation would have doomed the whole enterprise. The fate of good ideas with a bad implementation is to be reimplemented elsewhere, cf the Xerox Alto and for that matter the Apple Lisa.

PPS: Also note that the one serious competitor to systemd is Upstart, which is also the product of a great deal of work and polishing.

5 Ways To Burn Out Programming

Tue, 11 Feb 2014 21:28:31 -0800

Comments: "5 Ways To Burn Out Programming"

URL: http://blog.braegger.pw/5-ways-to-burn-out-programming/

I've only recently come out of my burnout, despite it happening years ago. It sucks. It sucks bad. But looking back, I can see many of the causes crystal clearly, that weren't so apparent at the time. Here's a list:

1. Think about your project and only the project

Let's face it. Business wants you to make the best product you can "for our customers". You put off fun features for the sake of hitting a deadline. You plan and analyze and break a project into sets of deliverables that then must be coded by a monkey (you). You demo it, gather feedback, iterate. All without thinking anything for yourself.

But newsflash: you started programming because you thought it was fun, why not keep programming because it's fun? Take that little extra time to put in a feature you want. Challenge yourself a little bit in doing something you didn't think you could. Show it to everyone you know, and don't just ask for feedback, but brag about what you've done.

2. Have a negative attitude toward everything.

You know Docker? It sucks. Who would trust their production environment to a new, unstable, toy. Go? Do I look like I want to write every library myself? Everything I need is already in PyPI. This project I'm working on is so caught up in office politics, it's never going to work. Jenkins? 2008 wants their tech back.

It's really easy to fall into the "being critical" trap. It's easy to tell other people what the "wrong" choice is. I imagine it's because as software engineers, our job is to find faults in our applications and fix them. And if we don't find them, someone else finds them for us.

But I don't think we need to be negative about our job, decisions that are being made (even if it's not our decision) and what we're working on. Some of the best projects I've worked on worked out that way because we had a great, positive team. We enjoyed showing up every day to work, told each other when we did awesome things, held back heavy-handed criticism and phrased it in a productive manner.

So you're an uber expert in Java + Spring + Hibernate. Nobody can touch your python skillz. Every personal project you do should be in these, because all that matters is the business side of things, right?

Wrong.

While it definitely makes good business sense, you should prototype, play around, and become an expert in new tech, even if it's unvetted. While this might seem like obvious advice (it's repeated alllll the time), it becomes a lot harder to do as you grow more experienced.

4. Switch jobs often

Otherwise known as "chasing butterflies". Getting bored with what you're working on? Have an itch? Time to dust off that resume!

This is bad, bad, bad.

When you have several short employments, it can usually help boost your salary quite a bit, but you are robbing yourself of:

Growing in the company (developer -> manager -> director) Gaining an expertise in a specific area. Considering it takes 4-6 years for a PhD student to get their PhD, that's a lot of time you need for learnin. You are having to start from scratch often. If you are a good developer, you have to "prove" yourself (people listen to you) all over again.

So how do these contribute to burnout? Your career stagnates, you don't develop your skills as deeply (only breadth), people dont trust you'll stay employed for a while, and you're constantly having to prove yourself.

5. Work long hours, ignore your life

"You don't have to work a lot of hours, but some people choose to." You want to impress your boss. Hell, you want to impress yourself. So you go die-hard to meet an impossible deadline. You delivered the project on time, with all the extra features you wanted. You are the hero. High fives all around. And if you're lucky, you'll get that bonus.

That's great the first time. But how about the second. And the third. It's a bomb, and you dont know how short the fuse is.

Summary

In short, it's easy to burnout. Do these 5 things, and you can burnout too.

The Day the Internet Didn’t Fight Back

Wed, 12 Feb 2014 00:31:03 -0800

Comments: "The Day the Internet Didn’t Fight Back"

URL: http://bits.blogs.nytimes.com/2014/02/11/the-day-the-internet-didnt-fight-back/

IBM layoffs strike first in India; workers describe cuts as 'slaughter' and 'massive' :: Editor's Blog at WRAL Tech Wire

Wed, 12 Feb 2014 02:45:00 -0800

Comments: "IBM layoffs strike first in India; workers describe cuts as 'slaughter' and 'massive' :: Editor's Blog at WRAL Tech Wire"

URL: http://wraltechwire.com/ibm-layoffs-strike-first-in-india-workers-describe-cuts-as-slaughter-and-massive-/13379415/

Updated Feb. 12, 2014 at 7:49 a.m.

IBM layoffs strike first in India; workers describe cuts as 'slaughter' and 'massive'

Published: 2014-02-11 11:34:00
Updated: 2014-02-12 07:49:18

By RICK SMITH, WRALTechWire Editor

Research Triangle Park, N.C. — Dispatches from the firing line at IBM:

"People broke down after seeing the inhuman treatment. Laptops along with the cases were confiscated, so several employees were seen crying and exiting building carrying and balancing their personal belongings with their two hands."
"IBM STG INDIA is doing RA and it is very deep and numbers are huge."
"Just heard from a colleague in Bangalore that job cuts there have begun. Workers asked to leave on the spot. He claims 6 out of 23 people in his department."

RESEARCH TRIANGLE PARK, N.C. - IBM's latest $1 billion "rebalancing," as described by its CFO last month, is underway. And the first country hit is India, based on reports from Blue Blue workers there on Tuesday. The so-called "Resource Action" struck in the country where IBM reportedly employs its greatest number of workers.

One employee describes the layoff process as a "slaughter."

The business unit hit is the Systems Technology Group - IBM's troubled hardware group which is selling its low-end x86 server business to Lenovo for $2.3 billion.

Workers were given little notice. Reads one note just sent to the union seeking to represent IBM workers:

"Job cuts in India STG. .Announced today including managers.Asked to return laptops with in 2 hrs and leave premises."

Other notes as cited below were much more detailed. All reflect sudden notices - and quick departures, with many reportedly being quite emotional.

One analyst has estimated that IBM will cut 13,000 of its more than 434,000 workers, based on the amount of money set aside for the rebalancing IBM disclosed after another quarter of disappointing earnings. A similar action in 2013 led to some 3,500 job cuts in North America alone, with several hundred hitting IBM's North Carolina work force.

Lee Conrad, head of union efforts to unionize IBMers, has received emails from workers as well as a reporter in India. The "RA" is expected to hit IBMers in North America as early as Feb. 19, based on internal speculation.

"Still waiting here in the U.S.," Conrad wrote.

IBMers have described the rebalancing as "Project Apollo." The 2013 action was called "Project Phoenix."

WRALTechWire has reached out to IBM for comment but as of early Wednesday had received no response.

Given IBM's importance to the company both as a market for its business and the number of employees based there, Conrad said the cuts starting first in that country came as an unexpected development.

"I am a bit surprised, but given it is STG it was expected," he said.

One email from India is printed in full in order to capture the atmosphere of what took place. Acronyms refer to various IBM work units, except TP which is believed to refer to employee ThinkPad computers. Bangalore is the headquarters for IBM's India-based operations.

"Slaughter House"

"STG Bangalore literally turned into a slaughter house today.

"Several employees were called to a meeting and RA'd.

"Their TPs were confiscated and they were asked to vacate premises immediately.

"Severance package was on an average 3 months basic component of salary, which is like 6 weeks full pay.

"RA per department as on today

SRDC 40%
Processors 15%
SRAM 80%
SSE Just began, final numbers not available

"RAs expected to last till friday.

"The fear is that HCM might be wiped off totally in a day or two. EDA and methodology numbers not yet available"

"People broke down after seeing the inhuman treatment.

"Laptops along with the cases were confiscated, so several employees were seen crying and exiting building carrying and balancing their personal belongings
with their two hands"

Two posts at the Alliance website also referenced layoffs at Bangalore, which is known as India's "Silicon Valley."

IBM employs more than 100,000 people across India, according to India media reports. IBM no longer discloses employee numbers by location.

"RAs Started in Bangalore"

Three other posts give insight into what's happening in Bangalore:

"IBM today had a massive layoff in STG bangalore more than 40% staff was let go off in a single day. Be it PBC 1 or 2 doesnot matter you are just asked to leave IBM premises by immediate effect. I fail to understand how joB cut will help management to achieve 20 EPS. The upper executives lack vision and clarity to restructure business process. Good bye to ibm hopefully will be in better place than ibm."

(Note: PBC refers to personal evaluations. EPS references IBM's target of delivering $20 per share in earnings by 2015.)

"RAs started in STG Bangalore, hearing that large numbers impacted. Will keep you posted on the details as i get them. the job market is reasonably good outside and the average experience here is about 6-7 years.people are not too worried. most were anyway sick of the company and its junk policies. -Bangalore RAs started-"
"People were reacting to the sale of semiconductor business news rather harshly, well wishers were asking me when was I planning to leave IBM, and the others sarcastically asked me what am I still doing here in STG. Felt very embarrassed walking in the corridor today. The general perception here is that if you are still with IBM, it is because you did not find a job elsewhere. Phew !! that just means to say IBM STG is being considered the worst place to be at this time. We all at STG India would like to express solidarity with the IBM USA work force preparing for yet another RA in the coming weeks. Rest assured we do not feel any better being employed with IBM either. The flood gates are opening and I anticipate that we will lose all our above average work force in the next 3 months. Good luck to you all, I am sure there is life outside of IBM. bye for now, time to update my resume, keep in touch. -STG INDIA-"

The second poster is referring to reports that IBM is looking to sell or find a partner for its semiconductor business.

IBM still employs some 9,500 people across North Carolina. Of those, some 2,000 would be transferred to Lenovo if government regulators approve the x86 deal.

WRAL Tech Wire any time: Twitter, Facebook

Hemingway

Wed, 12 Feb 2014 05:29:50 -0800

Comments: "Hemingway"

I challenged hackers to investigate me and what they found out is chilling | PandoDaily

Wed, 12 Feb 2014 05:07:49 -0800

Comments: "I challenged hackers to investigate me and what they found out is chilling | PandoDaily"

URL: http://pando.com/2013/10/26/i-challenged-hackers-to-investigate-me-and-what-they-found-out-is-chilling/

By Adam L. Penenberg
On October 26, 2013

It’s my first class of the semester at New York University. I’m discussing the evils of plagiarism and falsifying sources with 11 graduate journalism students when, without warning, my computer freezes. I fruitlessly tap on the keyboard as my laptop takes on a life of its own and reboots. Seconds later the screen flashes a message. To receive the four-digit code I need to unlock it I’ll have to dial a number with a 312 area code. Then my iPhone, set on vibrate and sitting idly on the table, beeps madly.

I’m being hacked — and only have myself to blame.

Two months earlier I challenged Nicholas Percoco, senior vice president of SpiderLabs, the advanced research and ethical hacking team at Trustwave, to perform a personal “pen-test,” industry-speak for “penetration test.” The idea grew out of a cover story I wrote for Forbes some 14 years earlier, when I retained a private detective to investigate me, starting with just my byline. In a week he pulled up an astonishing amount of information, everything from my social security number and mother’s maiden name to long distance phone records, including who I called and for how long, my rent, bank accounts, stock holdings, and utility bills.

The detective, Dan Cohn, owned and operated Docusearch, a website that trafficked in personal information, and at the time, he was charging $35 to dig up someone’s driving record, $45 for his bank account balances, $49 for a social security number, $84 to trace a mobile number, and $209 to compile his stocks, bonds, and securities. The site offered a simple clickable interface and Amazon-like shopping cart. It’s still around today, boasting similar services. “Licensed Investigators for Accurate Results” reads the tag line, calling itself “America’s premier provider of on-line investigative solutions.”

For Cohn, digging through what I had assumed was personal information, was less challenging than filling in a crossword puzzle. He was able to collect this amalgam of data on me without leaving the air-conditioned cool of his office in Boca Raton, Florida. In addition to maintaining access to myriad databases stuffed with Americans’ personal information, he was a master of “pre-texting.” That is, he tricked people into handing over personal information, usually over the telephone. Simple and devilishly effective. When the story hit newsstands with a photo of Cohn on the cover and the eerie caption: “I know what you did last night,” it caused quite a stir. It was even read into the Congressional Record during hearings on privacy.

All it takes is a person or persons with enough patience and know-how to pierce anyone’s privacy — and, if they choose, to wreak havoc on your finances and destroy your reputation.

A decade and a half later, and given the recent Edward Snowden-fueled brouhaha over the National Security Agency’s snooping on Americans, I wondered how much had changed. Today, about 250 million Americans are on the Internet, and spend an average of 23 hours a week online and texting, with 27 percent of that engaged in social media. Like most people, I’m on the Internet, in some fashion, most of my waking hours, if not through a computer then via a tablet or smart phone.

With so much of my life reduced to microscopic bits and bytes bouncing around in a netherworld of digital data, how much could Nick Percoco and a determined team of hackers find out about me? Worse, how much damage could they potentially cause?

What I learned is that virtually all of us are vulnerable to electronic eavesdropping and are easy hack targets. Most of us have adopted the credo “security by obscurity,” but all it takes is a person or persons with enough patience and know-how to pierce anyone’s privacy — and, if they choose, to wreak havoc on your finances and destroy your reputation.

I’ve never actually met Nick Percoco, which, for all he knows about me, might seem strange.

Earlier this year I contacted him to pen a guest post for PandoDaily. In it, Percoco warned that unscrupulous people could potentially intercept your private messages and inject malevolent code into your computer over a coffee shop’s Wi-Fi. I liked how he wrote the piece. He didn’t hype the threat. Instead he laid out the facts, relayed some anecdotes from his work, and offered basic, actionable prescriptions.

You can tell a lot about a person by the way he writes. As a journalism professor, I get to know my students’ writing better than they know it themselves. And Percoco, through his prose, struck me as someone who was smart, well informed on security issues, and careful with what he said and how he said it. “Comp-sec,” as it’s called – short for computer security – is rife with charlatans. It often seems the more fame someone accrues in that world, the less he’s accomplished and even less he knows.

For this particular job, trust would be vital. If I were to invite someone to wheedle his way into my life, sneak into my finances, sniff my email, capture my web surfing, maybe even break into my home, I had to be damn sure he and the people he worked with wouldn’t use this information for nefarious purposes. I checked up on Percoco and couldn’t find anything that reflected badly on his character.

Over the years [Percoco] has performed hundreds of pen-tests and physical break-ins, slipping into hospitals, insurance companies, manufacturers, magazine and newspaper companies, power companies, and many more.

Percoco, 38, considers himself a white hat hacker, and has been breaking into companies (with their blessing) for 14 years. In what is perhaps the perfect metaphor for what he does and who he is, he lacks recognizable fingerprints, a quirk of nature, he assures. Once in Colombia, he says, he was denied entry into a building because the turnstile, equipped with a fingerprint identification pad, couldn’t get a fix on his digits. Percoco prides himself on having the skills of a black hat hacker while maintaining what he calls the highest ethical standards.

Not only does he attack computer vulnerabilities, Percoco performs on-site intrusions. Over the years he has performed hundreds of pen-tests and physical break-ins, slipping into hospitals, insurance companies, manufacturers, magazine and newspaper companies, power companies, and many more – clients, he says, that he’s forbidden to reveal.

Once, he says, he was hired to gain access to a hospital’s computer systems housed in a data center. Wandering the hallways, he followed the signs until he saw one for the IT department. It led him to a server room behind a glass door. Inside there was a woman printing out patient records. All Percoco had to do was knock and she let him in, no questions asked. He ambled over to a computer with a mouse and in a few clicks logged on as the systems administrator. Now he had access to patient records, and could have, if he’d wanted, taken down the entire network. The hospital’s chief information officer had wanted more resources for security. He got them.

Percoco told me he was intrigued by my proposal because he and his team almost always investigate corporations, not individuals. He wondered aloud whether I would be easier or harder to attack than a corporation. Both he and I were eager to find out.

In 1999, detective Dan Cohn’s most powerful weapons were a telephone and unmitigated gall. True to his word, exactly one week after he started my investigation, he faxed me a three-page summary of my life. It began with my base identifiers – full name, date of birth, social security number, home address – which he obtained from my credit report. Companies like Equifax claim they have protections in place to prevent against fraudsters, but Cohn told me he went through a reseller.

Equipped with my credit header, Cohn had what he needed to access a Federal Reserve database listing my deposit accounts, some of which I had long forgotten – $503 at Apple Bank for Savings in an account held by a long-ago landlord as a security deposit; $7 in a dormant savings account at Chase Manhattan Bank; $1,000 in another Chase account. A few days later Cohn located my Merrill Lynch cash management account, which I had opened a few months earlier. He then had my checking and savings account balances, direct deposits from work, withdrawals, ATM visits, check numbers with dates and amounts, and the name of my broker. In addition to my finances, he also obtained utility bills and two unlisted phone numbers, which cataloged a bevy of long distance and local phone calls I had made.

Armed with this information, Cohn could have easily mapped out my routines. He knew how much cash I withdrew from ATMs each week, how much Forbes deposited into my checking account twice a month, the cafes and restaurants I frequented, the monthly checks I wrote to a shrink. He possessed my latest phone bill and a list of long distance calls to and from my home, including late-night fiber-optic dalliances with a woman I was dating and who worked for an advertising agency and traveled a lot. Cohn also divined phone numbers of a few of my sources, including a couple of computer hackers who had told me of their black hat activities.

While databases assisted him with my basic information, to secure the nitty-gritty detail of my life, he needed help, which he wrangled from the actual companies I did business with.

Part of the deal I struck with Cohn required him to tell me exactly how he did what he did, but he held back when it came time to pony up. To fill in the gaps I contacted my phone company (Bell Atlantic, now Verizon), long distance phone provider (Sprint), and bank (Merrill Lynch), telling them what Cohn had done and demanding an explanation. Each, in turn, launched an investigation. With the results I went back to Cohn, who confirmed the information and added additional detail.

Sprint informed me a Mr. Penenberg had called to inquire about my most recent bill. He posed as me, and had enough information to convince the customer service representative he was me. The caller had the operator run through the last couple of dozen calls I had made. It was a similar story with Bell Atlantic, only this time it was a Mrs. Penenberg who did the dirty deed.

He knew how much cash I withdrew from ATMs each week, how much Forbes deposited into my checking account twice a month, the cafes and restaurants I frequented, the monthly checks I wrote to a shrink.

With Merrill Lynch, Cohn also phoned customer service. This time, however, he was relatively upfront. “Hi,” he said, “I’m Dan Cohn, a licensed state investigator conducting an investigation of an Adam Penenberg.” Later Cohn told me official-sounding words like “licensed” and “state” make him sound legit, as if he worked in law enforcement. Then he reeled off my social security number, birth date and address, which he had gleaned from my credit report, and, he told me later, “before I could get out anything more he spat out your account number.”

Cohn wrote it down then told the helpful operator, “I talked to Penenberg’s broker, um, I can’t remember his name…”

“Dan Dunn?” the Merrill operator asked.

“Yeah, Dan Dunn,” Cohn repeated.

Merrill’s minion then recited my balance, deposits, withdrawals, check numbers and amounts. “You have to talk in the lingo the bank people talk so they don’t even know they are being taken,” Cohn said, obviously pleased with himself.

Such pretext calls are technically illegal under the Gramm-Leach-Bliley Act of 1999, at least if used to obtain financial data from individuals or financial institutions, but it’s rarely enforced and hard to catch.

But I needn’t have worried, Cohn assured me. He promised he would never resell the information to anyone else. “Unlike an information broker, I won’t break the law,” he told me. “I turn down jobs, like if a jealous boyfriend wants to find out where his ex is living.”

At the time, I thought this was an odd statement, strangely specific, which he had volunteered. What I didn’t know was that at the same time he was digging up dirt on me, Cohn was embroiled in a tragic case involving a stalker, who had paid Docusearch to locate his victim.

According to court documents, on July 29, 1999, New Hampshire resident Liam Youens paid Docusearch for the social security number, home and work addresses for 20-year-old Amy Lynn Boyer, another New Hampshire resident. Docusearch went through a subcontractor, Michele Gambino, who relied on pretexting. She called Boyer in New Hampshire, lying about who she was and why she was calling in a bid to trick Boyer into revealing her employment information. Gambino passed this information on to Docusearch, which provided it to Youens.

A week later Youens drove to the dentist’s office in Nashua, New Hampshire, where Boyer worked. He waited in ambush while she got in her car and drove up beside her. Leaning out of his car, he put the barrel against her window. He called her name so that she would look up.

Then he shot and killed her.

Seconds later he turned the gun on himself.

“Amy never knew it was coming,” her stepfather, Tim Remsberg, said in an interview with the tabloid news show, “48 Hours.”

Youens, who was unemployed and lived with his mother, had been stalking Boyer for years, chronicling his obsessions on a web site. On it, he confessed that he had fallen in love with her in 8th grade. Later, after Boyer rebuffed his advances, he decided she must die. On the website, “48 Hours” reported, he foretold how he would kill her: “When she gets in, I’ll drive up to the car blocking her in, window to window. I’ll shoot her with my Glock.”

Amy Boyer’s mother sued Docusearch, alleging that Cohn and his partner had invaded her daughter’s privacy and broke other laws when it assisted Youens in locating her while the online information broker claimed the information wasn’t private. After the case wound through the courts, the New Hampshire Supreme Court ruled that the lawsuit could proceed to a jury trial, and Cohn and Zeiss ended up settling with the family for a reported $85,000.

Afterward, Cohn promised, “Our policies and the way we do business has changed as a result.”

After Nick Percoco and I hammered out the broad outlines of our project – his team would not break any laws, and they would leave my kids out of this – I signed a waiver (courtesy of Trustwave’s lawyers) that barred me from suing the company if my information ended up in the wrong hands. Percoco kept the timetable vague and frankly, after a month dragged into two, I almost forgot about it. But his team, comprised of security analyst Garret Picchioni, digital forensics specialist Josh Grunzweig, and hacker Matthew Jakubowski (Jaku), were anything but idle.

Percoco didn’t tell me who my investigators would be, and even if he had told me in advance it wouldn’t have done me much good. Like most information security professionals who pen-test for a living, Picchioni and Grunzweig had taken steps to limit their online footprints. Google their names and you won’t find all that much, other than they have all given presentations at hacker conferences on highly technical topics.

Garret Picchioni’s Twitter bio says “Information Security Professional for {redacted}, Network Engineer, and resident pain in the ass” accompanied by a photo of South Park’s Cartman wearing a cheese hat. His LinkedIn profile also reveals little. He’s been in the information security business since 2004, authored an academic paper that analyzed more than 2.5 million anonymized passwords, took six years to graduate from the University of Arizona, where he majored in history and minored in information security). Meanwhile, SpiderLabs “has performed over a thousand incident investigations, thousands of penetration tests and hundreds of application security tests globally.”

Josh Grunzweig is even more stealthy. His Twitter bio is simply “malware reverser | beer drinker | hockey fan” and his LinkedIn profile barely qualifies as a profile. He graduated from Rochester Institute of Technology with a degree in Applied Networking and System Administration, and minored in criminal justice. Some activities he listed are information security, snowboarding, running, movies, music, traveling, and grabbing a drink with friends.

Of the three, Matthew Jakubowski, or “Jaku,” as he likes to be called, has the most Google juice. Last year he turned a dry erase marker into a tool that could pick a hotel lock in seconds flat. In the avalanche of media attention that followed, he revealed that he could steal credit cards wirelessly using a radio identification reader without your having to pull your Mastercard out of your wallet. His Twitter bio warns, “Neque dicas, quid neque,” which in Latin means “Don’t tell me what to do.” According to his scant LinkedIn profile Jaku majored in “Sandwich Engineering” and minored in “Witch Hunting” at “College University.”

Percoco told me they began the project by pulling up everything they could about me on the Web, sifting through my website and various writings, looking for anything that could point to potential vulnerabilities. They gleaned some interesting nuggets, including the type of computer I use (I’ve written that I’m an Apple aficionado), my home and work addresses (easily found through public records searches), and the location of the Pilates studio my wife, Charlotte, owns and operates. This helped them formulate a plan of attack.

Here’s the strategy they sketched out (from the confidential report I was provided afterward):

After doing some initial research on Adam and his family, a preliminary game plan was created before traveling onsite that included both technical and physical (security) attacks on Adam.

The initial rough plan is outlined as follows and included multiple attack vectors as contingency plans:

Just rereading that feels weird. Even though I brought this on myself, I still marvel at how many attack vectors someone like me can provide any would-be attacker. Substitute your name for mine; your wife’s, husband’s, or partner’s business for my wife’s; your office locations for mine. How would you feel?

SpiderLabs’ three-member team failed at some of these tasks, like 1) breaking into my apartment, 2) cracking the security on my Time Warner cable modem, and 6) gaining access to my computer and office at NYU. Sneaking into my home would have necessitated coming through neighbors’ apartments or trespassing through their yards, or climbing a fence at the courthouse down the street. One thing they did not want to do was violate any laws. Others, like 8) luring me to a malicious blog and 9) using my web designer to help them access my website, turned out to be unnecessary.

Still, what they did end up doing is impressive. They flew to New York on August 20th to stake out my home and immediately ran into problems that an urban environment can present. Brooklyn Heights is saturated with Wi-Fi networks. The team sniffed 1200 of them within a tenth of a mile radius of my brownstone. The fact they knew I use Apple computers narrowed it down somewhat, since they determine that through their canvassing. But they couldn’t identify, with any certainty, which specific Wi-Fi network was mine, and they could run afoul of the law if they intercepted traffic from someone else’s.

They then repaired to my wife’s Pilates studio, located 10 blocks away, and confronted similar wireless saturation. From the 2nd floor of a Barnes & Noble they could see through the studio’s side windows, which offers a limited view inside. While they had pictures of my wife they found online on her studio’s website – just typing that creeps me out – they couldn’t see her while she was working nor could they determine her work schedule.

One bench gave them a bird’s eye view of my front stoop if they looked through binoculars. From there they watched my wife and me come and go.

As a result, they hatched a plan for a “client-side attack.” A female friend of Jaku, the hacker, signed up for a Pilates group class at my wife’s studio. Since Charlotte only teaches private sessions at Streamline (although she runs a class at another studio she operates, but it has a private membership) the friend enrolled in an introductory class taught by one of the other instructors. Before leaving she left behind a large purple flash drive in a changing room. The SpiderLab’s team hoped an instructor would find it and plug it into the studio’s computer in an attempt to identify the owner. The flash drive held various payloads titled “Resume” but would actually install a remote backdoor on the system upon opening of the file and “phone home” to the team.

No one, however, plugged the thumb drive into the studio’s computer. A few days later Jaku took the decoy back to the Pilates studio for another session, this time equipped with another flash drive. After the class was over the decoy informed the instructor that she had a job interview shortly after class, and asked if she could print out her resume, which was located on the flash drive. What the team didn’t know was that the studio runs an old version of Apple’s operating system – so old, in fact, that the hacker program Jaku coded couldn’t execute its nefarious deeds.

Meanwhile, the team, back in front of my apartment, had to cope with nosy neighbors. I live in a city but my block is quiet and residential, home to many families. The SpiderLabs guys had a police scanner tuned to the local Brooklyn Heights precinct, just in case someone called the cops. Three men hanging around in front of my building, however, was bound to attract attention, and it did. While trying to secret a laptop computer behind a potted plant on my stoop in an attempt to try and isolate my Wi-Fi network, they spotted a woman in a red shirt glaring from a short distance away. Eventually she gave up. Another neighbor confronted the men as she was walking her dog, telling them she had noticed them hanging around the past few days. Picchioni, the team leader, finessed an answer, claiming they were from out of town, here on business, and wanted to work outside because it was such a nice day.

The SpiderLabs gang had been put on notice. They ended up renting a ZipCar and trawled around the front of my building by hiding in the back of the car and whiling away hours in a nearby park. One bench gave them a bird’s eye view of my front stoop if they looked through binoculars. From there they watched my wife and I come and go.

Around this time, I published a piece on PandoDaily about my experience with an iPad app that coaxes children into purchasing virtual crap if they want to progress in the game. I talked a bit about my own children’s screen habits and how they read insatiably on Kindle Paperwhites. I wrote: “I prefer ebooks to hardcovers and paperbacks because we live in Brooklyn and don’t have space for all the books they read. Our basement is packed with them. Feel free to come by and cart them away to your favorite library or charity.”

Shortly after my piece posted, a woman on Twitter asked if she could take me up on my offer. It was a real Twitter account, which, I learned later, belonged to a friend of Jaku’s. “We really wanted to get into your basement,” he later told me. Not suspecting anything, I responded that my wife and I would have to go through these books before we’d give any away.

Really, though, all this on-premises mishegas would turn out to be for naught. Like Dan Cohn, the team from SpiderLabs was able to get the information they sought through other means. Not with pretext calls, which are oh-so last century. Nick Percoco and his minions are children of the Internet, and have little need of a telephone. Instead, they know the art of the phish.

The first one they tried was a message to me from a student in Ohio who expressed interest in attending NYU to study journalism.

I read the email but didn’t open the attachment because it was a file type I didn’t recognize. I remember thinking why would a high school student send me an attachment with a JAR suffix? Plus, I was on break from teaching and filed the email away for the week after the semester would begin.

Since I didn’t reply the team took aim at Charlotte with a phish.

(Editor’s note: Amber, whose last name has been redacted from these images, is a real person — a Pilates instructor, in fact. The SpiderLabs team did what hackers often do, which was to use a real person’s identity in case Charlotte looked her up online. The email address, however, was fake. The real Amber contacted us after the story was published. We then added the redactions.)

When Charlotte didn’t respond, they re-sent Amber’s message, and at 4:30 p.m. ET on August 27, she clicked on the link and by doing so downloaded the malware that Jaku had coded especially for us.

The video didn’t work, so Charlotte sent a reply, telling Amber that while she couldn’t meet over Labor Day, she would like to see her resume, and said she couldn’t open the video clip.

There was, however, a bug in the malware (Jaku says this was his first time writing it for a Mac) and the SpiderLabs gang couldn’t maintain persistent access. So they replied to Charlotte’s reply. This time, instead of a web link the payload was a zip file:

The newly updated OSX malware, which another member of the team, digital forensics specialist Josh Grunzweig coded, was dropped on to her machine. SpiderLabs now had complete access to her laptop whenever it was on the Internet.

They got into our checking and savings accounts, a corporate bond account, our credit card statements and online bills. They could, if they had wanted to, wipe us out financially.

On Charlotte’s machine were our family’s W2s, which included our social security numbers as well as our income and all of our deductions, paperwork and copies of credit card and banking statements. They also came upon a password to our home router. More frightening, they discovered her password and log in to our Chase online banking account.

Chase.com uses a two-step verification system, which momentarily stymied SpiderLabs’ hackers. Every time she or I logs on from an IP address that Chase doesn’t recognize, it offers to send us an activation code via text to our mobile phones. But a search of Charlotte’s hard drive revealed Chase cookies, which the team copied and used to convince Chase that she was logging in from home. While inside they got into our checking and savings accounts, a corporate bond account, our credit card statements and online bills. They could, if they wanted to, have wiped us out financially.

What’s more, buried deep on the hard drive, they located something else: old files of mine. Some years earlier I had bequeathed Charlotte my old PowerBook G4 Titanium, and didn’t bother to wipe clean the hard drive. Months later I smelled acrid smoke in our apartment and saw that the keyboard was on fire. After I put out the flames, the laptop refused to boot up. (The motherboard had melted.) We brought it to the Apple Store and staff — I refuse to call them geniuses, except ironically — copied the hard drive to a new Mac laptop. That was two or three computers ago, and each time my wife has had her hard drive ported over to the next machine. All these years these files of mine have persisted. One of them contained passwords for several online accounts, including Amazon.

In and of itself, you might think this wasn’t much of a find. So what? The SpiderLabs boys could rack up charges on my Amazon credit card. But like many people, I have developed my own system for passwords. Because I can’t possibly remember every single one to every site I use not only do I reuse passwords, I also have come up with an informal formula to create them. I might spell out a common name like Gracie (my old cat) but spell it ‘Gray see’ and use an ‘8’ to stand in for the ‘G’ and a ‘5’ for the ‘s.’ You get the idea. Recall that one of SpiderLabs’ team members is an expert in computer forensics. It didn’t take him long to crack all of my passwords.

The SpiderLabs gang broke into my Twitter account and tweeted “I love Stephen Glass,” which led to some head scratching on Twitter from those who know my role in that story. (I’m the one who outed the serial fabulist from The New Republic.) They breached my Facebook account and ordered 100 plastic spiders from Amazon then had them shipped to my home. And they cracked my iCloud password, sending me an email with the subject: SpiderLabs was here and a message consisting of a single emoticon: :-)

Once they cracked iCloud they activated the “find my iPhone” app. Apple had also enabled this functionality for laptops, so they put both my iPhone and laptop in stolen mode.

The first I learned to what extent SpiderLabs had penetrated my privacy was during my class at NYU, when my laptop shut down and demanded a four-digit code to gain re-entry and my iPhone began beeping.

During our debriefing, Percoco told me that I had been, in some ways, more difficult to get to than many of his corporate clients. With a company employing thousands, there are thousands of potential vulnerabilities that can be attacked. What’s more, the rules are more constrained. For example, a corporate client will tell SpiderLabs which specific servers to target once they’re inside the network or what division to focus on within the corporate hierarchy.

With me, however, there were fewer paths that could lead to the mother lode: my laptop, email, bank, social media accounts, and home. Once in, though, his team found few firewalls protecting my data, and mostly in the form of a pastiche of passwords and log-in credentials. These, I quickly learned, were not secure.

My wife, Charlotte, was practically speechless when I told her about the hack. I had not given her any advance warning, hoping to keep the experiment as realistic as possible. At first she was fascinated, but the more she thought about it, the more uncomfortable she became. The idea that an undercover client had visited her studio and a team of spies had put our home under surveillance made her uneasy. She was relieved, as I was, that our children had been off limits.

“Promise me you’ll never do anything like this again,” she said. And, of course, I did.

Earlier this month, Percoco left SpiderLabs for a new job as Director at KPMG, the professional services firm, in the Information Protection practice where he’s running the same kinds of penetration tests.

As for me, since we concluded this exercise I’ve changed my passwords and log ins but I don’t delude myself into thinking I’m protected from prying eyes — the government’s or anyone else’s, if they belong to someone with the right combination of skills, resources and determination.

And if I’m not safe, are you?

[Illustrations by Alex Schubert for Pandodaily]

Introducing Wit Speech API

Wed, 12 Feb 2014 06:33:40 -0800

Comments: "Introducing Wit Speech API"

URL: https://wit.ai/blog/2014/02/12/speech-api

12 Feb 2014 feature, speech

Voice commands are the future. Science-fiction has had them for decades and yet, we still have to reach for the remote to turn on the TV or set an alarm. Our mission is to change this. Adding a voice interface to an app or device should be simple.

Turning speech into actionable data

Today, we’re very excited to announce our new “Speech to JSON” API, four months after the launch of the “Text to JSON” API.

From now on, your app, device or even your website can stream audio to our server, and get actionable data in return.

See it in action for home automation:

How does it work?

Behind the scene, Wit combines various state-of-the-art Natural Language Processing techniques and several speech recognition engines in order to achieve low latency and high robustness to both surrounding noise and paraphrastic variations (there are millions of ways to say the same thing).

Fortunately, you don’t need to care about all this machinery. We focus all our energy into creating the simplest developer experience possible. You can be up and running in a few minutes using our website. Wit will adapt to your domain over time, from ice-cream distribution to space missions. Wit makes no assumptions and remains 100% configurable.

It will take you 5 minutes to build your own Wit configuration:

Consuming the API

Then, calling the API is simple. We provide client-side SDKs that handle audio recording and streaming for iOS, Android or even a simple webpage like this one. You can also use the HTTP interface to stream live audio or post a sound file:

Let’s take this sound (recorded from a celebrity in the valley – do you know who?):

Submit it to the Wit API with a POST request:

curl -XPOST 'https://api.wit.ai/speech' \
 -i -L \
 -H "Authorization: Bearer $TOKEN" \
 -H "Content-Type: audio/wav" \
 --data-binary "@sample.wav"

You’ll get this in return:

{
 "msg_id" : "6a84eae3-969c-41ad-94d9-85076fbbdc99",
 "msg_body" : "set the kitchen table on fire",
 "outcome" : {
 "intent" : "set_fire",
 "entities" : {
 "object" : {
 "value" : "kitchen table",
 "body" : "kitchen table"
 }
 },
 "confidence" : 0.997
 }
}

Interested to build your own voice interface? Sign up here!

Team Wit

@WitNL

FlapMMO -- flapmmo.com

Wed, 12 Feb 2014 07:31:19 -0800

Comments: "FlapMMO -- flapmmo.com"

Chris Hates Writing • The anonymity I know

Wed, 12 Feb 2014 08:05:19 -0800

Comments: "Chris Hates Writing • The anonymity I know"

URL: http://chrishateswriting.com/post/76431353368/the-anonymity-i-know

The anonymity I know

Yesterday Sam Altman published a short post containing his thoughts on Secret, and also anonymity in general—namely that it breeds meanness, and that anonymous social networks are destined to decay and grow worse over time.

I strongly disagree. What I’ve observed is the opposite—that anonymity facilitates honest discourse, creates a level playing field for ideas to be heard, and enables creativity like none other.

"Anonymity" is itself a slippery term because people frequently use it to refer to everything that isn’t "real identity." Obviously identity is more nuanced than that (I prefer "prismatic”), but for some reason we choose to paint it in broad strokes comprised of two extremes. In the interest of simplicity, I’ll use anonymity to encompass the part of the spectrum that is not real identity, including pseudonymity and everything in between.

When I spoke at TED four years ago I concluded the talk with my concern that in the race to embrace social networking, anonymous communities were quickly going the way of the dinosaur, and that the world was on the verge of losing something incredibly valuable.

It’s an issue near and dear to me, as I’ve had the privilege of founding and presiding over one of the largest anonymous online communities—4chan—for more than a decade.

I was myself once ignorant of the benefits of anonymity. As a 15-year-old who spent his childhood and early teens hanging out in online chatrooms and forums, I wasn’t particularly drawn to the idea of anonymous contribution. When I encountered the inspiration for 4chan, a Japanese website called Futaba Channel, I found myself captivated by its unconventional imageboard format and how quickly content seemed to roll on and off the site—not its emphasis on anonymity and impermanence. Fascinated and frustrated by my inability to contribute (my Japanese is abysmal), I quickly translated the source code and threw it up for a few Internet friends to use.

The very things I overlooked as a teen quickly became the driving force behind the site, and are now deeply ingrained in its ethos as well as my own. Few communities have grown in size and come to influence mainstream culture as 4chan has, for as long as it has, and it is without a doubt the result of allowing people to interact without the burden of identity, and to share and explore new ideas together. For many, 4chan has become their “third place,” and provided a sanctuary away from the everyday stresses of home, school, and work life.

The combination of anonymity and ephemerality has fostered experimentation and creativity rarely seen elsewhere. It’s incredible what people can make when they’re able to fail publicly without fear, since not only will those failures not be attributed to them, but they’ll be washed away by a waterfall of new content. Only ideas that resonate with the broader community persist, creating the most ideal conditions for the production of viral content, which established 4chan as one of the Web’s earliest “meme factories.”

The conversation is “raw” to say the least—almost everyone checks their filter at the door. The resulting dialogue is about as honest as it gets. In lieu of traditional barriers to membership, the community uses cryptic and crude language to regulate who can and cannot participate. On the surface this may seem offensive, but it’s often meant to do little more than keep newcomers on their toes and encourage they lurk and learn the house rules before participating.

Few sites give their users a platform to share ideas quite like 4chan’s—a virtual Speakers’ Corner—where anyone can express their opinions on equal footing. Every person who creates a thread has that thread appear at the very top of the index, and no amount of karma or social capital can save it from the depths of irrelevance. It’s ideas, not reputations, that shine here.

4chan isn’t without its problems and is by no means a utopia, but in many ways provides an accurate representation of who we are: flawed, imperfect. I see beauty in that, and something worthy of continued exploration.

As someone who has spent their entire adult life educating the public about the benefits of anonymity and advocating for alternatives to “real identity,” I’m simultaneously excited and hesitant about what the next few months might bring.

Snapchat has changed the game. Its success has demonstrated that given the right offering, there is in fact mainstream demand for products that incorporate anonymity and ephemerality, and I’ve watched with bated breath as it’s kicked off renewed interest and debate over their merits.

It feels like we’re on the cusp of a fever pitch to explore this new—well, rediscovered—terrain, with entrepreneurs, investors, and journalists all lining up to understand and capitalize on the opportunities that await. I welcome these expeditions, but pray we will see people create thoughtful products that truly reimagine identity for the digital age rather than simply incorporating “anonymity” and “ephemerality” as marketing buzzwords.

It’s bound to be an interesting ride. Whatever may happen, I’m grateful to have a front row seat.

Elasticsearch.org 1.0.0 Released | Blog | Elasticsearch

Wed, 12 Feb 2014 08:53:10 -0800

Comments: "Elasticsearch.org 1.0.0 Released | Blog | Elasticsearch"

URL: http://www.elasticsearch.org/blog/1-0-0-released/

Rails 4 Engines - TechRabbit

Wed, 12 Feb 2014 09:04:10 -0800

Comments: "Rails 4 Engines - TechRabbit"

URL: http://tech.taskrabbit.com/blog/2014/02/11/rails-4-engines/

At TaskRabbit, we have gone through a few iterations on how we make our app(s). In the beginning, there was the monolithic Rails app in the standard way with 100+ models and their many corresponding controllers and views. Then we moved to several apps with their own logic and often using the big one via API. Our newest project is a single “app” made up of several Rails engines. We have found that this strikes a great balance between the (initial) straightforwardness of the single Rails app and the modularity of the more service-oriented architecture.

We’ve talked about this approach with a few people and they often ask very specific questions about the tactics used to make this happen, so let’s go through it here and via a sample application.

Rails Engines

Rails Engines is basically a whole Rails app that lives in the container of another one. Put another way, as the docs note: an app itself is basically just an engine at the root level. Over the years, we’ve seen sen engines as parts of gems such as devise or rails_admin. These example show the power of engines by providing a large set of relatively self-contained functionality “mounted” into an app.

At some point, there was a talk that suggested the approach of putting my our functionality into engines and that the Rails team seemed to be devoting more and more time to make them a first class citizen. Our friends at Pivotal Labs were talking about it a lot, too. Sometimes good and sometimes not so good.

Versus Many Apps

We’d seen an app balloon and get out of control before, leading us to try and find better ways of modularization. It was fun and somewhat liberating to say “Make a new app!” when there was a new problem domain to tackle. We also used it as a way to handle our growing organization. We could ask Team A to work on App A and know that they could run faster by understanding the scope was limited to that. As a side-note and in retrospect, we probably let organizational factors affect architecture way more than appropriate.

Lots of things were great about this scenario. The teams had freedom to explore new approaches and we learned a lot. App B could upgrade Rack (or whatever) because it depend on the crazy thing that App A depended on. App C had the terrible native code-dependent gem and we only had to put that on the App C servers. Memory usage was kept lower, allowing us to run more background workers and unicorn threads.

But things got rough in coordinating across these apps. It wasn’t just the data access. We made APIs and allowed any app to have read-only access to the platform app’s database. This allowed things go much faster by preventing creation of many GET endpoints and possible points of failure. The main issue in coordinating releases that spanned apps. They just went slower than if it was one codebase. There was also interminable bumping of gem versions to get shared code to all the apps. Integration testing the whole experience was also very rough.

So it’s a simple one, but the main advantage that we’ve seen in the engine model is that it is one codebase and git repo. A single pull request has everything related to that feature. It rolls out atomically. Gems can be bumped once and our internal gems aren’t bumped at all as they live unbuilt in a gems folder in the app itself. We still get most of the modularization that multiple apps had. For example, the User model in the payments engine has all the stuff about balances and the one in the profile engine doesn’t know anything about all that and it’s various helper methods.

The issue with gem upgrades and odd server configurations does continue to exist in the engine model and is mostly fine in the many app model. The gem one is tough and we just try to stay on top of upgrading to the newest things and overall reducing dependencies. The specs will also run slower in the engine app, but you’ll have better integration testing. I’ll go over a little bit about we’ve tackled server configurations and memory further down.

Versus Single App

It’s very tempting when green-fielding a project to just revert back to the good-old-days of the original app. Man, that was so nice back before the (too) fat models and tangled views and combinatorics of 4 years of iterating screwed things up. And we’ve learned a lot since then too, right? Especially about saying no to all those combinatorics and also using decorators and service objects and using APIs. Maybe.

What we do know is that you can feel that way again even a year into an app. Inside any given engine, you have the scope of a much smaller project. Some engines may grow larger and you’ll start to use those tools to keep things under control. Some will (correctly) have limited scope and feel like a simple app in which you understand everything that is happening. For example, decorators are great tool and they came in handy in our big app and larger engines. However, we’ve found in an a targeted engine that only serves its one purpose, it feels like there is room in that model to have some things that would have been decorated in a larger app. This is because it doesn’t have all that other junk in it. Only this engine’s junk :-)

Engine Usage

We’ve seen a few different ways to use engines in a Rails project. A few examples are below. The basic variables are what is in the “operator” (root) app and what kind of app we’re making (API driven or not).

Admin

The first engine we’ve recommend making to people is the admin engine. In the first app, we made the mistake of putting admin functionality in the “normal” pages. It was very enticing. We had that form already for the user to edit it. Just by changing the permissions, we could allow the admin to edit it, too. Forms are cheap and admins want extra fields. And more info. And basically a different UI.

So we can made an engine basically just like rails_admin did and gave it’s own layout and views and JS and models and controllers, etc. Overall, we started treating our hardworking admins like we should: a customer with their own needs and dedicated experience.

The structure looked something like this…

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 app assets controllers models user.rb post.rb views layouts admin app assets controllers models admin user.rb post.rb views layouts config db migrate gems spec

When we had this all mixed into one interface and set of models, at least a third of the code in a model like Post or User would be admin-specific actions. With this approach, we can give the admins a better, targeted experience and keep that code in admin-land.

Throughout these engine discussions, the question of sharing code and/or inheriting from objects will keep coming up. Specifically, for the admin scenario, we say do whatever works for you and on a case by case basis. In the above approach, we would probably tend to have Admin::Post < ::Post and other such inheritance. In Rails 2, we probably wouldn’t have done what as they would have different attr_accessible situations but that’s happening in the controller these days, so now inheriting from them will just get the benefit of the data validations, which is something we definitely want to share.

Note that inheriting is probably a bad choice if you have callbacks in the root model that you don’t want triggered when the admin saves the record. In that case, it would be better to Admin::Post < ActiveRecord::Base and either duplicate the logic, have it only in SQL table (unique indexes for example), or have a mixin that is included in both.

Shared Code

The note about controllers being in charge of the parameters involved leads to the next possibility. You can have your models (at least the ones you need to have shared) in the operator and all the other stuff in the engines. At this point, maybe you could add the engines namespace to be more clear.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 app models user.rb post.rb config db migrate engines customer app assets controllers models customer something_admin_doesnt_use.rb views layouts admin app assets controllers models admin admin_notes.rb views layouts gems spec

Now you can use Post from both and everything is just fine. This would work out well if it’s mostly the data definition you are using and like to use things like decorators and/or service objects and/or fat controllers in your engines.

You could also put layouts or mixins in the operator. This might be a good idea if you were sharing the layout between two engines. At that point, maybe we’ll just go all in on the engines by making a shared engine. Having a namespace for clarity is much simpler.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 apps shared app assets controllers shared authentication.rb models shared post.rb user.rb views shared layouts marketing app controllers marketing application_controller.rb home_controller.rb content controllers models content something_admin_doesnt_use.rb admin app assets controllers models admin admin_notes.rb views layouts config db migrate gems spec

In this structure, admin can still get it’s own layout if it wants, but marketing and content can easily share the same layout in addition to the models.

The example in Github takes this just one step farther by not sharing models at all. Sharing the actual model can still lead to the god model situation of a mono-Rails app without the use of other mitigating objects. To keep things as tight as possible, we’ve allowed each engine to have their own User object, for example. If there is model code to share, it would still go in the shared engine, but as a mixin like this one. Note that in a well-designed schema, only one of these actually writes to the database and the others include a ReadOnly module from the shared engine.

The repo’s structure looks as follows:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 apps shared app assets controllers shared controller authentication.rb models shared model read_only.rb user user_display.rb views shared layouts marketing app controllers marketing application_controller.rb home_controller.rb models marketing user.rb db migrate account app controllers models content user.rb post.rb db migrate content app assets controllers models admin post.rb user.rb db migrate admin app assets controllers models admin admin_notes.rb post.rb user.rb views layouts db migrate config gems spec

API Server

Our latest project at TaskRabbit basically looks the the above and the example with one difference: we don’t share layouts between our engines. We’ve made the choice to have all the frontend code in one engine and all of the other engines just serve API endpoints. There are several shared mixins for these backend engines, but they don’t need a layout because they are just using jbuilder to send back JSON to the frontend client. The frontend engine, therefore, doesn’t really use any models and has all the assets and such. Admin still has its own layout and uses a more traditional Rails MVC approach.

It looks like this:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 apps shared app assets controllers shared controller authentication.rb models shared model read_only.rb user user_display.rb frontend app assets controllers marketing application_controller.rb home_controller.rb models marketing user.rb views frontend layouts account app controllers models content user.rb post.rb views account users show.json.jbuilder db migrate content app controllers models admin post.rb user.rb views db migrate admin app assets controllers models admin admin_notes.rb post.rb user.rb views layouts db migrate config gems spec

The API setup alleviates one of the odder things about the example approach. Ideally, there is no interaction between engines. Particularly in the models and views, this is critical. However, some knowledge leaks out in the example though from the controllers. For example, the login controller redirects to /posts after login. This is in the content engine. It’s probably not the end of the world but that is coupling. We get around this using our one frontend engine and the several API ones, but this does some serious commitment.

Strategies

We’ve gotten lots of questions and read about issues people are having with engines so let’s go through them here.

Migrations and Models

Rails bills itself as “convention over configuration” so it’s not too surprising to be confronted with lots of questions about “where to put stuff” when deviating (slightly) from the conventions. The one people seem the most worried about are migrations. We’ve never had an issue, but there must be scenarios that get a little tricky. If you are sharing the models, wewould just put them in the normal db/migrate location. If your models live inside the engines, it’s probably not a huge deal to still do that, but we’ve decided to have the migrations live with their models.

As notes, each model/table (say users) ideally has one master model. In the sample app, the User model’s master is in the account engine. This engine is in charge of signing up and logging in users. Fleshed out, it would also be responsible for reseting a lost password and editing account information. It’s the only User model that mentions has_secure_password and knows anything about that kind of thing. The rest of the engines may need a User model but they have the ReadOnly module to prevent actually writing to the table.

Therefore, the account engine has the migrations having to do with the users table. In order to register that migrations are within these engines, we add a snippet like the following to each engine.

1 2 3 4 5 6 7 initializer 'account.append_migrations' do |app| unless app.root.to_s == root.to_s config.paths["db/migrate"].expanded.each do |path| app.config.paths["db/migrate"].push(path) end end end

This (via here) puts the engine’s migrations in the path. Migrations continue to work as they normally do with the timestamps and such. So our db/migrate folder doesn’t have any files in it (and is not checked into git). I have one locally, just because when I make a migration, Rails creates it automatically. However, I end up doing something like this immediately.

1 2 3 4 $ bundle exec rails g migration CreatePosts invoke active_record create db/migrate/20140207011608_create_posts.rb $ mv db/migrate/20140207011608_create_posts.rb apps/content/db/migrate

You might wonder, and it does come up, what to do when you are adding a column to the users table for some other feature in some other engine. For example, we added a boolean admin column to the example users table to know if the given user is allowed to do stuff in the admin engine. We see the notion of permissions as being within the account engine’s scope, even if it’s not being actively leveraged there. Tt’s still part of the account. Therefore, we added the migration to the account engine.

In part, if I couldn’t justify to myself why it would be part of the account engine, it would be a red flag. Specifically, should this even be in the users table at all. If the answer is “yes” for whatever reason, then I’d likely still put the migration in the account engine, but usually it helps me realize that it shouldn’t be in the users table at all. A good example that came up in our app was the notion of profile. It seemed like it was 1-to-1 with users and what ever columns supported it should go in the users table. For a variety of reasons, including that we wanted a different engine for that, we ended up making it’s own table with a a has_one relationship in that engine. This paid off even further as we realized that a User should actually have two profiles, one for their activity as a TaskPoster and one as a TaskRabbit, as they record and display very different information. Each has their own table and engine now.

Let’s say we wanted to cache the number of posts the user had made. That’s a pretty clearcut case to use counter_cache and put a posts_count in the users table. We’ll want to look closely at this situation. First of all, the counter_cache code would clearly go on the User model in the content engine. That would also require that model to not be read-only or at least not in spirit (depending on the specifics used to implement the feature). It’s not a good feeling when you do all this architecture stuff and it gets in the way of something that is so easy and we have to look out for those cases. If this is one of those cases, just do it; literally, however you want. We would probably keep the migration in the account engine.

It might not be one of those cases, though. I have almost never been sorry when I’ve made another model in these cases. So we could make a PostStatistic model or something in the content engine which belongs_to :user for recording this (and likely other things that come up). The counter cache feature is not magic - we just increment that table as necessary. It also doesn’t feel that superfluous as it exists only inside that engine (which. in turn, doesn’t have all the random stuff internal to other engines). We have some tables that started out that way. Mostly because we actively try not to do JOINs on our API calls, these tables ending up being the hub of the most relevant data of what has happening in our marketplace. Another option that we’ve used in similar situations is not to make the column at all. The content engine, or whoever is using this kind of data, would use the timestamp of the last Post or some other data to use as the cache key to look up all kinds of stuff in a store like memcache or Redis. If it’s not there, it will take bit the bullet and calculate it and store it in the cache.

Again, architecture does not exist for fun or to get in the way. If something is super-simple and obvious and easy to maintain while doing the “right” way for the design is difficult and fragile, we just do it the easy way. That’s the way to ship things for customers. However, we’ve found that in most case the rules of the system kick off useful discussions and behaviors that tend to work out quite well.

Admin

One of the cases where it’s important to really examine the value and return on investment in engine separation is with the admin engine. We believe it’s a special case.

In our system, the admin engine has it’s own migrations. For example, we have a model called AdminNote where an admin can jot down little notes about most objects in the system. It clearly owns that. But the reason this whole experience exists in the first place is that it also is able to write more or less whatever it wants to all the objects in the system. This clearly violates our single-model-master rule. So we don’t fight an uphill battle here by making a special case and saying that the admin engine can literally do whatever it wants. All the other engines live in complete isolation from each other for a variety of reasons. Admin can depend directly on any or all of them. It’s at the top of the food chain because it needs to regulate the whole system.

So it’s fine if Admin::Post < Content::Post or just uses Content::Post directly in it’s controllers. It’s just not worth it to share all of the data definitions and validations with when it will almost always be with engine X and admin. Note that it’s important to have the same validations because admin might be in charge, but it still needs to produce valid data as that other engine will be using it.

In our much larger app, we inherit from and/or use most of the models in the system as well as service objects from other engines. We do not use outside controllers or views. Our admin engine does use it’s own layout and much simpler request cycle than our much fancier frontend app. We tried to show the admin engine using a different layout in the example app, but they’re both bootstrap so it might be hard to tell. The header is red in admin :-)

Assets

Everyone seems to have struggled with this one and I can’t even imagine pulling apart assets if they weren’t coded in a modular way at the start. However, starting with them separate in Rails 4 has been fairly straightforward. We add the following code to our engine much like the migration code.

1 2 3 initializer 'account.asset_precompile_paths' do |app| app.config.assets.precompile += ["account/manifests/*"] end

You could list all the manifests one by one, but we’ve found that it’s simpler to just always put them in a folder created for the purpose. This works for both css and js. You would would reference those files something like this:

1 2 = stylesheet_link_tag 'account/manifests/application' = javascript_include_tag 'account/manifests/application'

Routes

In an Engine, routes go within the engine directory at the same config/routes.rb path. It’s important to note here that in order for these routes to be put into use in the overall app, the engine needs to be mounted. In a normal engine use case, you would mount rails_admin (say to /admin) to give a namespace in the url, but we think it’s important that all of these engines get mounted at the root level. You can see our root routes.rb file here.

1 2 3 4 5 RailsEnginesExample::Application.routes.draw do BootInquirer.each_active_app do |app| mount app.engine => '/', as: app.gem_name end end

So as expected, the operator app has no routes of it’s own and it’s all handled by the engines. I’ll add little more about the BootInquirer in a bit. It is just a helper class that knows all the engines. This means that the code is functionally something more like this:

1 2 3 4 5 6 RailsEnginesExample::Application.routes.draw do mount Admin::Engine => '/', as: 'admin' mount Account::Engine => '/', as: 'account' mount Content::Engine => '/', as: 'content' mount Marketing::Engine => '/', as: 'marketing' end

It would really clean to have something other than root in these mountings, but it doesn’t seem practical or that important. We want to be able to have full control over our url structure. For example, mounting the account engine at anything but root would prevent it from handling both the /login and /signup paths. The tradeoff is that two engines could claim the same URLs and conflict with much confusion. That’s something we can manage with minimal effort. We’ve found that most engine route files start with scope to put most things under one directory or a few resources which does basically the same thing.

Another important note is to use isolate_namespace in your Engine declaration. That prevents various things like helper methods from leaking into other engines. This makes sense for our case because the whole point is to stay contained. Another side effect is route helpers like ‘posts_path’ to work as expected without needing to prefix them like content.posts_path in your views. I believe it might also make the parameters more regular (for example having params[:post] instead of params[:content_post]). Oh, just put it in there.

Tests

Many of the issues noted here revolve around testing. One of the promises of engines is the existence of the subcomponents that you could (theoretically) use in some other app. This is not the goal here. We are using engines maximize local simplicity in our application, not create a reusable library. To that end, we don’t think the normal Engine testing mechanism of creating a dummy app within the engine is helpful.

On our first engine application, we put a spec folder within each engine and then wrote a rspec_all.sh script to run each of them. It was not the right way. To do that really correctly, you’d test at that level and you’d have to test again at the integration level. This is another case of it not being worth it. Now we just put all our specs in the spec directory and run rspec spec to run them all.

Each engine has it’s own directory in there to keep it somewhat separate and to be able to easily test all of a single engine and it ends up looking like a normal app’s root spec folder with models, requests, controllers, etc. Much like the admin engine, there are no rules about what you can and can’t use in the tests. The goal is make sure the code is right, not to follow some architectural edict. For example, in a test that checks whether a Task can be paid for, it’s fine to use the models from the payment engine to make sure everything worked together well.

One thing that is interesting is fixtures. We like using fixtures because it’s a pretty good balance between speed and fully executing most of the code in out tests. We use fixture_builder to save the hassle of maintaining those yml files precisely. Anyway, the issue in the case where we have multiple engine’s each with their own model class is that fixtures (and factories for that matter) only get one class. So if you do something like this while testing in the content engine, you’d be in trouble:

1 2 3 4 5 6 7 8 9 10 11 describe Content::Post do fixtures :users it "should be associated with a user" do user = users(:willy) post = Content::Post.new(content: "words") post.user = user post.save.should == true user.posts.count.should == 1 end end

This is a problem because of classes expecting to be a certain type. You’d get this error:

1 2 3 4 5 6 Failures: 1) Content::Post should be associated with a user Failure/Error: post.user = user ActiveRecord::AssociationTypeMismatch: Content::User(#70346317272500) expected, got Account::User(#70346295701620)

So the user has to be and instance of the Content::User and not an Account::User class. We use a helper to say what the classes are as well as switch between them. So this test will use the correct classes:

1 2 3 4 5 6 7 8 9 10 11 describe Content::Post do fixtures :users it "should be associated with a user" do user = fixture(:users, :willy, Content) post = Content::Post.new(content: "words") post.user = user post.save.should == true user.posts.count.should == 1 end end

The same sort of thing could be done with FactoryGirl too. Often, we end up just using the ids more than we would in a normal test suite. The important thing to note is to just do whatever you feel gives you the best coverage with the most return on investment for your time.

Memory

You may have noticed the BootInquirer class mentioned earlier. This is a class that know about all the engines in the system.

1 2 3 4 5 6 APPS = { 'a' => 'account', 'c' => 'content', 'm' => 'marketing', 'z' => 'admin' }

It is called from three places.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 # Gemfile gemspec path: "apps/shared" BootInquirer.each_active_app do |app| gemspec path: "apps/#{app.gem_name}" end # application.rb require_relative "../lib/boot_inquirer" BootInquirer.each_active_app do |app| require app.gem_name end # routes.rb BootInquirer.each_active_app do |app| mount app.engine => '/', as: app.gem_name end

The main point here is to simplify even further how to add a new engine to the app. The secondary point is somewhat interesting, though. One of the potential downsides of an engine-based app over multiple apps is the larger memory footprint or larger scale production rollout of some obscure and complicated native library for just one of the engines. This would not be a problem if you could “boot” the app with the just some of the engines enabled. The BootInquirer makes that possible. It inspects and environment variable to know which engines to add to the gemspec and require and route towards.

1 2 3 4 $ ENGINE_BOOT=am bundle exec rails c => will boot the account and marketing engines - but not content, admin, etc. $ ENGINE_BOOT=-m bundle exec rails c => will boot all engines except marketing

We haven’t actually seen memory be that different that in our large Rails app. In fact, it is less because of a combination of Ruby upgrades and less conspicuous gem consumption. However, memory-wise this setup allows us to use our one codebase like multiple apps. In that case, we use a load balancer to map url paths to the correct app.

This is also useful in processing background workers. You would likely get an extra Resque worker or two. It’s important to have a good queue strategy (different queues per engine) and to really not have the engines depend on each other to make this work, of course.

In order for this to work, we need to be more mindful of our gem usage. The first step is changing application.rb to say Bundler.setup(:default, Rails.env) instead of Bundler.require(:default, Rails.env) as usual. This mean we will have to explicitly require the gems we are using instead of it happening automatically. Most of those dependencies are in the engines’ gemspecs and they’d have to be required anyway. However, by changing this line, we’ll have to require what is needed from the main Gemfile as well. Ideally, there wouldn’t be anything in there at all, but we have some Rails and test stuff that all the engines use.

You may notice that the exception we made for the admin engine rears its head here. If admin depends on the other engines, you won’t be able to use admin experience unless you launch the app with all those engines. This is definitely true. The servers that the admin urls route to will have to have all of the engines running. We found it was useful to quarantine admin usage anyway as there are a few requests and inputs that could blow out the heap size fairly easily.

Folders and Files

If you’re interested in this setup, you’re just going to have to get used to it. There are a lot of directories. There are lot of files named the same thing. I’ve found that Sublime Text is better for this than Textmate. I’m a huge fan of ⌘T to open files and Sublime allows the use of the directory names in that typeahead list. If your editor doesn’t do this, then you’ll spend more time than you want to look through the six different user.rb or application_contoller.rb files in the project.

Interaction Between Engines

So we’ve gone through a lot of trouble to keep that shiny new Rails app feel. Each engine has a particular goal in life and everything is nice and simple. Particularly in the API case, it writes and reads its data and generally just takes of business. But the world isn’t always perfect and sometimes the engines need to talk to each other. If it’s happening too much, we probably didn’t modularize along the right lines and we should consider throwing them together. We don’t have all the answers, but engine naming and scoping seems to be a fine art. It’s very tempting to go very narrow for cleanliness and it’s also very tempting to just throw stuff in to an existing one so I’m not surprised when we find that the lines are a not drawn quite right.

There are other cases, though, that are not systemic errors in engine-picking and future-prediction. It’s the kind of case I talked about with the posts_count above. Let’s say we had a good reason to make that happen. Actually let’s change it just a little bit to be more realistic. Let’s say we had a profile engine where user could manage his online presence. Let’s also say that other users could see and rate his posts. It’s a completely reasonable thing to have an average post rating shown on his profile. Does this data about posts mean that the profile pages or API should be part of the content engine? We don’t think so. This is likely just one tiny detail in an engine otherwise setup to upload photos, quote favorite movies, or whatever. We just need a little average rating on the there somewhere with a link to the posts.

In this case, we use our Resque Bus gem extensively. This is a minor add-on to Resque that changes the paradigm just enough to allow us to decouple these engines. In a normal Rails apps using Resque, we would queue up a background worker to process the rating. This worker would calculate the new average rating and store it in the profile. Resque Bus uses publishing and subscription to accomplish similar goals. If you buy into this model, you have all of your engines and in this case the content engine, publishing to the bus when interesting things happen. Creation of a post or rating would be a good example. Other engines (or completely separate apps) then subscribe to events they find interesting. There can be more than one subscriber. Even when there is nothing particularly interesting to do, we’ve found that always having a subscriber to record the event produces a really useful log. In the rating case, though, the profile engine would also subscribe to the event and record the new rating. By one engine simply noting that something happened and the other reacting to the occurrence, we maintain the conceptual as well as physical (these engines could be on different servers) decoupling.

What exactly gets published and how that is used is up to the developers involved. There seems to be a few options in this specific case.

A) The content engine is publishing data changes. ResqueBus.publish('post_rated', {post_id: 42, author_id: 2, rated_by: 4, rating: 4}) B) The content engine adds some calculations. ResqueBus.publish('post_rated', {post_id: 42, author_id: 2, rated_by: 4, rating: 4, new_average: 4.25, total_ratings: 20})

Choosing option B is interesting for a few reasons:

It is predicting the information other engines will want to know.
It decreases the coupling because now the profile engine now just records the info instead of having to calculate it.
It creates a record of the averages in our event store. Maybe we’ll draw a graph of it sometime.
It adds to the time required to complete the request to create the rating.

This would mean the post engine would have something like this in an initializer:

1 2 3 4 5 6 7 8 ResqueBus.dispatch('profile') do subscribe 'post_rated' do |attributes| profile = Profile::Document.find_by(user_id: attributes['author_id']) profile.post_ratings_total = attributes['total_ratings'] profile.post_rating_average = attributes['new_average'] profile.save! end end

Or in the way that we prefer using a subscriber class that we would put in profile/app/subscribers:

1 2 3 4 5 6 7 8 9 10 11 12 class Profile::ContentSubscriber include ResqueBus::Subscriber subscribe :post_created def post_created(attributes) profile = Profile::Document.find_by(user_id: attributes['post_author_id']) profile.post_ratings_total = attributes['total_ratings'] profile.post_rating_average = attributes['new_average'] profile.save! end end

It’s clearly a fine option and the added time probably isn’t too much assuming we have the right indexes on our database, but we actually tend to use option A. We don’t particularly like trying to predict which events are interesting and how other engines will use them so we just publish on all creations or updates. We are fine with the profile engine having read-only Rate model and code to calculate the average. It could keep a running tally of the total number and just add this one to it, but we tend to recalculate it every time because it’s not that hard and is less fragile.

It would look something like this:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 class Profile::ContentSubscriber include ResqueBus::Subscriber subscribe :post_created def post_created(attributes) total = Profile::Rate.where(author_id: attributes['author_id']).count sum = Profile::Rate.where(author_id: attributes['author_id']).sum(:rating) profile = Profile::Document.find_by(user_id: attributes['post_author_id']) profile.post_ratings_total = total profile.post_rating_average = sum.to_f / (5*total.to_f) profile.save! end end

However you do it, the point is that this engine is working on it’s own for it’s own purposes. Layering it on, it’s quite straightforward to see how we could build spam detection as its own engine or into the admin one. We could subscribe to ratings or post creation and react accordingly, maybe pulling the post or giving the user a score that limits his visibility, etc. Or we could add a metrics engine, to report the conversion of a user on his first post to a variety of external services. Then, when a new developer starts and asks where the metrics code is, we don’t have to say what we said before which was, “everywhere.” We could show very simple mappings between things that are happening throughout the system and the numbers like revenue or engagement that are getting reported to something like Google Analytics.

Summary

Try out engines. We like them.

Discuss on Hacker News.

Scientific method: Statistical errors : Nature News & Comment

Wed, 12 Feb 2014 09:35:31 -0800

Comments: " Scientific method: Statistical errors : Nature News & Comment "

URL: http://www.nature.com/news/scientific-method-statistical-errors-1.14700

DALE EDWIN MURRAY

For a brief moment in 2010, Matt Motyl was on the brink of scientific glory: he had discovered that extremists quite literally see the world in black and white.

The results were “plain as day”, recalls Motyl, a psychology PhD student at the University of Virginia in Charlottesville. Data from a study of nearly 2,000 people seemed to show that political moderates saw shades of grey more accurately than did either left-wing or right-wing extremists. “The hypothesis was sexy,” he says, “and the data provided clear support.” The P value, a common index for the strength of evidence, was 0.01 — usually interpreted as 'very significant'. Publication in a high-impact journal seemed within Motyl's grasp.

But then reality intervened. Sensitive to controversies over reproducibility, Motyl and his adviser, Brian Nosek, decided to replicate the study. With extra data, the P value came out as 0.59 — not even close to the conventional level of significance, 0.05. The effect had disappeared, and with it, Motyl's dreams of youthful fame1.

It turned out that the problem was not in the data or in Motyl's analyses. It lay in the surprisingly slippery nature of the P value, which is neither as reliable nor as objective as most scientists assume. “P values are not doing their job, because they can't,” says Stephen Ziliak, an economist at Roosevelt University in Chicago, Illinois, and a frequent critic of the way statistics are used.

For many scientists, this is especially worrying in light of the reproducibility concerns. In 2005, epidemiologist John Ioannidis of Stanford University in California suggested that most published findings are false2; since then, a string of high-profile replication problems has forced scientists to rethink how they evaluate results.

At the same time, statisticians are looking for better ways of thinking about data, to help scientists to avoid missing important information or acting on false alarms. “Change your statistical philosophy and all of a sudden different things become important,” says Steven Goodman, a physician and statistician at Stanford. “Then 'laws' handed down from God are no longer handed down from God. They're actually handed down to us by ourselves, through the methodology we adopt.”

Out of context

P values have always had critics. In their almost nine decades of existence, they have been likened to mosquitoes (annoying and impossible to swat away), the emperor's new clothes (fraught with obvious problems that everyone ignores) and the tool of a “sterile intellectual rake” who ravishes science but leaves it with no progeny3. One researcher suggested rechristening the methodology “statistical hypothesis inference testing”3, presumably for the acronym it would yield.

The irony is that when UK statistician Ronald Fisher introduced the P value in the 1920s, he did not mean it to be a definitive test. He intended it simply as an informal way to judge whether evidence was significant in the old-fashioned sense: worthy of a second look. The idea was to run an experiment, then see if the results were consistent with what random chance might produce. Researchers would first set up a 'null hypothesis' that they wanted to disprove, such as there being no correlation or no difference between two groups. Next, they would play the devil's advocate and, assuming that this null hypothesis was in fact true, calculate the chances of getting results at least as extreme as what was actually observed. This probability was the P value. The smaller it was, suggested Fisher, the greater the likelihood that the straw-man null hypothesis was false.

R. NUZZO; SOURCE: T. SELLKE ET AL. AM. STAT. 55, 62–71 (2001)

For all the P value's apparent precision, Fisher intended it to be just one part of a fluid, non-numerical process that blended data and background knowledge to lead to scientific conclusions. But it soon got swept into a movement to make evidence-based decision-making as rigorous and objective as possible. This movement was spearheaded in the late 1920s by Fisher's bitter rivals, Polish mathematician Jerzy Neyman and UK statistician Egon Pearson, who introduced an alternative framework for data analysis that included statistical power, false positives, false negatives and many other concepts now familiar from introductory statistics classes. They pointedly left out the P value.

But while the rivals feuded — Neyman called some of Fisher's work mathematically “worse than useless”; Fisher called Neyman's approach “childish” and “horrifying [for] intellectual freedom in the west” — other researchers lost patience and began to write statistics manuals for working scientists. And because many of the authors were non-statisticians without a thorough understanding of either approach, they created a hybrid system that crammed Fisher's easy-to-calculate P value into Neyman and Pearson's reassuringly rigorous rule-based system. This is when a P value of 0.05 became enshrined as 'statistically significant', for example. “The P value was never meant to be used the way it's used today,” says Goodman.

What does it all mean?

One result is an abundance of confusion about what the P value means4. Consider Motyl's study about political extremists. Most scientists would look at his original P value of 0.01 and say that there was just a 1% chance of his result being a false alarm. But they would be wrong. The P value cannot say this: all it can do is summarize the data assuming a specific null hypothesis. It cannot work backwards and make statements about the underlying reality. That requires another piece of information: the odds that a real effect was there in the first place. To ignore this would be like waking up with a headache and concluding that you have a rare brain tumour — possible, but so unlikely that it requires a lot more evidence to supersede an everyday explanation such as an allergic reaction. The more implausible the hypothesis — telepathy, aliens, homeopathy — the greater the chance that an exciting finding is a false alarm, no matter what the P value is.

These are sticky concepts, but some statisticians have tried to provide general rule-of-thumb conversions (see 'Probable cause'). According to one widely used calculation5, a P value of 0.01 corresponds to a false-alarm probability of at least 11%, depending on the underlying probability that there is a true effect; a P value of 0.05 raises that chance to at least 29%. So Motyl's finding had a greater than one in ten chance of being a false alarm. Likewise, the probability of replicating his original result was not 99%, as most would assume, but something closer to 73% — or only 50%, if he wanted another 'very significant' result6, 7. In other words, his inability to replicate the result was about as surprising as if he had called heads on a coin toss and it had come up tails.

Critics also bemoan the way that P values can encourage muddled thinking. A prime example is their tendency to deflect attention from the actual size of an effect. Last year, for example, a study of more than 19,000 people showed8 that those who meet their spouses online are less likely to divorce (p < 0.002) and more likely to have high marital satisfaction (p < 0.001) than those who meet offline (see Nature http://doi.org/rcg; 2013). That might have sounded impressive, but the effects were actually tiny: meeting online nudged the divorce rate from 7.67% down to 5.96%, and barely budged happiness from 5.48 to 5.64 on a 7-point scale. To pounce on tiny P values and ignore the larger question is to fall prey to the “seductive certainty of significance”, says Geoff Cumming, an emeritus psychologist at La Trobe University in Melbourne, Australia. But significance is no indicator of practical relevance, he says: “We should be asking, 'How much of an effect is there?', not 'Is there an effect?'”

Perhaps the worst fallacy is the kind of self-deception for which psychologist Uri Simonsohn of the University of Pennsylvania and his colleagues have popularized the term P-hacking; it is also known as data-dredging, snooping, fishing, significance-chasing and double-dipping. “P-hacking,” says Simonsohn, “is trying multiple things until you get the desired result” — even unconsciously. It may be the first statistical term to rate a definition in the online Urban Dictionary, where the usage examples are telling: “That finding seems to have been obtained through p-hacking, the authors dropped one of the conditions so that the overall p-value would be less than .05”, and “She is a p-hacker, she always monitors data while it is being collected.”

“The P value was never meant to be used the way it's used today.”

Such practices have the effect of turning discoveries from exploratory studies — which should be treated with scepticism — into what look like sound confirmations but vanish on replication. Simonsohn's simulations have shown9 that changes in a few data-analysis decisions can increase the false-positive rate in a single study to 60%. P-hacking is especially likely, he says, in today's environment of studies that chase small effects hidden in noisy data. It is tough to pin down how widespread the problem is, but Simonsohn has the sense that it is serious. In an analysis10, he found evidence that many published psychology papers report P values that cluster suspiciously around 0.05, just as would be expected if researchers fished for significant P values until they found one.

Numbers game

Despite the criticisms, reform has been slow. “The basic framework of statistics has been virtually unchanged since Fisher, Neyman and Pearson introduced it,” says Goodman. John Campbell, a psychologist now at the University of Minnesota in Minneapolis, bemoaned the issue in 1982, when he was editor of the Journal of Applied Psychology: “It is almost impossible to drag authors away from their p-values, and the more zeroes after the decimal point, the harder people cling to them”11. In 1989, when Kenneth Rothman of Boston University in Massachusetts started the journal Epidemiology, he did his best to discourage P values in its pages. But he left the journal in 2001, and P values have since made a resurgence.

Ioannidis is currently mining the PubMed database for insights into how authors across many fields are using P values and other statistical evidence. “A cursory look at a sample of recently published papers,” he says, “is convincing that P values are still very, very popular.”

Any reform would need to sweep through an entrenched culture. It would have to change how statistics is taught, how data analysis is done and how results are reported and interpreted. But at least researchers are admitting that they have a problem, says Goodman. “The wake-up call is that so many of our published findings are not true.” Work by researchers such as Ioannidis shows the link between theoretical statistical complaints and actual difficulties, says Goodman. “The problems that statisticians have predicted are exactly what we're now seeing. We just don't yet have all the fixes.”

Statisticians have pointed to a number of measures that might help. To avoid the trap of thinking about results as significant or not significant, for example, Cumming thinks that researchers should always report effect sizes and confidence intervals. These convey what a P value does not: the magnitude and relative importance of an effect.

Many statisticians also advocate replacing the P value with methods that take advantage of Bayes' rule: an eighteenth-century theorem that describes how to think about probability as the plausibility of an outcome, rather than as the potential frequency of that outcome. This entails a certain subjectivity — something that the statistical pioneers were trying to avoid. But the Bayesian framework makes it comparatively easy for observers to incorporate what they know about the world into their conclusions, and to calculate how probabilities change as new evidence arises.

Others argue for a more ecumenical approach, encouraging researchers to try multiple methods on the same data set. Stephen Senn, a statistician at the Centre for Public Health Research in Luxembourg City, likens this to using a floor-cleaning robot that cannot find its own way out of a corner: any data-analysis method will eventually hit a wall, and some common sense will be needed to get the process moving again. If the various methods come up with different answers, he says, “that's a suggestion to be more creative and try to find out why”, which should lead to a better understanding of the underlying reality.

Simonsohn argues that one of the strongest protections for scientists is to admit everything. He encourages authors to brand their papers 'P-certified, not P-hacked' by including the words: “We report how we determined our sample size, all data exclusions (if any), all manipulations and all measures in the study.” This disclosure will, he hopes, discourage P-hacking, or at least alert readers to any shenanigans and allow them to judge accordingly.

A related idea that is garnering attention is two-stage analysis, or 'preregistered replication', says political scientist and statistician Andrew Gelman of Columbia University in New York City. In this approach, exploratory and confirmatory analyses are approached differently and clearly labelled. Instead of doing four separate small studies and reporting the results in one paper, for instance, researchers would first do two small exploratory studies and gather potentially interesting findings without worrying too much about false alarms. Then, on the basis of these results, the authors would decide exactly how they planned to confirm the findings, and would publicly preregister their intentions in a database such as the Open Science Framework (https://osf.io). They would then conduct the replication studies and publish the results alongside those of the exploratory studies. This approach allows for freedom and flexibility in analyses, says Gelman, while providing enough rigour to reduce the number of false alarms being published.

More broadly, researchers need to realize the limits of conventional statistics, Goodman says. They should instead bring into their analysis elements of scientific judgement about the plausibility of a hypothesis and study limitations that are normally banished to the discussion section: results of identical or similar experiments, proposed mechanisms, clinical knowledge and so on. Statistician Richard Royall of Johns Hopkins Bloomberg School of Public Health in Baltimore, Maryland, said that there are three questions a scientist might want to ask after a study: 'What is the evidence?' 'What should I believe?' and 'What should I do?' One method cannot answer all these questions, Goodman says: “The numbers are where the scientific discussion should start, not end.”

CandySwipe Open Letter to King regarding trademark.

Wed, 12 Feb 2014 09:57:31 -0800

Comments: "CandySwipe Open Letter to King regarding trademark."

URL: http://www.candyswipe.com/king.html

Scaling Asana.com - Asana Engineering Blog

Wed, 12 Feb 2014 10:36:32 -0800

Comments: "Scaling Asana.com - Asana Engineering Blog"

URL: http://eng.asana.com/2014/02/scaling-asana-com/

This post isn’t about the Asana app – it’s about Asana.com, our “content site” – the place where we sign up new users, market the app & ecosystem, publish help and other content, recruit new Asanas, and more. Ultimately, it is the place where we tell our story to the world.

When people think about web scalability, they often think about requests, servers, and load balancers. We use (and love) Elastic Beanstalk, so this isn’t a problem for us. The real issue is people: as Asana has grown from 20 to 60, and soon to 100, the challenge is creating a system where many people from different teams can contribute to the various parts of our site.

Every team that reaches a certain size has this problem, and it is a much harder scaling challenge than adding servers. Teams that overcome it have great websites – where content is updated and the story evolves effortlessly with the team.

At Asana, we’ve figured out a way to rapidly deploy changes to our content site. Non-technical teammates can jump right in and see their updates realtime, without bottlenecks, and without requests to developers. Everything we do is version controlled with Git – even copy changes. Virtually every team is contributing in some way to the site every week, and we do it all without a dedicated web developer. We’re saving so much time that we can write blog posts about it.

Here is how we do it:

Everyone learns Markdown; all content is in Markdown

Markdown is the minimum required skill to contribute to Asana.com. You don’t have to know code. Luckily, if you can write, you can write Markdown. Everyone at Asana reads this Daring Fireball article to get familiar with the conventions. If you need to use an HTML snippet, such as a video embed code, Markdown parses it without a problem.

Every page on our site is based on a Markdown file: every landing page, every help article, every job listing, and every showcase app. Literally every word on our site is in a Markdown file where anyone on our team can edit it.

We put the Markdown in Statamic, a file-based CMS

Statamic has an intuitive folder structure that anyone can understand. Content contributors know they can jump right into the _content folder and edit the Markdown files. Designers and developers know they can use templates and partials in virtually unlimited ways – we’ve never had to say “no” to a design because of limitations related to our CMS.

We make our changes in Github

Our content site is in a private Github repository that all Asanas have access to. We create and edit the Markdown files directly through their web interface. Technical teammates can clone the repo locally and use whatever tool stack they choose.

Non-technical teammates don’t need to know Git – they just need to know that “commit” basically equals “save” – and that they can’t break anything because it’s all version controlled. The Github editor renders the Markdown for you when you commit – so you can see that your styling worked. But even better than that, we’ve found a way to show Asanas their changes immediately on the website itself. Here’s how…

Our staging server is realtime

Using Github’s Post-Receive Hooks, every time anyone commits, we ping our staging server to pull and deploy a fresh version of our master branch. The person who added content can immediately see their change live on our staging server – and share it with the team if it requires feedback.

Why is this such a big deal? The vast majority of contributors don’t need a local development environment – they just need a Github account and a web browser. In fact, they don’t need technical skills of any kind.

The work in Github is linked to Asana

Not surprisingly, we use Asana to track all the work we’re doing across the company. Using the Asana Service Hook (available to everyone), we seamlessly connect Asana to Github. Anytime anyone commits, they add the relevant Asana #task_id to the commit message; this comments on the associated task in Asana with a link to the commit in Github, showing the whole team that the work has been done.

Before deploying, we test the site with CasperJS

CasperJS is a web testing utility written on top of PhantomJS. Before we push any change to our production server, we run a quick suite of tests to make sure that links resolve, resources load, and there are no Javascript errors. CasperJS can also do CSS regression testing by comparing screenshots.

Deploy daily

After running tests, I review the changes to the master branch since our last deployment (which are mainly Markdown changes – PHP/MySQL, HTML, JS, or CSS changes require a more thorough code review). We deploy daily to our production servers using the Elastic Beanstalk CLI.

This process has profound effects on our team and our site. Adding content to Asana.com is exciting – a recruiter presses “commit” and sees a new job listing; someone on user operations improves a support article and immediately sees the results on a staging server. This decentralizes responsibility, makes peoples’ jobs more fun, and gives Asanas a strong sense of ownership over the content that relates to their function.

It also gives us tremendous leverage (and we’re all about that). This January, we completely redesigned a large portion of our site called the Asana Guide. The effort involved a dozen people writing, editing, changing design, updating links and navigation, and adjusting our CMS – at the same time. We tracked the work in Asana, did much of the work in Github, and watched it all unfold on a staging server… and we launched the site ontime without any issues.

We are frequently asked by other startups (and our customers) how we keep our content site so up to date. The answer is simple: with the right tools, it’s a team effort.

Saying Goodbye To Python

Wed, 12 Feb 2014 11:31:42 -0800

Comments: "Saying Goodbye To Python"

URL: http://www.ianbicking.org/blog/2014/02/saying-goodbye-to-python.html

This post is long overdue; this isn’t a declaration of intent (any intent was long ago made real), just my reflection about my own path. I left the Python world a long time ago but I never took a chance to say goodbye.

While I had moved on from Python years ago, I felt a certain attachment to it well past then, not quite admitting to myself that I wasn’t coming back. When my proposal for PyCon 2013 was rejected I was frustrated (it was going to be a fun talk!) but for some reason that made me fully realized that I wasn’t part of the Python community anymore.

Python was the first — and I sometimes wonder if the only — programming community I was part of. Coming to Python was a conscious choice. In college I was interested in Scheme and Smalltalk. High-minded languages with interesting ideas, but hard to find practical uses. Scheme was always too underserved by its libraries and sparse academic community. In contrast Smalltalk was productive — it was built by and for people who loved to build things. (More credit here should probably go to Dan Ingalls than Alan Kay — I can sense in Dan Ingalls’s work a real passion for making things, and a pure but unproductive language would not have satisfied him.) But Smalltalk was and is a world of its own. It was culturally and technically pre-internet, pre-open-source, pre-online-community. And despite all the great things about the Smalltalk environment and language it couldn’t fulfill these then-new potentials, event as it tried to adapt. (I wrote a couple posts about this, intended as a sort of sympathetic explanation of why I couldn’t stick with Smalltalk: Where Smalltalk Went Wrong and a followup.)

At the end of my college career (1999ish) I consciously looked for a new home. I flirted with Perl, C, but knew they weren’t for me. Somewhere along the way I came upon Python, and it was good enough that I didn’t look for anything better. I can’t say I fell in love with Python like I had with Scheme and Smalltalk — Scheme was like an opening up of the world after going far beyond what anyone should ask of GW-BASIC, and Squeak/Smalltalk was a deep mysterious world, like coming upon the ruins of an ancient and advanced civilization. Python in comparison was practical — but I wasn’t in the mood at that time to discover, I wanted to build.

And I built a lot of things in Python. I was doing workaday web programming and my enthusiasm went more towards building tools to build stuff than in the building itself. I did build some cool products in those days, not just libraries, but for some reason it’s only the smaller units that I was able to push out. And I found a community in Python.

And I built. Back in the days I contributed to Webware, what felt like a completely different generation of web development in Python than today. I wrote SQLObject, my first foray into a oh-shit-people-are-using-this-I’m-not-sure-how-I-feel-about-that library. But SQLObject explored a lot of metaprogramming concepts that were quite novel in the Python world at that time. At the same time maintaining it felt like a terrible burden. It took me far too long to resolve that, and only once interest had died down (in no small part due to my lack of attention) did I hand it over to Oleg who has been a far more steady hand. This would be a pattern I would unfortunately repeat. But if SQLObject helped the next generation be better that’s good enough for me.

Later came WSGI which excited me with its subtly functional basis. I built a whole web framework toolkit (or a framework for building web frameworks?) in Paste. Few people really understood what I was trying to do with Paste — at times including me. Some people like Ben Bangert were able to see the principles underneath the code, and get them out into the world in the kind of usable state that I intended to enable. (And along the way I sometimes felt like I was doing the same for PJE.) Python web frameworks were a mess back then; ultimately reaching for the crown using a monolithic approach was a more successful technique than trying to build bridges as I attempted with Paste. I’m still not sure what lesson to take from that. Not a general lesson, but more understanding the landscape. And understanding what you can bring to a problem, and who you can bring with. I feel like I’m only now really understanding the importance of vision combined with a diverse group of skills and perspectives, and I have even more to learn about how to actually assemble and coordinate the right group of people in the right environment to succeed.

Somewhere around here I feel like I reached my 10,000 hours of Python coding. I wrote WebOb, taking the lessons of Paste and a better intuition for library design. I still think it’s the best mapping of HTTP to Python. Other libraries include more aspects of web development in their scope, or have better documentation, and more users, but when viewed with a particular lens I’m still very proud of WebOb. And it’s been an important building block in a lot of people’s explorations into building a web framework. Other libraries from this period are lxml.html, smaller things like WebTest, ScriptTest, MiniMock, Tempita, perhaps the over-ambition (or just mis-ambition) of Deliverance. That period felt like a clearing out of my system, unloading a bunch of ideas.

But for whatever reason my most successful tools were virtualenv and pip. These were never my greatest passions, or even close. They were about: (a) fixing personal annoyances in deployment (virtualenv), and (b) getting people to stop fucking whining about Setuptools and easy_install (pip). I’m not sure whether I blame the uneasy success of these tools on broad appeal, or that they are in a sense user-visible tools and not libraries, or that I didn’t like doing them because no one liked doing them and so there was a vacuum waiting to be filled.

And then my last project, Silver Lining. It was early on in the devops revolution, an attempt to think about what a generic container for web applications might look like. It was in a sense going deeper down the hole of virtualenv and pip, but with an aim to build a full product and not just a set of eclectic tools. No one cared. And I only cared a little — I cared because it was completing some ideas I’d long had about deployment, because for a certain kind of web application development it felt nimble and reliable, because it removed or automated tasks I didn’t like to do. But I didn’t care, no more than I cared about virtualenv or pip. I’d gone down a path that was about code and technical design, but if I stepped back it was unexciting.

And when I did step back there wasn’t anything in Python that excited me. Python was doing great, my interest had nothing to jumping on or off bandwagons. Python is doing great — better than ever (minor bumps with versions aside). But I think in my mind I’d always imagined I could build up just the right toolset, and using that toolset create the product I actually wanted to create — what exactly that product was, I don’t know, but with the right tools I imagined I could move fast enough and confidently enough to find it.

Sometimes when I’m feeling particularly excited about an idea, like really excited, I have to take a break. I need to calm down. Try to wrap my head around the ideas, because I know if I push forward directly that I’ll just muddle things up and feel disappointed. No, I don’t know that is true: maybe I don’t want to have to confront, in that moment, that the idea is not as cool as I think it is, or as possible as I think it is. But often I do step back into the problem, with ideas that are more mature for having thought more deeply about them. In a sense I think creating tools and libraries was a similar process: I felt too excited about creating something great, because I worried I’d muddle everything up, or afraid I just couldn’t pull it off, and so I stepped away and would work on tools.

I always directed my attention to the web, even if I got bogged down in the server. Somehow I skipped native GUIs, even as a user. But pure data processing without consideration for what you do with the data felt unexciting. And ops — that’s just the worst. I was, and am, a true believer in Free Software; and I was, and am, a true believer in the web. That is, I don’t see either as simply a means to an ends. But not for the same reasons, and I can believe in a web that isn’t open source, and open source that isn’t for the web. And yet writing it down I realize I don’t care about open source that isn’t for the web.

When I stepped back Python no longer seemed relevant to the web, at least not the part of the web that interested me. The tools I had built were no longer relevant either, they were not the tools with which I could realize my ambitions. The database-backed website, or the dynamic-HTTP-based web application, templates and deployments, anything you’d call “REST” — none of it seemed like the future, and whatever this vague thing was that I’d been looking for, I wasn’t going to find it there.

This wasn’t an actual revelation, I’m constructing it in retrospect. If you’d asked me I would have agreed with this notion even years earlier, and it’s not like I came up with some unique idea, if anything I would call it self-evident, don’t we all know this is where the world is going? And so I started to look towards Javascript and the browser and the DOM.

Somewhat before this I also joined Mozilla. But it would be backwards to say that Mozilla induced this change in perspective, that it tempted me away from Python. In fact I would have had a much easier time of it if I had just stuck with doing Python backend stuff at Mozilla.

The last few years of transition have been a struggle. With Python and the server I knew what I was doing. I was good at it, I felt competent. I could construct an opinion with confidence about all kinds of design questions. I was respected and my opinion would be listened to. I’d put in my 10,000 hours, I had achieved mastery.

Moving to Javascript none of this was true, and most of it still isn’t true. It might be easier to pull off this change if I was doing web development, surrounded with people making similar transitions, a little fish in the little pond of whatever group I was working with. But Mozilla is not that kind of environment. Which is okay — if I had been felt confident it would only be because there was no one to correct me.

It’s oddly common to see people talk about how a programmer can pick up something new in the matter of a few days or months. To find programmers that consider all that knowledge transferable (for instance). I don’t know what to make of it — my less forgiving self thinks these people have never known what real mastery is. I don’t think it takes another 10,000 hours to get mastery in a new language and environment… but it definitely takes some thousands of hours, some years of hard work. I only now feel like I’m getting close.

Maybe it’s my perspective on what mastery is. Deciding to do something and then doing it is good. It is not mastery. You have to pick the right problem to solve. You have to pick the right way to solve it. You need to know when to revise that plan, and understand the constraints that inform that revision. You need both large scale and small scale intuitions. And you need to be good enough at all the details of programming in that environment that you don’t get overwhelmed with the “easy” stuff, so you have mental energy to spare on the big stuff. The jump from Python to Javascript isn’t that big, the languages have a very similar shape. And the browser was already the environment focused on. And yet redeveloping my intuition for this new environment has taken time.

Sadly I’m not going to get back where I was, because Javascript is not Python. If there’s a Javascript community I haven’t found it, or it’s at least not a single entity. There is no community that created Javascript like the Python community created Python. Python comes from the internet in a way Javascript does not; Javascript was built for the internet, but Python was built on the internet. And I do miss the Python community, you’re good people.

But also whatever language partisanship I had is gone, and won’t come back in the guise of a new favorite language. This shouldn’t be confused with a disinterest in language. I still get as annoyed as ever by “use the right tool for the job” — the bland truism meant to shut down critical discussion and engagement with the tasks and choices in software engineering, replacing it with a weak passionless technical fatalism.

I suppose it is the platform that I am drawn to now before language. And the browser seems like the most interesting platform, not because it’s novel (though it is, it’s a runtime like few others), but because of how concrete it is, and of course how relevant it is to… anything. And the browser is no longer just the servant of a server, I prefer now to think of the browser as an independent agent, connecting to services instead of servers. Obviously that doesn’t describe a great number of running web sites, but it’s the model I see for the future, and a better perspective for understanding future architectures.

Still this only addresses which direction I’m looking towards, I still have to walk the path. I don’t want to get caught up in the weeds again, building tools for something I never manage to make. Right now I think I’m on to something in the area of collaboration, first with TogetherJS and now I’m thinking bigger with a new experiment. But while I feel like I’ve reached some competence in executing on these projects, programming is only one piece of bringing forward a larger vision. I still have a lot of learning to do, skills for which I haven’t put in the necessary time. How to recruit support, how to manage the project, how to negotiate between feasibility and user value, how to negotiate compromises in strategy and design. And collaboration itself is a whole domain of expertise. I’ve learned a lot, I can do things, but I am definitely not yet experienced enough to choose to do the right thing in these areas. And at this moment I’m worried I won’t have the room to learn these things, it feels like time is running out just when I’m pulling stuff together.

Anyway, that’s where I am now. No longer a language partisan, unclear of what community I am even participating in, I am less sure how to identify and self-identify myself. How do I describe myself now? Even as I find my technical footing I am still adrift. And so it’s hard to say goodbye. So instead I’ll say, Pythonistas: until we meet again. Maybe I’ll meet some of you over here.

Scientists Say Their Giant Laser Has Produced Nuclear Fusion : The Two-Way : NPR

Wed, 12 Feb 2014 12:28:19 -0800

Comments: "Scientists Say Their Giant Laser Has Produced Nuclear Fusion : The Two-Way : NPR"

URL: http://www.npr.org/blogs/thetwo-way/2014/02/12/275896094/scientists-say-their-giant-laser-has-produced-nuclear-fusion

hide captionThe National Ignition Facility's 192 laser beams focus onto a tiny target.

LLNL

The National Ignition Facility's 192 laser beams focus onto a tiny target.

LLNL

Researchers at a laboratory in California say they've had a breakthrough in producing fusion reactions with a giant laser. The success comes after years of struggling to get the laser to work and is another step in the decades-long quest for fusion energy.

Omar Hurricane, a researcher at Lawrence Livermore National Laboratory, says that for the first time, they've produced significant amounts of fusion by zapping a target with their laser. "We've gotten more energy out of the fusion fuel than we put into the fusion fuel," he says.

Strictly speaking, while more energy came from fusion than went into the hydrogen fuel, only about 1 percent of the laser's energy ever reached the fuel. Useful levels of fusion are still a long way off. "They didn't get more fusion power out than they put in with the laser," says Steve Cowley, the head of a huge fusion experiment in the U.K. called the Joint European Torus, or JET.

The laser is known as the National Ignition Facility, or NIF. Constructed at a cost of more than $3 billion, it consists of 192 beams that take up the length of three football fields. For a brief moment, the beams can focus 500 trillion watts of power — more power than is being used in that same time across the entire United States — onto a target about the width of a No. 2 pencil.

The goal is fusion: a process where hydrogen atoms are squeezed together to make helium atoms. When that happens, a lot of energy comes out. It could mean the answer to the world's energy problems, but fusion is really, really hard to do. Hurricane says that each time they try, it feels like they're taking a test.

hide captionInside a capsule the width of a No. 2 pencil sits a tiny ball of hydrogen fuel. The lasers squeeze the fuel until it fuses, releasing energy.

E. Dewald/LLNL

Inside a capsule the width of a No. 2 pencil sits a tiny ball of hydrogen fuel. The lasers squeeze the fuel until it fuses, releasing energy.

E. Dewald/LLNL

"Of course you want to score real well, you think you've learned the material, but you just have to see how you do," he says.

Over the past few years, NIF has been getting a fat "F." For all its power, it just couldn't get the hydrogen to fuse, and researchers didn't know why. The failures have led NIF's critics to label the facility an enormous waste of taxpayer dollars. In 2012, the government shifted NIF away from its fusion goals to focus on its other mission: simulating the conditions inside nuclear weapons.

But the fusion experiments continued, and Hurricane says researchers now understand why their original strategy wasn't working. In the journal Nature, he and his colleagues report that they've finally figured out how to squeeze the fuel with the lasers. By doing a lot of squeezing right at the start, they were able to keep the fuel from churning and squirting out. The lasers squeezed evenly and the hydrogen turned into helium.

The new technique can't reach "ignition," which is the point at which the hydrogen fusion feeds on itself to make more. Even so, JET's Cowley says, this is still a big moment for NIF.

"I think it's still a very important step forward, they reached fusion conditions, they made some fusion happen, and that's not been done before [with a laser]," he says.

Hurricane says no one knows for sure whether NIF can really reach the point of ignition. "It's not up to me; it's up to Mother Nature," he says. "But we're certainly going to try."

Obama Sued by Rand Paul Over Phone Surveillance Program - Bloomberg

Wed, 12 Feb 2014 13:47:52 -0800

Comments: "Obama Sued by Rand Paul Over Phone Surveillance Program - Bloomberg "

URL: http://www.bloomberg.com/news/2014-02-12/obama-sued-by-rand-paul-over-surveillance-as-challenges-grow-1-.html

President Barack Obama was sued by Senator Rand Paul over U.S. electronic surveillance he claims is illegal, adding to challenges that may land post-Sept. 11 government data collection in the U.S. Supreme Court.

The Kentucky Republican announced today that he had filed his complaint in Washington federal court. Paul was joined as co-plaintiff by FreedomWorks Inc., a Tea Party-backed group. The filing couldn’t be immediately confirmed in court records.

The government is collecting phone data about U.S. citizens “without any belief by defendants at the time of collection or retention or searches that any of the information is connected with international terrorism or an international terrorist organization,” in violation of the U.S. Constitution’s Fourth Amendment prohibition of unreasonable searches, according to a draft copy of Paul’s suit provided by his office.

Consumers’ willingness to provide companies with information about themselves to get phone service “does not reflect a willingness or expectation that they are surrendering the privacy of the information,” Paul said in his complaint.

The suit challenges the National Security Agency’s bulk collection of phone records of millions of Americans, a program disclosed last year by former agency contractor Edward Snowden.

Caitlin Hayden, a spokeswoman for the National Security Council, referred a request for comment on the suit to the Justice Department. The council, a White House group, consists of administration advisers, mostly from the Cabinet and the military.

Found Legal

“We believe the program as it exists is lawful,” Hayden said by e-mail, addressing the data collection generally. “It has been found to be lawful by multiple courts. And it receives oversight from all three branches of government.”

“We remain confident that the Section 215 telephone metadata program is legal, as at least 15 judges have previously found,” Peter Carr, a Justice Department spokesman, said in an e-mailed statement. White House spokesman Jay Carney declined to comment on the specific litigation. He repeated Obama’s position that the program is lawful and has been upheld by courts.

A federal judge in New York ruled Dec. 27 that the program is legal. The ruling came less than two weeks after a federal court in Washington said it may be illegal. The two judges came to opposite conclusions about a landmark 1979 ruling on telephone data in the pre-Internet age.

A divided U.S. privacy-policy board last month concluded the NSA program is illegal and should be stopped.

‘Minimal’ Usefulness

The five-member Privacy and Civil Liberties Oversight Board, created by Congress under post-Sept. 11 anti-terrorism laws, said in a 238-page report that the program to collect and store the records has provided only “minimal” help in thwarting terrorist attacks.

The NSA receives phone records from U.S. telecommunications companies and stores them in a database that can be queried to determine who is in contact with suspected terrorist organizations.

The surveillance was authorized by President George W. Bush after the Sept. 11, 2001, terrorist attacks,. It has been defended as “critically important” to national security, according to records declassified this month by National Intelligence Director James Clapper.

In the two court rulings, U.S. District Judge William H. Pauley III in Manhattan granted a government motion to dismiss a suit filed by groups led by the American Civil Liberties Union.

D.C. Decision

In Washington, Judge Richard Leon barred collection of metadata from the Verizon Wireless accounts of the two plaintiffs. Leon suspended the injunction for a government appeal.

The ACLU appealed Pauley’s ruling to the federal Court of Appeals in New York. If appeals courts uphold their respective lower courts, creating a split, the Supreme Court is more likely to take the case.

The information at issue in all three cases involves “metadata,” which includes the numbers used to make and receive calls and their duration.

It doesn’t include information about the content of the communications or the names, addresses or financial information of parties, according to government filings in the Washington case.

To contact the reporters on this story: Andrew Zajac in federal court in Washington at azajac@bloomberg.net; Phil Milford in Wilmington, Delaware, at pmilford@bloomberg.net.

To contact the editor responsible for this story: Michael Hytha at mhytha@bloomberg.net.

Evan M Rose. - Last Thursday, my brother, Stephen Rose fought...

Wed, 12 Feb 2014 14:30:24 -0800

Comments: " Evan M Rose. - Last Thursday, my brother, Stephen Rose fought... "

URL: http://evanmrose.tumblr.com/post/76463244600/last-thursday-my-brother-stephen-rose-fought

Last Thursday, my brother, Stephen Rose fought courageously against symptoms of mental illness. In the last 12 hours of his life, he did all the things he was supposed to do to get help in a moment of crisis - made multiple calls to his care providers, to 911 and even went to the hospital. He had been experiencing symptoms for several years but was in a highly-respected treatment program that was working intimately with him. He was planning for his happiness, his dreams and his future. Nonetheless, in a sudden and surprising turn, Stephen lost the battle and passed from this world. But despite this loss, through the efforts he inspired in us, we will not lose the war.

We often have preconceived notions of what someone struggling with mental health issues looks like. Steve did not fit any of them. He was polite, athletic, good looking and highly intelligent. He was a Harvard graduate and achieved a GPA of 3.5 in his recently completed psychology masters program. He applied to law schools and was receiving acceptances and even full scholarships. He planned to visit one of his top law school campuses and had social events planned for the week. Steve struggled with, but was not defined by his mental health issues. While his affliction sought to isolate him, he fought to connect with others. Where anxiety would cripple most, Steve put on a brave face and did his best to reach out. He wasn’t a tortured soul, crumbling under the weight of an impossible burden. He was a warrior, fighting an invisible and silent assailant in a battle to the death. His story — and the stories of many others struggling silently — must be told for the war to be won.

Steve’s legacy is one of love for his family and friends, kindness to all, deep thought and desire to connect with others despite the obstacles mental illness presented in his life. One of his favorite quotes was by Oliver Wendell Holmes, Sr.: “The mind, once stretched by a new idea, never regains its original dimensions.” This perfectly encapsulates how he was approaching his life and what he wished for others. Stephen sought to connect deeply with the ideas he engaged with as well as the people he met. The week before he passed, he started a blog for this exact purpose. We believe that even in death, his ideals can inspire others to share and connect—both those who are struggling silently to tell their story and connect with others, and for others who are not.

What happened to Stephen and our family is something that “happens to someone else.” Mental health issues exist at and beyond the fringe of acceptable conversation. In certain populations, like African American men and women, there is stigma. People are afraid to speak about it and those afflicted are even more afraid to seek help for fear of being ostracized. In populations of ambitious young people who are high achievers, the challenges are even more profound. This cannot be allowed to continue.

Mental health disorders should be better understood by young people in our society, as many of these disorders strike people in their teens and twenties. Families, schools, and communities need to be empowered to spot, understand, and embrace those who are struggling. And our failsafe systems to prevent a tragedy such as Stephen’s must be truly failsafe.

Our family will be inviting others to join us in sharing Steve’s story in the hopes of opening the conversation around mental health and eventually making an impact so that other families don’t have to go through the pain of losing a loved one. We will also focus on promoting the values that defined Stephen—kindness and sensitivity to others, inclusiveness, and open dialogue about ideas and ideals.

If you’re interested in building Steve’s legacy or hearing about the memorial service, leave your contact information at: http://bit.ly/stevecrose.

For those interested: He had just started a Tumblr at http://scrose.tumblr.com We are probably going to make it a memorial and keep it alive.

robsheldon.com: Hello, world.

Wed, 12 Feb 2014 16:31:17 -0800

Comments: "robsheldon.com: Hello, world."

URL: http://www.robsheldon.com/blog/depression-programming/

So here I am again, in a mild funk. My sleep patterns are looking like the plot from a random number generator; I don't want to talk to anybody, and I try to get through the day using as few words as possible; I don't really want to do anything, though I keep forcing myself to spit out just enough work to keep most of the bills paid. Depression is rarely without a cause of some kind, and in this case there are a handful of causes that are converging down to a single sense of defeat: I am stuck.

Depression and burning out seem to be a common theme amongst programmers. I started thinking about some of the stuff I've read from fellow programmers about burning out, and some of the threads that have popped up periodically on programmer forums like HN regarding depression. I started wondering why it seemed more prevalent than in other professions.

I have a theory

Let's set aside for the moment some obvious assumptions: that, because I'm a programmer, I will be more aware of trends amongst programmers than I will be of, say, stage actors; and that programmers as a whole tend towards being introspective and analytical individuals who are sometimes articulate and interested in sharing their thoughts with others, and therefore more likely to think about and discuss the subject.

Let's instead just ask the question, "Is there something unique, or at least less common, about programming compared to other professions?"

I think there is.

It occurred to me while I was watching this video of some incredible talent that if you play the guitar, and play it well, you can demonstrate it to people. Even if people don't fully understand how long it took you to develop the skills to play a musical instrument that way, how much work and effort went in to it, or how talented you really are, they at least can have some appreciation of it. You are doing something that they can't do, and it sounds good and looks good besides.

But if you are a programmer, you can't do that. If you muster decades worth of practice and experience and, over some amount of time, write something brilliant — let's say, some kind of amazing new database interface, some piece of code architecture that will make something that was difficult into something that is now more straightforward — well, nobody else is really going to ever see that. What are you going to do, stand up in a coffee house one night and show off your code on a projector? "And this, ladies and gentlemen, is where I got the brilliant idea to use my new mutex to manage simultaneous multithreaded access to the same chunk without the risk of a race condition!" There's wild cheering, somebody buys you drinks, and you get to go home with a cute girl who thinks your brain is super hot.

No, not really.

What you can do is share your code with other programmers. I think that's the real value of things like Github. But, programmers are a notoriously savage bunch, and Github attracts top talent from all corners of the world. You are just as likely to hear a comment breaking down why your approach was wrong, or why you're using the incorrect syntax here or there, or why there's already some library by some other Joe Schmuck that does the same thing, only better.

Imagine if you were that guy playing that guitar, and at the end of it, some guy in the crowd gets up on stage and tells you, "Look, it's not that you're bad, it's just that you weren't chording some of those notes right. Look, I'll show you," and then you get completely schooled in front of everyone by some guy you've never met before.

Auto mechanics are about the closest thing I can think of in this regard. To many people, they take their car in to a mechanic, they say, "it's making this funny noise, like whirr-ker-chunk-ker-chunk-ker-chunk, but only when I'm coasting downhill," and then a few days and a few hundred or thousand dollars later it's fixed.

Magic. Except that, even then, at some point in their life most people put the hood of their car up at least once, or they try to fix something around the house, or they have a friend or family member that does basic simple repairs. So, people still have some notion of what it takes to be a good mechanic, even if that notion is wildly inaccurate.

But programming?

I've yet to meet the grocery clerk that says, "Yeah, so like I was browsing this website the other day, and I noticed it was doing something pretty neat with the menus, so I looked around in the code and whoever did it came up with this super cool approach..."

So at the end of several days' worth of programming, and problem-solving, and forward-thinking, all a programmer might get is a "thanks, now here's the next thing I need you to do."

Hueman, how are you?

Wed, 12 Feb 2014 18:57:58 -0800

Comments: "Hueman, how are you?"

URL: http://www.huemanapp.com/apple.html

Apple,
Have a Little Huemanity

We made something. We use it. We love it. Apple rejected it.

The Hueman Collective got together over a long weekend, made up of developers, designers, musicians and filmmakers. Our goal was simply to get together to make something that weekend; anything.

The seed of Hueman was planted by data: A friend that has methodically tracked his personal relative happiness level for 10 years. Walking us through his data, he could instantly recall events from his life that affected his happiness. We could clearly see the 3 month period when his previous long-term relationship was faltering, and later, the 6 months leading to the marriage proposal to his (now) wife. He was able to see the ebbs and flow of life.

We were psyched. We wanted everyone to be able to see their own data. That weekend, we built a prototype that asked you every day, How was today compared to yesterday?

Deceptively Simple

One bad day can't really ruin you, just like one amazing day doesnt actually change your life - but one day can start a domino effect. You don't always realize it's happening, you just know that you've felt shitty or great for a while.

By taking 10 seconds a day to answer one question, you quickly figure out a few things about yourself:

Are you an optimist or pessimist?
Is something happening that is out of the ordinary?
Is it time to make a change?

8 months of private beta later, what have we learned?

Each of these graphs are an individual's life map, the spikes and dips instantly recognizable. In these intial months, our beta users have excitedly shared their personal graphs with others. They have told stories in response to the "What the hell happened there??" question. They've taken a moment, daily, to think about how things are going.

In our beta metrics, the average session length was 31 seconds. The submission takes, on average, between 5-10 seconds. The remaining time seems to be spent on the graph view, seeing how their most recent submission stacks up against their history.

Take a moment to reflect

Thank you for your reply.

We have found the following issue with your app.

2.12

We found that your app only provides a very limited set of features. It only functions as a once a day mood tracker. While we value simplicity, we consider simplicity to be uncomplicated - not limited in features and functionality.

We understand that there are no hard and fast rules to define useful, but Apple and Apple customers expect apps to provide a really great user experience. Apps should provide valuable utility, draw people in by offering compelling capabilities or content, or enable people to do something they couldn't do before or in a way they couldn't do it before.

We encourage you to review your app concept and evaluate whether you can incorporate additional content and features to provide a more robust user experience. For information on the basics of creating great apps, watch the video "The Ingredients of Great Apps".

Apple seems to think Hueman lacks features and functionality, but they are completely missing the point.

For those that will argue that Apple rejected it for those reasons, the app is quite simple from a code perspective. It is written primarily in two simple web views: the submission screen and the graph.

If it's just using webviews, why does this need to be an app?

Hueman is free and anonymous, and to create that seamless experience, it needs to store data on your phone. If it was a web page on mobile safari, that data is more ephemeral.
The user needs daily local iOS Notifications to make Hueman part of their day.
The user needs the ability to share with other native apps such as Twitter, Facebook, Messages and more.
The next couple planned releases on our roadmap will heavily rely on native iOS functions and code to include things like tagging, additional graph views and scrubbing, ability to add media. etc... And by eventually letting people combine their data, you will be able to see how their relative happiness aligns to other users, a neighborhood and even the world.

Bring Hueman to life

Automatically tracking every instant, every moment, using technology is great. We do it ourselves, and have invested in fun upcoming products like Scanadu Scout Tricorder and TRACE. We're users of Moves and Strava, and the newly released Reporter from Feltron.

And for all the valuable data these apps collect, it still doesn't address the core value of Hueman. Hueman is about personal reflection, and presenting that data in a useful way.

It's been humbling for each of us to look back over the days and months that shape us, and look forward to when others can too.

How you can help

We'd love to hear your thoughts. Please contact us via Twitter, Facebook or email and let us know what you think of our app concept and the situation we're in. Please share this and help spread the word. Together, we can make this simple, yet extremely valuable app freely available to everyone.

Sincerely, The Hueman Collective

Comcast Acquiring Time Warner Cable in All-Stock Deal Worth $45 Billion - WSJ.com

Wed, 12 Feb 2014 19:22:58 -0800

Comments: "Comcast Acquiring Time Warner Cable in All-Stock Deal Worth $45 Billion - WSJ.com"

URL: http://online.wsj.com/news/articles/SB10001424052702303704304579379801986541412

Comcast Corp. said it agreed to buy Time Warner Cable for about $45.2 billion in stock, in a deal that would combine the nation's two biggest cable operators.

The boards of both companies have approved the transaction, which was announced Thursday morning.

With the proposed deal, Comcast almost certainly ends an eight-month takeover...

The Facebook Comment That Ruined a Life - Page 1 - News - Dallas - Dallas Observer

Wed, 12 Feb 2014 23:31:54 -0800

Comments: "The Facebook Comment That Ruined a Life - Page 1 - News - Dallas - Dallas Observer"

URL: http://www.dallasobserver.com/2014-02-13/news/the-facebook-comment-that-ruined-a-life/

Approximately one hour after Justin Carter posted a sarcastic comment on a Facebook thread, his life began to unravel.

The first reaction occurred behind the scenes, in another country. The 18-year-old Carter had no way of knowing that, while he did grunt work at a drapery shop in San Antonio, a person in Canada saw his comments — posted 60 days after the Sandy Hook school-shooting tragedy in Newtown, Connecticut — freaked out and initiated a 24-hour chain reaction of insanity that would wind up with Carter facing 10 years in prison.

Carter's comments were part of a duel between dorks, and may have had something to do with a game with strong dork appeal called League of Legends. But the actual details and context of the online exchange are, in the eyes of Texas authorities, unimportant. Prosecutors say they don't have the entire thread — instead, they have three comments on a cell-phone screenshot.

Prosecutors have failed to produce the entire thread containing Carter’s alleged threat, according to his attorney, Don Flanary.

Josh Huskin

Attorney Don Flanary of San Antonio says Justin Carter was coerced into confessing something that wasn’t even a crime.

One of the comments appears to be a response to an earlier comment in which someone called Carter crazy. Carter's retort was: "I'm fucked in the head alright, I think I'ma SHOOT UP A KINDERGARTEN [sic]."

Carter followed with "AND WATCH THE BLOOD OF THE INNOCENT RAIN DOWN."

When a person writing under the profile name "Hannah Love" responded with "i hope you [burn] in hell you fucking prick," Carter put the cherry on top: "AND EAT THE BEATING HEART OF ONE OF THEM." (The Austin police officer who wrote up the subsequent report noted: "all caps to emphasize his anger or rage." )

That's when someone in Canada — an individual as yet unidentified in court records — notified local authorities. Because Carter's profile listed him as living in Austin, the Canadians sent the tip to the Austin Police Department. Along with a cell-phone screenshot of part of the thread and a link to Carter's Facebook page, the tipster provided this narrative: "This man, Justin Carter, made a number of threats on Facebook to shoot up a class of kindergartners. ... He also made numerous comments telling people to go shoot themselves in the face and drink bleach. The threats to shoot the children were made approximately an hour ago."

The information was forwarded to the Austin Regional Intelligence Center, an information clearinghouse for law enforcement agencies in Travis, Hays and Williams counties.

Center personnel ran Carter's name, found either a driver's license or a state ID card and discovered that the address listed was "within 100 yards" of Wooldridge Elementary School. Based on a Travis County prosecutor's belief that there was probable cause to charge Carter with a third-degree terroristic threat — which carries a penalty of two to 10 years — a judge issued an arrest warrant. U.S. marshals traced Carter to the drapery shop in San Antonio, where he worked, and handcuffed the cherub-faced, brown-haired teen. Until that point, his only brush with the law was a temporary restraining order two years earlier.

After his booking into the Bexar County Jail, authorities discovered that he actually lived in New Braunfels — Comal County. After his transfer there, his bond was increased from $250,000 to half a million dollars.

According to Carter's attorney, Don Flanary, the 18-year-old suffered brutal attacks in the Comal County Jail during the four months he was held there.

Police records allege that, upon being booked into Bexar County Jail, Carter stated, "I guess what you post on Facebook matters."

He had no idea.

When officers searched Carter's home, Flanary says, they did not find the hallmarks of a lunatic.

"They found no guns in his house," Flanary says from his San Antonio office. "They found no bomb-making materials." He follows this up with a dash of sarcasm that's not a far stretch from the rhetorical flourishes that put his client in peril: "They didn't find The Anarchist Cookbook. ... They didn't find, you know, a bunch of newspaper clippings on the wall — conspiracy theories, with yarn from one place to another. They didn't find pentagrams and candles. He wasn't listening to Judas Priest."

Flanary's explanation for this is simple: His client is not a nut. But Flanary can't say the same for the jam his client's in. "This whole thing is totally and completely bonkers."

In the absence of any other evidence mentioned in Comal County prosecutor Laura Bates' filings, it's hard to disagree. Despite repeated calls, the Houston Press was unable to speak with Bates or anyone else in the Comal County District Attorney's Office — a receptionist told us that the only person authorized to speak to the media was District Attorney Jennifer Tharp herself, and she was unavailable.

One of the most striking things about the evidence so far tendered by the state is what's missing: the entire thread — which wasn't on Carter's Facebook page — containing the damning comments.

"The state tells us Facebook didn't give it to them," Flanary says. He's unsuccessfully tried to find "Hannah Love," the only other profile included in the cell-phone screenshot; at this point, it's still unclear whether "Hannah Love" is the anonymous Canadian tipster.

Flanary believes it's paramount that if someone is criminally charged on the basis of his words, a jury needs to see all the words. In this case, that includes whatever comment precipitated Carter's hyperbolic rant.

Dug.js — A JSONP to HTML Script — Rog.ie

Thu, 13 Feb 2014 01:34:42 -0800

Comments: "Dug.js — A JSONP to HTML Script — Rog.ie"

URL: http://rog.ie/blog/dugjs-a-jsonp-to-html-script

So you want to display your Dribbble shots, recent pins on Pinterest, 500px or Instagram photos, Github commits, or recently listened to music on your blog or site? Then this chunk of javascript is for you. It was designed to be a lightweight, simple, library-independent script to pull in feeds of content available on the web as JSONP (there's lots of em!) to display on your site.

I wrote this script because I wanted a dead-simple way to show my dribbble shots on my site. I wanted to be able to customize the markup with a custom template and have the content client side cache. I wanted to do the same with Pinterest inspiration that I was finding for illustrations that I love. Instead of having dedicated scripts for every service, why not just have one very small raw script (~200 lines) to do it all?

Dug.js Setup

Enough windbaggery, let me show you how to use it. Because Dug.js is on my site, I can run the code in this blog post. Let's talk about that Dribbble implementation first. So, I want to pull in my latest Dribbble shots. Scanning the Dribbble api, I see that they do support JSONP and that you can pull in all player shots with the following url: http://api.dribbble.com/players/:id/shots

Rad! With Dug.js, we're nearly there. In it's simplest form, you really only need two things.

An api endpoint (jsonp callbacks supported) An HTML template to display the data

With that in mind, let's setup dug.js to pull dribbble shots from an awesome illustrator's account:

<script>
 dug({
 endpoint: 'http://api.dribbble.com/players/justinmezzell/shots',
 template: '{{#shots}}<img src="{{image_400_url}}">{{/shots}}'
 });
</script>

If you're familiar with mustache or similar templating, the {{#shots}}...{{/shots}} section apply the html within it over every shot, in this case, emitting an image for each shot. Hint: you can also supply a CSS selector to target a template script, and Dug.js will pull the content from it.

That's a pretty simple example, but what if you want each shot linking to it's detail page? Here ya go:

<script>
 dug({
 endpoint: 'http://api.dribbble.com/players/justinmezzell/shots',
 template: '<ul>\
 {{#shots}}\
 <li>\
 <a href="{{url}}" title="{{title}}">\
 <img src="{{image_400_url}}" alt="Image of {{title}}">\
 </a>\
 </li>\
 {{/shots}}\
 </ul>'
 });
</script>

Here's an actual demo of the above code, running:

That's it! Pretty simple huh? Yeah, thats why I dig it too!

Dug.js Parameters

For those that are curious, here's a few parameters (as json name-value pairs) that you can pass into dug to do some additional schnazzy things:

target — the id of an existing DOM element to put the html results in.
cacheExpire — # of milliseconds to cache data on the client side (using localstorage). 0 for no caching.
callbackParam — the name of the query variable a JSONP service will use for a callback function. Most services just use 'callback=functionName', but sometimes a service will use a different query variable name.
success — a function to call when JSONP data is successfully retrieved.
error — a function to call when JSONP data is not successfully retrieved.
beforeRender @param data — a function called before Dug.js renders the template. Helpful for trimming/changing the data before it renders.
afterRender @param data — a function called after Dug.js renders the template.

Download Dug.js

Dug.js is completely free! Although, if you're diggin' this script hard and want to help fund my graphic tablet purchase (for art like Dug the dog above. Yay!), feel free to donate.

Download Dug.js

License

Dug.js is 100% free under the WTFPL — no link backs or anything needed.

Why Your Python Runs Slow. Part 1: Data Structures – Blog ~ Saulius Lukauskas

Thu, 13 Feb 2014 03:59:22 -0800

Comments: "Why Your Python Runs Slow. Part 1: Data Structures – Blog ~ Saulius Lukauskas"

URL: http://lukauskas.co.uk/articles/2014/02/13/why-your-python-runs-slow-part-1-data-structures/

Technical Details Behind a 400Gbps NTP Amplification DDoS Attack | CloudFlare Blog

Thu, 13 Feb 2014 04:07:22 -0800

Comments: "Technical Details Behind a 400Gbps NTP Amplification DDoS Attack | CloudFlare Blog"

URL: http://blog.cloudflare.com/technical-details-behind-a-400gbps-ntp-amplification-ddos-attack

On Monday we mitigated a large DDoS that targeted one of our customers. The attack peaked just shy of 400Gbps. We've seen a handful of other attacks at this scale, but this is the largest attack we've seen that uses NTP amplification. This style of attacks has grown dramatically over the last six months and poses a significant new threat to the web. Monday's attack serves as a good case study to examine how these attacks work.

NTP Amplification 101

Before diving into the particular details of this attack, it's important to understand the basic mechanics of how NTP amplification attacks work. This is a quick overview of how these attacks occur. John Graham-Cumming on our team previously wrote a detailed primer on NTP amplification attacks if you're interested in further technical details. If you're interested in amplification attacks, you may also find interesting our posts about DNS Amplification attacks. These attacks use a similar method but target open DNS resolvers rather than NTP servers.

An NTP amplification attack begins with a server controlled by an attacker on a network that allows source IP address spoofing (e.g., it does not follow BCP38). The attacker generates a large number of UDP packets spoofing the source IP address to make it appear the packets are coming from the intended target. These UDP packets are sent to Network Time Protocol servers (port 123) that support the MONLIST command.

I'd personally be curious to talk with whoever added MONLIST as a command to NTP servers. The command seems of such little practical use -- it returns a list of up to the last 600 IP addresses that last accessed the NTP server -- and yet it can do so much harm. If an NTP server has its list fully populated, the response to a MONLIST request will be 206-times larger than the request. In the attack, since the source IP address is spoofed and UDP does not require a handshake, the amplified response is sent to the intended target. An attacker with a 1Gbps connection can theoretically generate more than 200Gbps of DDoS traffic.

Not Just Theoretical

Monday's DDoS proved these attacks aren't just theoretical. To generate approximately 400Gbps of traffic, the attacker used 4,529 NTP servers running on 1,298 different networks. On average, each of these servers sent 87Mbps of traffic to the intended victim on CloudFlare's network. Remarkably, it is possible that the attacker used only a single server running on a network that allowed source IP address spoofing to initiate the requests.

While NTP servers that support MONLIST are less common than open DNS resolvers, they tend to run on beefier servers with fatter connections to the network. Combined with the high amplification factor, this allows a much smaller number of NTP servers to generate very large attacks. For comparison, the attack that targeted Spamhaus used 30,956 open DNS resolvers to generate a 300Gbps DDoS. On Monday, with 1/7th the number of vulnerable servers, the attacker was able to generate an attack that was 33% larger than the Spamhaus attack.

Globally Distributed Threat

We saw attack traffic hitting every one of CloudFlare's data centers. While we were generally able to mitigate the attack, it was large enough that it caused network congestion in parts of Europe. The map above shows the global distribution of the 4,529 NTP servers used in the attack. The chart below lists the AS Numbers and names of the top 24 networks we saw traffic from in the attack, as well as the number of exploited NTP servers running on each.

 ASN Network Count 
 9808 CMNET-GD Guangdong Mobile Communication Co.Ltd. 136
 4134 CHINANET-BACKBONE No.31,Jin-rong Street 116
16276 OVH OVH Systems 114
 4837 CHINA169-BACKBONE CNCGROUP China169 Backbone 81
 3320 DTAG Deutsche Telekom AG 69
39116 TELEHOUSE Telehouse Inter. Corp. of Europe Ltd 61
10796 SCRR-10796 - Time Warner Cable Internet LLC 53
 6830 LGI-UPC Liberty Global Operations B.V. 48
 6663 TTI-NET Euroweb Romania SA 46
 9198 KAZTELECOM-AS JSC Kazakhtelecom 45
 2497 IIJ Internet Initiative Japan Inc. 39
 3269 ASN-IBSNAZ Telecom Italia S.p.a. 39
 9371 SAKURA-C SAKURA Internet Inc. 39
12322 PROXAD Free SAS 37
20057 AT&T Wireless Service 37
30811 EPiServer AB 36
 137 ASGARR GARR Italian academic and research network 34
 209 ASN-QWEST-US NOVARTIS-DMZ-US 33
 6315 XMISSION - XMission, L.C. 33
52967 NT Brasil Tecnologia Ltda. ME 32
 4713 OCN NTT Communications Corporation 31
56041 CMNET-ZHEJIANG-AP China Mobile communications corporation 31
 1659 ERX-TANET-ASN1 Tiawan Academic Network (TANet) Information Center 30
 4538 ERX-CERNET-BKB China Education and Research Network Center 30

At this time, we've decided not to publish the full list of the IP addresses of the NTP servers involved in the attack out of concern that it could give even more attackers access to a powerful weapon. However, we have published a spreadsheet with the complete list of the networks with NTP servers that participated in the attack. While the per server amplification makes these attacks troubling, the smaller number of servers and networks involved gives us some hope that we can make a dent in getting them cleaned up. We are reaching out to network operators whose resources were used in the attack to encourage them to restrict access to their NTP servers and disable the MONLIST command.

Somewhat ironically, the large French hosting provider OVH was one of the largest sources of our attack and also a victim of a large scale NTP amplification attack around the same time. The company's founder Tweeted:

We see today lot of new DDoS attacks from Internet to our network. Type: NTP AMP Size: >350Gbps. No issue. VAC is great :)— Oles (@olesovhcom) February 12, 2014

Time to Clean Up the Problem

If you're a network administrator and on Monday you saw network graphs like the one in the Tweet below then you are running a vulnerable NTP server.

and here's what it looks like when a device participates in the NTP DDOS against @CloudFlare pic.twitter.com/QcrPGxbcUz— Eric C (@ctrl_alt_esc) February 12, 2014

You can check whether there are open NTP servers that support the MONLIST command running on your network by visiting the Open NTP Project. Even if you don't think you're running an NTP server, you should check your network because you may be running one inadvertently. For example, some firmware on Supermicro's ICMP controllers shipped with a MONLIST-enabled NTP server on by default. More details on NTP attacks and instructions on how to disable the MONLIST command can be found on the Internet Storm Center's NTP attack advisory.

NTP and all other UDP-based amplification attacks rely on source IP address spoofing. If attackers weren't able to spoof the source IP address then they would only be able to DDoS themselves. If you're running a network then you should ensure that you are following BCP38 and preventing packets with spoofed source addresses from leaving your network. You can test whether your network currently follows BCP38 using tools from MIT's the Spoofer Project. If you're running a naughty network that allows source IP address spoofing, you can easily implement BCP38 by following the instructions listed at BCP38.info.

Finally, if you think NTP is bad, just wait for what's next. SNMP has a theoretical 650x amplification factor. We've already begun to see evidence attackers have begun to experiment with using it as a DDoS vector. Buckle up.

Making Remote Work Work - Learning - Source: An OpenNews project

Thu, 13 Feb 2014 07:53:50 -0800

Comments: " Making Remote Work Work - Learning - Source: An OpenNews project"

URL: https://source.opennews.org/en-US/learning/making-remote-work-work/

Christopher Groskopf’s Tricks for Going to the Office without Going to the Office

For the last two-and-a-half years I’ve been a 100% remote worker, first on PANDA Project and now with NPR visuals. Recently, several people have asked me how I make this work. At the same time, I’ve taken on new responsibilities at NPR as a project manager. This change has brought into sharp relief the things I’ve learned about how to work effectively from far away. Whereas I used to only have to figure out how to coordinate with my own small team, I now routinely connect remotely with our stakeholders, sponsorship and marketing departments, member stations, and more.

My view for NPR visuals 10:00 am (ET) scrum. On this morning there were so many of us that several people are off the left and right sides of the camera.

In this piece, I’ll codify some of the lessons I’ve learned. If you’re already a remote developer, you may pick up some new tricks and a more rigorous way of thinking about how you work. If you are planning to go remote, oh boy, you should clear your calendar because I have so many things to tell you. Effective remote work comes down to maintaining those things which are necessary no matter where you work—organization, communication and motivation.

What my coworkers see. If you’re curious about the thing on the wall, well, so is everybody else. It makes for a great long-distance icebreaker.

Staying organized

It goes without saying that if you’re going to be apart from your team, you need to take responsibility for your own organization. What may be less obvious is that you’re going to need to take that organization to much greater lengths than would probably otherwise be necessary. Why? The lack of tangible reminders. It’s amazing how much we rely on subtle environmental and social cues for how and when things get done. If you never lock eyes with a homepage producer, you might forget to tell them about an impending launch.

In lieu of these contextual, human reminders, you’re going to need to rely on systems to help you remember. Whenever possible, you should use systems that allow you to collaborate directly with your coworkers. The most basic example of this is your office calendaring software. Going remote has taught me the value of having a schedule. Your calendar software does lots of smart things for you, like compensating for time zones and preventing conflicts. The most important thing it does for you is prevent other people from creating conflicts for you. (You can’t just “not be around” when you work remotely.)

Pro tip: if you’ve got school-age kids, schedule yourself for the half-hour they get out of school for the next ten years. In practice nobody will even notice you’ve done this and it will save you the hassle of having to reschedule a meeting because you have to pick your kids up.

Get yourself a nice multi-timezone clock. It’s invaluable and looks cool. The Nexus 7’s clock works great for this.

The flip-side of this is that you’ll often be scheduling meetings for a group of folks who aren’t remote. When you do this you need to keep their needs in mind. Don’t schedule meetings for ten people and just send them a Google Hangout URL. Make sure a room is available (preferably one on their floor), reserve it and then annotate the calendar event with the location. Do yourself a favor and appoint someone to make sure the A/V setup in the room works well a few minutes before the meeting starts. Taking up a dozen people’s time with your telepresence issues is even more annoying than just being late. Everyone can relate to being late, but not everyone can relate to being unintelligibly garbled or echoing like the Grand Canyon.

Pro tip: If your office has a phone system that uses short extensions internally, have your IT department set one up to redirect to your cell phone. That way you can tell your coworkers to dial “5555”, instead of your whole number.

Beyond calendaring and meeting coordination, it’s important to extend your organized persona into other interactions with your team. The best example I can offer of this is being rigorous in your usage of tickets/issues. If you’re a programmer this should already be second nature, but even if you aren’t, you should be using some way of tracking who is working on what. Making sure everyone is being prompt and thorough in their ticketing can massively reduce the friction of remote collaboration. It simply minimizes how often you actually need to talk by making that information readily available to everyone on the team.

Staying in communication

No ticketing system will allow you to completely avoid conversation, and the synchronous part of remote collaboration is really where the bytes hit the information super highway. It goes without saying that modern tools like real-time video chat have made this style of work possible, but they aren’t a cure-all and they can be used ineffectively. I recommend a two-pronged approach to synchronous communication: one-part text chat and one-part video chat. Using each to its maximum potential requires some consideration.

For text chat our team uses Hipchat, a wonderful piece of software that has Android, iOS, Windows, Mac, Linux and web clients. Among its charms are its image sharing integration and its capacity to quickly spin up ad-hoc chat rooms. This is our “always-on backchannel,” which is to say that during working hours our entire team is in this chat. This serves a number of purposes:

It reduces the number of unnecessary phone or video calls that get made, because, “Hey, did you get that thing I sent you?” is much faster in text.
It serves as a status indicator of who is at work (available = green light), who is at lunch (away = yellow light) and who is out (not online).
It can be ignored. Sometimes you just need to focus. Text chat is asynchronous when you need it to be.
It’s perfect for quickly bringing the entire team up to date on the latest funny GIFs of football players crying or cats dancing. (Or if the production servers are down.)

When the backchannel isn’t good enough, then you switch into video chat mode. Never use a phone if you can video chat. Body language matters, even when you’re talking about database normalization. It’s the difference between “I don’t want to do it this way” and “I will hate my job for the next week if we do it this way.” You should get comfortable jumping into a Hangout even if it’s only for a five minute conversation. In fact, you should get comfortable telling people to do it. Don’t be afraid. It may annoy somebody once, but they’ll quickly get used to it and it’s much better than the alternative miscommunications.

Pro tip: If you have a Hubot in your backchannel, you can snag our hangouts script to automatically generate new hangouts on demand!

Among the best things you can do for yourself as a remote employee is to get comfortable asking for help. You’re going to need somebody to make sure the A/V in the conference room works. You’re going to need somebody to wander the halls looking for that stakeholder who can’t find the conference room in the first place. You’re going to need somebody to assure less technically savvy people that your presence in the room is not magic. Don’t be afraid to ask for help. You’re all on the same team and it’s in everyone’s interest to make things work.

Remote-work gear worth having

Your work is going to depend on video chat, so invest in a good quality webcam. Expense it if you can, but if you can’t, buy one anyway. I recommend the Logitech C525. It’s simple, inexpensive, and a huge upgrade over even the current generation MacBook’s embedded webcam. Your own meaningful body language will suddenly be visible to the rest of the team.

While you’re shopping, buy a pair of document cameras too. This is one of those wonderful bits of technology you don’t know you need. They are especially helpful for UX wire-framing and UI sketching, but I bet you’ll find other uses for them too. I recommend the Ipevo Ziggi-HD. They also make a nice travel case for it if you, like me, work from more than one location.

A stereo A/B switch is a simple but invaluable device for your home office.

If you switch back and forth between headphones and speakers a lot you can build yourself a little stereo A/B switch like one pictured above for just a few bucks. If you’re not handy with a soldering iron you can buy a really overpriced one on Amazon. It will save you constantly plugging and unplugging your headphones and, as side-benefit, it also serves as an inline mute switch when you have telepresence echo issues. I can just leave my speakers off and flip over to them when I’m talking to mute the echo.

Staying motivated

At work you bring something to remind you of home. At home you need something to remind you of work. To wit, this lovely letterpress of our team motto by Alyson Hurt.

You can master organization and communication, but those things won’t keep you from sometimes feeling horribly disconnected from your team and your mission. Do not despair. This is normal. There are several things you can do to combat the disconnected feeling of working apart from your teammates.

The single best advice I got when I went remote was from Matt Waite, who said, “Put on pants,” by which I’m pretty sure he meant, “Act like you’re going to work.” Get up, put on clothes you’d leave the house in, take a look in the mirror, and go to your work space. It is essential that you have a room (or nook) in your house that you use only for work. You need a place to go to at the start of the day and leave at the end. I even put an office nameplate over mine. Do the same things you would if you were going to the office. This might sound silly, but it will help keep you sane. Think of your home workspace like an exclave of your company’s offices. Act like you might run into your editor or the CEO at any moment! It’ll make you feel normal.

Now entering my office.

That being said, you also own your office, so you don’t have to deck it out with cubicle swag. Make yourself a very comfortable space to work in. My recommendations: big speakers, high-quality headphones, a comfortable desk (and an Aeron or Leap chair if you don’t stand), some nice pictures, and maybe even windows. Embrace the fact that you get to work at home. Sometimes my stereo competes with the helicopters that land at the hospital down the street. My coworkers don’t know the difference.

Force yourself to work reasonable hours. This is an easy one to screw up. If you’re comfortable and you like your job it’s pretty easy to just keep working until it’s time for bed. Don’t do it—unless you’re on deadline and then only if you can’t cut features instead of working into the night. Take breaks, walk outside, drink coffee, etc. You wouldn’t spend four solid hours at your desk if you worked in the office. You would talk to your coworkers or go for a walk once in a while. Try to maintain that normalcy and it will help the rest to feel normal.

Pro tip: With a little bit of practice and the right tools, you can make a much better cup of coffee at home than your Starbucks barista can push out the drive-through window.

It’s my earnest belief that some people will have higher expectations for you because you work remotely. It’s very easy for them to believe you’re in your underwear playing Final Fantasy instead of slogging through the documentation for Django. Not all work has obvious output and when they can’t see you at your desk, it’s tempting to log those blank hours as time wasted. There is absolutely nothing you can do about this, so I suggest you embrace their high standards. Aim to excel. You already have a tremendous productivity advantage because no one can interrupt you at your own house unless you let them. Eventually you will win people over and it won’t matter so much anyway, but it may take you a while.

And a final word: visit the office. If you’re in the same foxhole, but you’re never celebrating together, then it just isn’t the same. Visit your team as often as is realistic. At NPR I plan to visit at least once every three months. When you’re there get outside the office together. It pays to remind one other that nobody is just a face on a screen.

A few words for programmers

I’ve attempted to construct my advice in a way that would be useful to programmers and non-programmers alike. In general the same challenges crop up, but there are a few specific nuggets of advice I can offer to programmers:

Consistency is even more important when you’re not in constant contact. If you don’t have a style guide, write one. It’s crucial towards alleviating the frustration that arises from inconsistent programming practices.
Documentation: just do it. You really have to. Absolutely nobody is going to remember how to load that database dump you created (or even where it is). Document your processes, preferably by making them dead simple to repeat. Have READMEs. Have a spreadsheet index of your servers. Every minute you spend writing documentation is five minutes saved talking on the phone.
You should be pair programming. Doing this remotely can be very tricky. It will never be as good as passing a keyboard around can be, but you should do it anyway. If a screen share in a Hangout is all you can manage then do that, especially if you have junior programmers who will benefit from the collaboration. tmux sessions can also be used to enable pairing in the command line, which is especially useful for vim users. For something more integrated, we’ve recently had very good experiences with Floobits, though it’s still relatively new, so our opinions are subject to change.

This is everything I’ve learned in two-and-a-half years of remote development. It’s a process I’m constantly iterating, especially now that I’m working more with folks outside our team. However, I think at the end of the day any future improvements I make will fall into these problem categories of organization, communication, and motivation. These are the dragons you’ve got to slay if you’re going to work far from your team. It is possible. It can be good. It’s not a choice you should make lightly, but if you decide to make the leap, you deserve to do it well.

Christopher Groskopf is a news applications developer and project manager on the NPR Visuals team. He divides his time between Tyler, TX and Oakland, CA. Find him on Twitter: @onyxfish.

Please enable JavaScript to view the comments powered by Disqus. comments powered by

Thu, 13 Feb 2014 08:31:12 -0800

Comments: ""

URL: http://akaros.cs.berkeley.edu/files/Plan9License

Ken Shirriff's blog: Bitcoin transaction malleability: looking at the bytes

Thu, 13 Feb 2014 08:31:13 -0800

Comments: "Ken Shirriff's blog: Bitcoin transaction malleability: looking at the bytes"

URL: http://www.righto.com/2014/02/bitcoin-transaction-malleability.html

"Malleability" of Bitcoin transactions has recently become a major issue. This article looks at how transactions are modified, at the byte level.

An attacker has been modifying Bitcoin transactions, causing them to have a different hash. Recently an attacker has been taking transactions on the Bitcoin peer-to-peer network, modifying them slightly, and rapidly sending them to a miner. The modified transaction often gets mined first, pre-empting the original transaction. The attacker can only make "trivial" changes to a transaction, so exactly the same Bitcoin transfer happens as was intended - the same amount is moved between the same addresses, so this attack seems entirely pointless. However, each transaction is identified by a cryptographic hash, and even a trivial change to the transaction causes the transaction hash to change. Changing the hash of a transaction can have unexpected effects on the Bitcoin system.

A very quick explanation of transactions

A Bitcoin transaction moves bitcoins from one address to another. A transaction must be signed with the private key corresponding to the address, so only the owner of the bitcoins can move them. (This signing process is surprisingly complex.) The signature is then put in the middle of the transaction. Finally, the entire transaction (including the signature) is cryptographically hashed, and this hash is used to identify the transaction in the Bitcoin system. The important data is protected by the signature and can't be modified by an attacker. But there are few ways the signature itself can be changed, but still remain valid.

(This is oversimplified. For more details, see Bitcoins the hard way.)

Looking at a modified transaction

To find a transaction suffering from malleability, I looked at the unconfirmed transactions page. If a transaction gets modified, only one version will get mined successfully (and actually transfer bitcoins), and the other will remain unconfirmed (and have no effect). Among the many conditions enforced in mined blocks, the same bitcoins can't be spent twice, so both transactions will never be mined. This is why having two versions of a transaction doesn't result in two payments.

I picked a random unconfirmed transaction from Feb 11 to examine. (Unfortunately this transaction has been discarded since I wrote this article, breaking my links. But you can look up a different one if you want.) Blockchain.info helpfully includes a banner warning that something is wrong:

Warning! this transaction is a double spend of 112593804. You should be extremely careful when trusting any transactions to/from this sender.

Looking at the transactions, everything seems fine:

The confirmed transaction takes 0.01 BTC from 1JRQExbG6WAhPCWC5W5H7Rn1LannTx1Dix and transfers 0.0099 BTC to 1Hbum99G9Lp7PyQ2nYqDcN3jh5aw878bFt (the remainder is a mining fee of 0.001 BTC). This transaction has hash bba8c3d044828f099ae3bc5f3beaff2643e0202d6c121753b53536a49511c63f.

The unconfirmed transaction takes 0.01 BTC from 1JRQExbG6WAhPCWC5W5H7Rn1LannTx1Dix and transfers 0.0099 BTC to 1Hbum99G9Lp7PyQ2nYqDcN3jh5aw878bFt (the remainder is a mining fee of 0.001 BTC). This transaction has hash d36a0fcdf4b3ccfe114e882ef4159094d2012bc8b72dc6389862a7dc43dfa61c.

The scripts of both transactions appear identical:

Input Scripts
30450220539901ea7d6840eea8826c1f3d0d1fca7827e491deabcf17889e7a2e5a39f5a1022100fe745667e444978c51fdba6981505f0a68619f0289e5ff2352acbd31b3d23d8701 046c4ea0005563c20336d170e35ae2f168e890da34e63da7fff1cc8f2a54f60dc402b47574d6ce5c6c5d66db0845c7dabcb5d90d0d6ca9b703dc4d02f4501b6e44 OK
Output Scripts
OP_DUP OP_HASH160 b61c32ac39c63f919c4ce3a5df77590c5903d975 OP_EQUALVERIFY OP_CHECKSIG

Both transactions look identical: the bitcoins are moving between the same accounts in both cases, the amounts are equal, and the scripts look identical. So why do they have different hashes? A clue is the unconfirmed transaction is 224 bytes and the confirmed transaction is 228 bytes.

Looking at the raw transactions also fails to show what is happening:

{
 "hash":"bba8c3d044828f099ae3bc5f3beaff2643e0202d6c121753b53536a49511c63f",
 "ver":1,
 "vin_sz":1,
 "vout_sz":1,
 "lock_time":0,
 "size":228,
 "in":[
 {
 "prev_out":{
 "hash":"3ceafb1d6864091a6c40f0f0fa7d4072d71a909820444ac307dcaa7a2d4b88d4",
 "n":1
 },
 "scriptSig":"30450220539901ea7d6840eea8826c1f3d0d1fca7827e491deabcf17889e7a2e5a39f5a1022100fe745667e444978c51fdba6981505f0a68619f0289e5ff2352acbd31b3d23d8701 046c4ea0005563c20336d170e35ae2f168e890da34e63da7fff1cc8f2a54f60dc402b47574d6ce5c6c5d66db0845c7dabcb5d90d0d6ca9b703dc4d02f4501b6e44"
 }
 ],
 "out":[
 {
 "value":"0.00990000",
 "scriptPubKey":"OP_DUP OP_HASH160 b61c32ac39c63f919c4ce3a5df77590c5903d975 OP_EQUALVERIFY OP_CHECKSIG"
 }
 ]
}

Even though the scripts are mostly in hex in this raw display, they have been parsed slightly, which hides what is going on. We need to get the full scripts here and here.

The unconfirmed transaction has script:

4830450220539901ea7d6840eea8826c1f3d0d1fca7827e491deabcf17889e7a2e5a39f5a1022100fe745667e444978c51fdba6981505f0a68619f0289e5ff2352acbd31b3d23d870141046c4ea0005563c20336d170e35ae2f168e890da34e63da7fff1cc8f2a54f60dc402b47574d6ce5c6c5d66db0845c7dabcb5d90d0d6ca9b703dc4d02f4501b6e44

The confirmed transaction has script:

4d480030450220539901ea7d6840eea8826c1f3d0d1fca7827e491deabcf17889e7a2e5a39f5a1022100fe745667e444978c51fdba6981505f0a68619f0289e5ff2352acbd31b3d23d87014d4100046c4ea0005563c20336d170e35ae2f168e890da34e63da7fff1cc8f2a54f60dc402b47574d6ce5c6c5d66db0845c7dabcb5d90d0d6ca9b703dc4d02f4501b6e44

There are a couple differences (highlighted in red). But what do they mean?

This script is the scriptSig, the signature of the transaction using the sender's private key. This signature proves the sender owns the bitcoins. However, the scriptSig isn't just a simple signature, but is actually a program written in Bitcoin's Script language. This program pushes the signature data onto the execution stack. The program from the unconfirmed script is interpreted as follows:

PUSHDATA 4848signature (DER) sequence30length45integer02length20X539901ea7d6840eea8826c1f3d0d1fca7827e491deabcf17889e7a2e5a39f5a1integer02length21Y 00fe745667e444978c51fdba6981505f0a68619f0289e5ff2352acbd31b3d23d87SIGHASH_ALL01PUSHDATA 4141public key type04X6c4ea0005563c20336d170e35ae2f168e890da34e63da7fff1cc8f2a54f60dc4Y 02b47574d6ce5c6c5d66db0845c7dabcb5d90d0d6ca9b703dc4d02f4501b6e44

The program from the confirmed script is interpreted as follows:

OP_PUSHDATA2 00484d 48 00signature (DER) sequence30length45integer02length20X539901ea7d6840eea8826c1f3d0d1fca7827e491deabcf17889e7a2e5a39f5a1integer02length21Y 00fe745667e444978c51fdba6981505f0a68619f0289e5ff2352acbd31b3d23d87SIGHASH_ALL01OP_PUSHDATA 00414d 41 00public key type04X6c4ea0005563c20336d170e35ae2f168e890da34e63da7fff1cc8f2a54f60dc4Y 02b47574d6ce5c6c5d66db0845c7dabcb5d90d0d6ca9b703dc4d02f4501b6e44

Note the highlighted differences. The original transaction has a byte 0x48, which says to push (hex) 48 bytes of data. The modified transaction has a OP_PUSHDATA2 (0x4d), which says the next two bytes (48 00) are the number of bytes to push. In other words, both transactions do exactly the same thing (push the signature), but the original indicates this with 48, while the modified transaction indicates this with 4d 48 00. (Pushing the public key has a similar modification.) Since both scripts do exactly the same thing, both transactions are equally valid. However, since the data has changed, the transactions have two different hashes.

Why does malleability matter?

Transaction Malleability has been discussed for years and treated as a minor inconvenience. Both transactions have exactly the same effect, moving bitcoins between the same addresses. Only one transaction will be confirmed by miners, and the other will be discarded, so nobody gets paid twice even though there are two transactions.

There are, however, three problems that have turned up recently due to malleability.

First, the major Mt.Gox exchange stated they would stop processing bitcoin withdrawals until the Bitcoin network approves and standardizes on a new non-malleable hash. Apparently they were using the hash to track transactions, and would re-send bitcoins if the transaction didn't appear to go through. This is obviously a problem if the transaction did go through, but with a different hash.

Second, some wallet software would use both transactions to compute the balance, which caused it to show the wrong value.

Finally, due to the way Bitcoin handles change, malleability could cause a second transaction to fail. This requires a bit more explanation.

Failures due to change and malleability

The Bitcoin protocol doesn't really move bitcoins from address to address. Instead, it takes bitcoins from a set of inputs, and sends them to a set of outputs. Each output is an address (actually a script, but let's ignore that for now). Each input is an output from a previous transaction, and each input must be entirely spent.

As a result, if you have 3 bitcoins, and you want to spend one of them, the other two bitcoins get returned to you as change, sent to an address you control. If you then want to spend some of the change, your second transaction references the previous transaction that generates the change, referencing it by the hash of the first transaction. This is where malleability becomes a problem - if the first transaction's hash changed, the second transaction is not valid and the transaction will fail. Note that the change will still go to your proper address, so you can spend it as long as you use the correct (modified) transaction hash, so you don't lose any bitcoins. You just have the inconvenience of having a transaction rejected, and you'll need to redo it with the right hash.

The change problem only happens because some wallet software takes a shortcut, letting you (attempt to) spend the change before the transaction has been confirmed. The reasoning is that since it's your change from your transaction, you should be able to trust yourself. But that breaks down with malleability.

Malleability has been known for a long time

Transaction malleability has been known since 2011. The exact OP_PUSHDATA2 malleability used above was described four months ago here. There are many other types of malleability, which are explained here. The script code can be modified in several ways while leaving its operation unchanged. The signature itself can be encoded slightly differently. And interestingly, due to the mathematics of elliptic curves the numeric value of the signature can be negated, yielding a second valid signature.

Conclusion

Hopefully this has helped to make malleability more understandable. If you want to know more details of the Bitcoin protocol, including signing and hashing, see my previous article Bitcoins the hard way.

Tech Notes: React, JSX, and CoffeeScript

Thu, 13 Feb 2014 09:34:04 -0800

Comments: "Tech Notes: React, JSX, and CoffeeScript"

URL: http://neugierig.org/software/blog/2014/02/react-jsx-coffeescript.html

React is neat. It's a reimagining of how your DOM relates to your app state that dramatically simplifies code without being an enormous framework*. It also breaks a rule that I've never cared for, where your templates are supposed to be separate from your code.

In React's world, each component (re-)renders its entire virtual DOM on any state change; React then manages updating the on-page DOM based on what changed. This means the rendering code only needs to model the static state of a component, rather than poking around in the DOM to transition between states.

To facilitate this rendering, React includes an optional library/tool called JSX that lets you inline HTML into your render function. The translation is relatively simple. This code:

render: function() {
 return <p><a href="foo">bar</a></p>;
}

becomes, post-translation, the not-keyboard-friendly but not hard to understand:

render: function() {
 return React.DOM.p(null, React.DOM.a({href:"foo"}, 'bar'));
}

In words, the tag becomes a function call, the attributes become an object passed as the first argument, and the contents of the tag are further parameters to the function.

The nice thing about interleaving code and DOM is that you can use all the normal JavaScript functionality in your template. Here's another example from the React docs with an embedded Array.map:

return (
 <ol>
 {this.results.map(function(result) {
 return <li key={result.id}>{result.text}</li>;
 })}
 </ol>
);

CoffeeScript instead of JSX

If you're using CoffeeScript, your source code isn't JavaScript to begin with. But turns out that CoffeeScript's flexible syntax makes it relatively painless to use the underlying API directly.

Start with shortening the DOM alias and writing more or less the same code as above. Also note that you don't need to explicitly return as the last statement in a function is implictly returned, and that the function literal syntax for argumentless function is just a bare ->:

R = React.DOM
# ...
render: ->
 R.p(null, R.a({href:'foo'}, 'bar'))

But you can do better. First, CoffeeScript knows to insert the curlies on an object literal because of the embedded colon.

 R.p(null, R.a(href:'foo', 'bar'))

And then you can remove the parens by splitting across lines. When providing args to a function, a comma+newline+indent continues the argument list. Much like Python, the visual layout follows the semantic nesting.

 R.p null,
 R.a href:'foo', 'bar'

In fact, beyond the first argument, the trailing commas are optional when you have newlines. Here's the same thing again with two links inside the <p>:

 R.p null,
 R.a href:'foo', 'bar' # note omitted comma here
 R.a href:'foo2', rel:'nofollow', 'second link'

CoffeeScript also makes every statement into an expression, which is a familiar feeling coming from functional programming. It means you can use statement-like keywords like if and for on the right hand side of an equals sign, or even within a block of code like the above.

Here's a translation of the (7-line) <ol> example from above.

R.ol null,
 for result in @results
 R.li key:result.id, result.text

There is one final feature of CoffeScript that I find myself using, which is an alternative syntax for object literals. For example, suppose in the above example the "key" attribute needs to be computed from some more complicated expression:

R.ol null,
 for result, index in @results
 resultKey = doSomeLookup(result, index)
 R.li key:resultKey, result.text

The simplification is that, within a curly-braced object literal, entries without a colon use the variable name as the key. The above could be equivalently written:

R.ol null,
 for result, index in @results
 key = doSomeLookup(result, index)
 R.li {key}, result.text

This is particularly useful when the attributes you want to set have meaningful names -- key is pretty vague, but if you construct an href and a className variable it's pretty clear where they are going to be used. These can be mixed with normal key-value pairs, too, like:

href = ...
className = ...
R.li {href, className, rel:'nofollow'}, ...

Putting it all together, here's a larger example, part of an implementation of an "inline edit" widget. To the user, this widget is some text with a "change" button to its right, where clicking on "change" swaps the line of text out for an edit field positioned in the same place, allowing the user to make a change to the value directly. (Like how it works in a spreadsheet.) The first branch of the if is the widget's initial state; the @edit function flips on the @state.editing flag.

render: ->
 if not @state.editing
 R.div null,
 @props.text
 ' ' # space between text and button
 R.span className:'link-button mini-button', onClick:@edit, 'change'
 else
 R.div style:{position:'relative'},
 R.input
 style:{position:'absolute', top:-16, left:-7}
 type:'text', ref:'text', defaultValue:@props.text
 onKeyUp:@onKey, onBlur:@finishEdit

To get a feel for these rules, you can just experiment and look at the generated JavaScript. Or you can go to coffeescript.org and click the "Try CoffeeScript" tab, where you can enter nonsense expressions there just to experiment with the syntax.

* Though I do wish it was smaller. I wish I could cut all the support for old IE bits and the event handling abstractions.

Why I'm Done with Social Media Buttons - Sam Solomon

Thu, 13 Feb 2014 10:30:06 -0800

Comments: "Why I'm Done with Social Media Buttons - Sam Solomon"

URL: http://solomon.io/why-im-done-with-social-media-buttons/

As designers, it is our job to question why elements should exist. Sometimes we get so caught up in new ideas about navigation and animations that we forget to go back and examine more common elements—like social media buttons.

They are unsightly relics of the web 2.0 era, yet they continue to find their way into new designs. Why does this happen?

I’ve never been fond of them, but for a long time they seemed necessary. Trading some visual appeal for social network traffic is a compromise many make. Plus, it is easy to hop on the bandwagon. If Mashable uses them, why shouldn’t I?

I followed that premise for a long time, but this is my stop. I’m hopping off. Those magical social buttons aren’t worth a damn anymore, and they won’t bring you traffic.

Anti-Social Media Buttons

Signal Tower is a podcast where I interview designers, developers and entrepreneurs. It is a great project. Recently, I posted an interview with Jennifer Dewalt, who taught herself to program by building 180 websites in 180 days.

It was an amazing feat, and not surprisingly the interview is among the most popular on the site. It has been shared more than a hundred times.

However, there is something interesting about those shares—not one of them occurred by clicking on a social media button. According to my in-page analytics, the buttons haven’t been touched in the month since the interview was been published.

Initially, I thought that it was some sort of bug. Certainly someone clicked them, right? I decided to take a look at older posts. In all cases they had been used a negligible amount.

I’m not the only one who has seen lackluster performance out of sharing buttons. My friend Samuel Hulick, who runs useronboard.com, actually saw an increase in sharing once the buttons were removed.

Samuel’s experience may be an anomaly, but there is definitely a trend here. Either way, I’ve decided to remove the buttons from Signal Tower, and will not add them to this site.

The conversion-minded designers will say, “Why didn’t you test the buttons? You should change the color, move them to the top, or try fixed floating buttons.” And they are right. There is a good amount to be gained by A/B testing them, but there is another reason I’m getting rid of them.

Sharing, a POOR USER EXPERIENCE

I never paid much attention to sharing buttons. I always thought they were a poor user experience.

I’m a minimalist. When it comes to web design, I value white space, typography and function. People come to your site for the writing, which is why a focus on these elements will almost always result in a good user experience. Adding non-essential elements to a page reduces signal and creates noise.

If you are on the internet for the cat pictures, this does not apply.

Beyond the added noise there are other odd experiences using social buttons. Clicking on a button opens a popup window filled with oddly formatted text, marketing speak, and a ton of other stuff I don’t want to send out to friends and followers. All of the above degrades credibility.

Popups are awkward. Even moreso on phones and tablets, which brings up another point. Every phone and tablet has a browser with built-in sharing, and mobile devices are gaining marketshare.

WHAT MAJOR WEBSITES DON’T USE Sharing BUTTONS?

The one that immediately comes to mind is Information Architects. A design group known for devout minimalism. Oliver Reichenstein, IA’s founder, wrote a scathing article on social media buttons called, Sweep the Sleaze.

Reichenstein is right, “If readers are too lazy to copy and paste the URL, and write a few words about your content, then it is not because you lack these magical buttons.”

There are other popular sites that have forgone social buttons. 37signals (now Basecamp), the company that ushered in the era of SaaS businesses and invented Ruby on Rails, have a blog called Signal vs. Noise. You won’t find any on their blog.

Smashing Magazine, one of the world’s leading design publications has forgone them. As a result of removing the Like Button, they discovered that more readers shared articles on their timeline.

Closing Thoughts

There are some redeeming values to sharing buttons. A post with buttons that show thousands of shares definitely boosts credibility. It signals that there is quality content, and others are vouching for it. Simply having the buttons on the page also subtly reminds visitors that they should share the content.

On the other hand, what if nobody has shared your post? There are fewer things more sorry than a post that has a dozen buttons with zeros next to them.

The evidence against social media buttons is stacking up. The novelty and utility of social media buttons have worn off. It is time to reconsider how much value they add.

If people really love your content, they’ll share it.

Can’t We All be Reasonable and Speak English? « Blog – Stack Exchange

Thu, 13 Feb 2014 10:19:04 -0800

Comments: "Can’t We All be Reasonable and Speak English? « Blog – Stack Exchange"

URL: http://blog.stackoverflow.com/2014/02/cant-we-all-be-reasonable-and-speak-english/

Two weeks ago, we announced the public launch of Stack Overflow in Portuguese, our first-ever non-English Stack Overflow community. Which raises one very obvious question:

Have we lost our minds?

Wasn’t the whole point of Stack Overflow to aggregate as much developer knowledge as possible in one place? To get all the potential solutions together, and provide one canonical set of answers?

We are aware that, “Let’s all try speaking speaking different languages!” hasn’t always worked out for the best.

Yup. When we set out to “collectively increase the sum total of good programming knowledge in the world,” a big part of the plan was de-fragmenting information previously spread across myriad books, sites, and your brains. It’s why we mark things as duplicates – we want all the precious gems of knowledge stored in the same cave of wonders.

So know this: we are at least as worried about fragmentation as you are. And we have a plan:

Eventually, all of you are going to have to learn Portuguese.

Okay, not really. But, given that one of our core goals was knowledge aggregation, it does seem just a little bit crazypants to start launching sites in new languages, assuming that one very important fact is true:

Assumption: All of the serious developers in the world are highly proficient in English.

Which… actually sounds plausible. But it’s wrong.

Not every developer in the world speaks English. Just reading the comments from our announcement, you’ll see multiple readers sharing that they or their colleagues (and one dad) couldn’t participate on SO due to language constraints. But data beat anecdotes. We don’t have recent numbers for Brazil and Portugal, but we do for China, and they illustrate the same point: So, if the data tell us that we’re getting roughly 80% less activity from Asia than we should in the absence of language constraints, why does it feel so obvious that all serious programmers speak English? This may help: Quick – name any famous developer who doesn’t write well in English. I couldn’t. I can name over a dozen famous English-speaking coders. But even if you frequent all the hacker sites and conferences, how many devs have you met who aren’t solid in English? Roughly none, right? There’s just one problem. Try this: Without Googling, name any famous developer from Japan. Or China. Or Russia. Again, I couldn’t. Well, I came up with Shigeru Miyamoto. But he’s apparently a designer. I couldn’t name even one. Not like I can name Carmack or Stallman, or Hopper, or even “DHH.” (Does DHH have an actual name? I personally imagine him as a very handsome, talented, fast-driving set of initials. But I digress.) Is it plausible that there aren’t any devs good enough to be famous from those countries? Nope. Here’s what’s happening: It’s easy to assume that there aren’t any devs who can’t speak English because I never see any. But I never see any because I’m hanging around places where devs go to talk to each other in English. The startling truth is this: On the internet, If you don’t speak English, you’re completely invisible to me. I also assumed that since developers have to learn English-like syntax, they must speak English. Which is a bit like assuming that because I can order Uni, Hamachi, and Aji by their Japanese names, I could probably toss back some sake with Morimoto and discuss knife techniques in Japanese. Even when programming languages use words like “if” or “function,” they’re just terms to memorize, and don’t always even mean the same thing in English that they do in programming. It’s almost impossible to feel like part of a community if you’re not highly proficient in the language. Even non-native speakers who are fluent enough to read posts in their second or third languages often aren’t comfortable enough to write in them.I imagine myself at a professional meetup where everyone is speaking French (which I studied through college). How many jokes would I tell? How many would I even understand? Sure, I can function, and understand all the words, but I don’t feel like I belong to the group. Don’t get me wrong – some of our best users aren’t native English speakers, but they’re in that rare group who have achieved a far higher mastery of a language than their peers. When I hear, “Well, I didn’t need a site like this – English is my third language, and I’m in the top 1% on Stack Overflow!” I think: “Yes, that makes sense. You are insanely good at two difficult, language-based things. Most people will find both of them to be a lot more challenging than you did.” The truth is, by requiring fluency in English, we’re shutting out of a lot of developers who may know enough English to read it but not enough to feel comfortable participating. Requiring that all aspiring devs “just go learn English” first isn’t who we want to be. No child should be denied their chance to revolutionize tomorrow’s input technologies. Even if I believed that every programmer must eventually master English, it still wouldn’t make any sense to make them do it first. I believe that everyone – everyone – who can really fall in love with programming should get a chance to. So pre-filtering for the ones willing to learn a foreign freaking language before they first sit down with a code editor to see if it lights some spark in them just feels wrong.Think of the children. The children!! Okay, last quiz, just for the native English speakers: How old were you when you first realized you could type things on a keyboard and control machines? Great. Now, at that age, were you proficient enough in another language to have learned to code without any English? When I tell someone I work at Stack Exchange, my absolute favorite response is: “I basically learned to code from posts I found on Stack Overflow” We want that for every young programmer. Not just the ones lucky enough to be born somewhere that English gets taught in grammar school.

Okay, that all makes some sense. But why Portuguese?

To be clear, we still don’t think there needs to be a Stack Overflow in every language. We do want as much centralization as possible, and we know that devs who have mastered English will mostly keep going to the English site, since it has the most critical mass. Just like we want them to. So, you won’t need to learn new languages to find good answers – we expect almost every question asked on the Portuguese site to also be asked (and answered) on the English site.

We’re really only considering launching sites in languages that:

Have large, strong communities of high-talent developers, where
A meaningful percent of them aren’t comfortable enough to participate in an English-only community

That probably limits the list of potential candidates to Mandarin, Japanese, Portuguese, Russian, Turkish, and Spanish. From there, Portuguese was a no-brainer. The developer community in Brazil is awesome, and growing fast. And we wanted to start with a language with a similar alphabet, to minimize the localization work.

And it’s worth a shot. We’ve learned that it’s easier to just watch the future than to try to predict it. So we’re big on just trying stuff out (assuming it can’t break our other stuff). And we’re huge on getting stuff crazy-wrong, refusing to admit it, and instead doubling down on our wrong-minded idea, while nodding crazily er… admitting we made a mistake, and reversing course. So, given the number of user requests, we figured, “why not give it a it a try?” We’re committed to supporting one or two languages and seeing how they develop before we push any further.

And so far, it’s an incredible success. Despite an audience limited to portuguese-speaking devs, the site’s activity in its first week was higher than all but 4 out of 120 sites we’ve launched to date, including the original trilogy.

More importantly, people who couldn’t ask questions are asking them, and getting great answers. When in doubt, we want to err on the side of helping more people. If just one little girl in Brazil sticks with programming because an answer on this site helped her finish her first project, well… that’s not good enough! I want to help thousands of them. And the boys, too.

Still, it’s a good start.

Ron Paul Launches Snowden Clemency Petition - Hit & Run : Reason.com

Thu, 13 Feb 2014 11:29:18 -0800

Comments: "Ron Paul Launches Snowden Clemency Petition - Hit & Run : Reason.com"

URL: http://reason.com/blog/2014/02/13/ron-paul-launches-snowden-clemency-peti

Credit: Gage Skidmore/wikimedia

Today, it was reported that former Rep. Ron Paul (R-Texas) launched a petition calling for NSA whistle-blower Edward Snowden to be granted clemency.

On the page on the Ron Paul Channel’s website where visitors can sign the petition, the former congressman says,

Edward Snowden sacrificed his livelihood, citizenship, and freedom by exposing the disturbing scope of the NSA’s worldwide spying program. Thanks to one man’s courageous actions, Americans know about the truly egregious ways their government is spying on them.

The news of Paul’s petition comes on the same day it was reported that the European Parliament had voted against calling for Snowden to be granted asylum.

The New York Times and The Guardian both urged U.S. officials to grant Snowden clemency last month.

According to the Ron Paul Channel, Snowden’s temporary asylum in Russia will expire at the end of July.

Yesterday, Intelligence Squared hosted a debate on the motion “Snowden was Justified.” Speaking for the motion were legal adviser to Edward Snowden and ACLU attorney Ben Wizner and Pentagon Papers whistle-blower Daniel Ellsberg. Speaking against the motion were former CIA Director James Woolsey and former federal prosecutor and contributing editor to National Review Andrew McCarthy.

Wizner said Snowden was justified “because he provided to journalists and through them to us information that we had a right to know and that we had a need to know. The government had not just concealed this information, it had lied to us about it.”

Woolsey claimed that Snowden had released information to “Hezbollah, Al Qaeda, Hamas, Pyongyang, Tehran, and so on.” Thankfully, Ellsberg pointed out that Snowden released information to journalists who have since reported on the documents.

Before the debate began 29 percent were for the motion, 29 percent were against the motion, and 43 percent were undecided. After the debate, 54 percent were for the motion, 35 percent were against the motion, and 11 percent were undecided.

Watch the debate below:

Good Samaritan Backfire — Medium

Thu, 13 Feb 2014 12:29:44 -0800

Comments: "Good Samaritan Backfire — Medium"

URL: https://medium.com/p/9f53ef6a1c10/

Solitary Confinement — Safety Cell

I was led into a corner.

“First we have to get you ready,” one of the deputies said. His arm undid the button of my pants, which at first I thought was a cruel joke, and then he yanked them down to my ankles.

They pushed me forward against the wall. I stumbled in my handcuffs and pant shackles.

“Step out of your pants,” they ordered. And as soon as I did: “Step out of your socks!”

Naked from the waist down, someone said, “Take off your shirt.” It was topologically impossible, given the cuffs. One of the deputies said, “I’ll do it.” I was uncuffed, my shirt was stripped with force, getting caught on my neck, tugging my head backwards, then up, then off.

The night shift deputies were cruel. They responded to questions in the tone of schoolyard bullies—tauntingly. They giggled as they slammed the door behind me. “You’ll see the doctor alright.”

On the floor lay a straight jacket made from the material used to pad furniture when it is being moved, and a second piece of the same fabric that I later used to cover the dirty floor in an attempt to sleep.

There were no knobs or protrusions in the room, just soft corners. The toilet was a hole in the ground, no toilet paper. The hole dropped down a few feet where it was intersected by a grate of prison bars. The flushing happened automatically, periodically, though I never felt the urge. Even one’s feces left prison upon evacuation, presumably to leave the subject without anything to play with.

I say this, because while the room was dirty, it was not as dirty as the next two cells I experienced the following day, which were smeared with feces and peanut butter. Approximately every 6 hours, a pushcart made its way around the prison with regulation peanut butter sandwiches. Only a fraction were consumed. Many were used for wall decoration or splattered against the ceiling.

I couldn’t bear to eat, so I took my rations home as a souvenir. Aside from the milk, they still seemed edible a month later. This is their strength.

Trapped in a rendition of One Flew Over the Cuckoo’s Nest

While the metal door was too thick for me to be heard if I did not scream, I could hear the muted screams of others across the jail. The din was anything but soothing.

When I asked for water, I was given enough (a couple Dixie cups’ worth) to barely keep my throat lubricated.

I was cold. The two pieces of fabric were not enough to spread on the filthy ground and also cover my naked body. I tried to sleep but it proved fruitless. Every 15 minutes, the metal peephole was creaked open, and I was expected to react, presumably to confirm that I was still alive. This was noted on a clipboard hanging beside the door.

Eventually, I found it most comfortable to stand by the cell door with the coarse fabric draped over my body. I looked out through a narrow slit of Plexiglas and tried to call attention from passers’ by. “Sir, Ma’am, could you please tell me… how long should I expect to be in here?”

A streak of being ignored was broken by a couple disheartening responses. “Usually we put people in there for 24 hours.”

Now I really felt like I was going crazy. Those weren’t the reassuring answers my inner optimist had hoped for. When I had told the arresting officers that I accepted my lot, this wasn’t the lot I was referring to. I didn’t expect a medal for fulfilling my civic duty, but I still felt like I had some fleeting right to something other than this. I banged on the metal door repeatedly until Deputy Terry showed up.

“Why am I in here?”

“You are crazy. You are a lunatic,” he pronounced.

“Do you know how I got here?”

“Doesn’t matter.”

“This place—being in here—will make me crazy,” I pleaded.

“Good. That’s what you are and where you belong.” He spiraled his index finger by his muscular temple.

I tried to respond as he started walking way.

“Sir, might you consider for a moment that I am having a sane response to the conditions I’m being subjected? I was arrested by the very police I called to the scene of a medical emergency less than a block from my house, while heading home for the night.”

He stared at me bewildered, and never came near again.

Silk Road 2 Hacked, All Bitcoins Stolen – $2.7 Miliion | Deep Dot Web

Thu, 13 Feb 2014 12:29:44 -0800

Comments: "Silk Road 2 Hacked, All Bitcoins Stolen – $2.7 Miliion | Deep Dot Web"

URL: http://www.deepdotweb.com/2014/02/13/silk-road-2-hacked-bitcoins-stolen-unknown-amount/

Update 2: As the time passes there are more and more suspicions that this was in fact a SCAM by the Silk Road staff – and not a hack, we will post more details about it once, and if we get the full picture.

Update: The amount of BTC that was stolen was calculated by Nicholas Weaver @NCWeaver – Computer Security Researcher, to be around: 4474.266369160003BTC that are with the value of about $2.7 Million.

It was just announced in a post by Defcon the Silk Road administrator (this post will be updated as soon as we get more info) -
Yes, what seemed to be an imaginary situation until not long ago, just became true, the silk road2 – the site who counted to be the security fortress of the deep web just has been hacked with its bitcoin stolen. as he announced on the sites forums, we pasted his post here:

Link to the original thread on Silk Road 2 Forums: http://silkroad5v7dywlc.onion/index.php?topic=25091.msg491029#msg491029

=====Start Quote====

I am sweating as I write this.

Christmas brought grave news. I cannot adequately express how deeply honored I was by your unconditional support of my staff.

I do not expect the same reaction to today’s revelations. This movement is built on integrity, and I feel obligated to be forthright with you.

I held myself to a high standard as your leader, yet now I must utter words all too familiar to this scarred community:

We have been hacked.

Nobody is in danger, no information has been leaked, and server access was never obtained by the attacker.

Our initial investigations indicate that a vendor exploited a recently discovered vulnerability in the Bitcoin protocol known as “transaction malleability” to repeatedly withdraw coins from our system until it was completely empty.

Despite our hardening and pentesting procedures, this attack vector was outside of penetration testing scope due to being rooted in the Bitcoin protocol itself.

This attack hit us at the worst possible time. We were planning on re-launching the new auto-finalize and Dispute Center this past weekend, and our projections of order finalization volume indicated that we would need the community’s full balance in hot storage.

In retrospect this was incredibly foolish, and I take full responsibility for this decision.

I have failed you as a leader, and am completely devastated by today’s discoveries. I should have taken MtGox and Bitstamp’s lead and disabled withdrawals as soon as the malleability issue was reported. I was slow to respond and too skeptical of the possible issue at hand. It is a crushing blow. I cannot find the words to express how deeply I want this movement to be safe from the very threats I just watched materialize during my watch.

I’ve included transaction logs at the bottom of this message. Review the vendor’s dishonest actions and use whatever means you deem necessary to bring this person to justice. More details will emerge as we continue to investigate.

Given the right flavor of influence from our community, we can only hope that he will decide to return the coins with integrity as opposed to hiding like a coward.

It takes the integrity of all of us to push this movement forward. Whoever you are, you still have a chance to act in the interest of helping this community. Keep a percentage, return the rest. Don’t walk away with your fellow freedom fighters’ coins. DPR2 returned the cold storage. I didn’t run with the gold. But two people alone cannot move us forward. It takes an entire community committing to integrity – and though this crushing blow will not stop us, it sure is a testament to how greedy some bastards truly are.

Being a part of this movement might be the most defining thing you do with your entire life.

Don’t trade that for greed, comrades.

I will fight here by your side, even the greedy bastards amongst us.

This community has suffered great financial loss over and over again, and I am devastated that it has happened again under my watch.

Hindsight is already suggesting dozens of ways this could have been prevented, but we must march onward.

The only way to reverse a community’s greed is through generosity. Our true character is revealed during trying times.

If this financial hardship places you at risk of physical harm, contact me directly and I will do my best to help you with my remaining personal funds.

Now what.

Never again store your escrow bitcoins on a server.

Silk Road will never again be a centralized escrow storage.

This week has shown the collateral damage we can cause by being a huge target and failing in just one unforeseen area.

I am now fully convinced that no hosted escrow service is safe.

If I cannot trust myself to keep a hosted escrow solution safe, I cannot trust anyone.

Multi-signature transactions are the only way this community will be protected long-term.

I am aggressively tasking our devs on building out multi-sig support for commonly-used bitcoin clients. Expect a generous bounty if you have the skill to implement this.

Until then.

1. We will never again allow ourselves to be a single point of failure. We will never again host your Escrow wallets.

2. Vendor registration is closed while we regroup.

3. All listings on Silk Road are now No-Escrow (Finalize-Early) for 1-2 months while we implement multi-signature transactions and lobby for mainstream Bitcoin client multi-sig support.

4. All unshipped orders have been cancelled.

5. Vendors may link to other marketplaces on a trail basis until we launch multi-sig, then we will re-evaluate based on community input. We do not want to be a centralized point of failure, but we also do not want to lead our buyers into dangerous waters.

6. From this point forward DO NOT trust markets with centralized escrow. Use multi-signature transactions whenever possible, with trusted third parties as escrow providers.

Everything will be offline for 24-48 hours to minimize variables as we continue to investigate. The evidence we have below will be expanded based on our findings.

- ——————

No marketplace is perfect. Expect any centralized market to fail at some point. This is precisely why we must unite in the decision to decentralize.

We are relieved that our security procedures protected user identities, and that no servers were compromised. This was not a worst-case scenario: nobody will be getting arrested from this. Financial loss is terrible, but will not put all of us behind bars.

The details we have on the hacker are below. Stop at nothing to bring this person to your own definition of justice.

Humbled and furious,

Defcon

- ——————

# Attacker Intel as of 2014-02-13 18:00:00 UTC

We normally do not doxx anyone, and hold user information sacred. But this is an extreme situation affecting our entire community, and all three users who have exploited this vulnerability are very much at risk until they approach us directly to assist with any information.

Do not reveal any details of the attack. This will jeopardize your reward. Contact us directly.

If anyone has purchased or sold to these usernames, expect generous bounties for any information you can contribute which leads to identification.

# Attacker 1: (Responsible for 95% of theft)
Suspected French, responsible for vast majority of the thefts. Used the following six vendor accounts to order from each other, to find and exploit the vulnerability aggressively.

## Usernames used:
narco93
ketama
riccola
germancoke
napolicoke
smokinglife

Transactions listed at bottom of this file. Finding Attacker 1 is top priority.

# Attacker 2: (Responsible for ~2.5% of theft, using same methods towards end of attack lifecycle, likely knows Attacker 1)
LethalWeapon – Australia – “stumbled upon” large amount of BTC

# Attacker 3: (Responsible for ~2.5% of theft, using same methods towards end of attack lifecycle, likely knows Attacker 1)
mrkermit – Australia

# Theft Withdrawal Transactions and historical withdrawals by Attacker 1
address,txid_cleaned
{Here some big list of withdrawal addresses with the stolen bitcoins}

=====End Quote====

Aside from the endless marketplaces being hacked every day now, this is the most shocking event we have encountered – as Silk Road being the largest DarkNet market nowadays was probably holding the largest sum of money of them all – it is not yet clear how many Bitcoins were stolen exactly, but its almost certain that this is about to become the largest theft in the Deep Web history – bigger than the Sheep Marketplace Scam that had amount equal at the time to $40 million in bitcoins stolen by its admins.

This case only serves as ANOTHER, Very Painful lesson about – why on-site escrows are bad, and should not be used! only direct transaction or mulsig escrow like the one offered at themarketplace.i2p are the safe way to conduct business on these sites.

Is this the end of the centralized marketplaces?

We sure hope so! as we posted here again and again, they are not safe, and will always end up being hacked or having the money stolen by their admins.

So who were the hackers?

Few hours before the announcement we at DeepDotWeb received a mail saying: “SilkRoad hacked, 150 BTC stolen, you heard it first from me” this was sent to us by a reddit user who claimed since yesterday he was going to hack SR and steal the sites money – we are trying to verify if this amount matches the amounts that were stolen by the “smaller” hackers that Defcon reported in his post, the others remain unknown.

The Silk Road moderators ranged from pleading or threatening the hackers:

To a complete shock:

To an Apology:

The users reaction was not much different obviously and ranged between shocked / angry / desperate or accusing the sites admins to the thief’s themselves:

IS ANYONE ELSE BUYINGGGG THIS? !!! WE ARE FIXING ESCROW WE ARE FIXING VENDOR REFUNDS? WE ARE DOING ALL WE CAN THIS SHEEP !!!! STYLE FUCKING BY OUR TRUSTED SR GUYS , ITS FUCKING PLAIN AND SIMPLE ESCROW SYSTEM WAS A SCAM SO EVERY COCKSUCKER WHO DIDNT FINALZE THE COINS STAYED IN THE BANK AND OPPS WE HAVE BEEN HACKED !!! WE ARE FIXING THE VENNDOR REFUND ? YEAH RIGHT RIGHT ANOTHER PERFECT SCAM, MORE COIN IN THE BANK AND AT THE RIGHT TIME AGAIN OOPSS WE HAVE BEEN HACKED \ DEFCON GO FUCK YOUR SELF , U GUYS HAVE NOT DOMNE NOTHING ABOUT THE ESCROW SYSTEM , U HAVE DONE NOTHIGN ABOUT VENDOR REFUND , ALL U GUYS DID IS LET THE FUCKING BANK BUILD UP AND SORRY GUYS WE HAVE BEEN HACKED EVERY DOG GETS THERE DAY AND I CANT WAIT TILL I SEE ONE OF U FALL

Some even tried to help in some way.

For us – the big question is “how much”? , we will keep following up on this and updating this post as we get new information – for now, you can check out other site on this list.

Share and Enjoy

Startup Sales Negotiations 101 - How to Respond to Discount Inquiries

Thu, 13 Feb 2014 12:59:44 -0800

Comments: "Startup Sales Negotiations 101 - How to Respond to Discount Inquiries"

URL: http://blog.close.io/startup-negotiation-101-how-to-respond-to-discount-inquiries

People will sometimes reach out and ask for a discount on your product before they took the time to sign up for a trial and use it at all. What do you do when that happens?

Instead of debating if you should or shouldn't offer them a discount right away, you need to refocus their energy on what really matters: your product!

Let's explore the 3 core reasons why you never want to negotiate pricing before someone had a chance to trial your product and determine that it's a good fit.

#1: You're starting the relationship on the wrong foot

People who ask you to lower your prices before having invested any time using your product are usually trouble.

This can often lead to winning a new customer that is going to expect you to give 24/7 premium phone support, prioritize features based on their needs all while trying to pay you pennies on the dollar. If you start the relationship by giving them everything they ask for, don't be surprised if they keep asking for more in an unreasonable fashion. This is ultimately unsustainable and unhealthy for both sides.

#2: They're buying for the wrong reason

At this point they can't tell if your product is a good fit for them since they never used it. Your first priority should always be to help people explore and discover that your product is really solving their problem before negotiating what the final pricing should be.

Discounting your product upfront might help you close some deals faster but will often lead to these customers ultimately discovering that they should have never bought in the first place. Always be wary of prospects that don't want to do their homework upfront. Nothing sucks more than a new customer that cancels immediately after having created a ton of support and onboarding cost.

#3: You're negotiating on price vs. value

The problem with people trying to negotiate pricing before testing your product is that you are forced to negotiate on price rather than value.

They didn't have a chance to build up any desire to buy and discover the massive value your product could deliver to them. All of the sudden your product turns into a commodity and your only differentiation is offering them the lowest price possible.

#4: You're negotiating without leverage

The more time people invest in your product the more "invested" they become and naturally the harder it is for them to "throw away" the time they put into exploring your product and making it part of their daily workflow.

You always want to postpone the most difficult/complex parts of the sales negotiation till the end of the sales cycle. That way you ensure the right amount of momentum as you move forward in the sales process and avoid too much upfront friction.

Here is what your response should be when someone asks for a discount without having tried your product:

"Thanks for inquiring about pricing options! Why don't you sign up for a trial and give the product a go? If you find out that it's a great fit I'll take care of you and make sure you get a price that makes you happy. Sound fair enough?"

This works every time. The reply you usually get will be:

"Great! Just signed up and giving the product a go. Thanks!"

What's the result you should expect?

9 out of 10 times the people that turn out to be a bad fit will self select during a trial and just leave. The prospects that are a good fit will love your product so much that they will not negotiate hard for a discount since they now really understand its value.

Even if they do it's fine to give great customers a good price because you know they are buying for all the right reasons and will probably stay with you for a long time.

We've done this thousands of times and it always works. I hope this startup sales negotiation tactic serves your business as much as it has ours :)

P.S. The same tactic doesn't just work for people that are "unreasonable" and potentially a bad fit. Even when uber nice prospects are worrying about pricing upfront we tend to ask them to first discover if the product is even a good fit for their needs. See below a recent twitter exchange as a good example for this:

OCaml: what you gain - Thomas Leonard's blog

Thu, 13 Feb 2014 14:30:11 -0800

Comments: "OCaml: what you gain - Thomas Leonard's blog"

URL: http://roscidus.com/blog/blog/2014/02/13/ocaml-what-you-gain/

Way back in June, in Replacing Python: second round, I wrote:

The big surprise for me in these tests was how little you lose going from Python to OCaml.

Of course, I was mainly focused on making sure the things I needed were still available. With the port now complete (0install 2.6 has been released, and contains no Python code), here’s a summary of the main things you gain.

Table of Contents

This post also appeared on Hacker News and Reddit, where there are more comments.

( This post is part of a series in which I converted 0install from Python to OCaml, learning OCaml in the process. The full code is at GitHub/0install. )

A note on bias

I started these blog posts unemployed (taking a career break), with no particular connection to any of the languages, and motivated to make a good choice since I’d be using it a lot. I wasn’t biased towards OCaml; it wasn’t even on my list of candidates until a complete stranger suggested it on the mailing list. But I must now disclose that, since my last blog post, I’m now getting paid for writing OCaml.

Functional programming

Some people commented it was good to see more projects moving to functional programming. So, what’s it like doing functional programming after Python? To be honest, not much has changed. According to OCaml’s What is functional programming?, “In a functional language, functions are first-class citizens” and “The fact is that Perl is actually quite a good functional language”.

So, if you’ve ever used Python’s (built-in) map, reduce, filter or apply functions, ever written or used a decorator or ever passed a function as an argument to another function, you’re already doing functional programming as far as OCaml is concerned. By contrast, “pure functional programming” (as in Haskell) would be a major change.

OCaml does make partially applying functions easier, which is sometimes convenient, and it supports tail recursion. Tail recursion allows you to write loops in a functional style (without needing break, continue or mutable state). That can make it easier to reason about loops, but I couldn’t find any examples in 0install where this style was clearly better than a plain Python loop.

Type-checking

I’ve used statically-typed languages before (I used to program in Java for my day job). That can catch many errors that Python would miss, but OCaml’s type system is far more useful than Java’s. Here’s an example, where we want to display an icon for some program in the GUI:

1 2 get_icon program |> widget#set_icon (* Error! *)

Error: This expression has type Icon.t -> unit
 but an expression was expected of type Icon.t option -> 'a

Oops. The program might not have an icon (icons are optional). We’ll need to use a default one in that case:

1 2 3 get_icon program |> default generic_program_icon |> widget#set_icon

Downloading some data:

1 2 3 4 match download url with | `success data -> process data | `network_failure msg -> show_error_dialog msg (* Error! *)

Warning 8: this pattern-matching is not exhaustive.
Here is an example of a value that is not matched:
`aborted_by_user

Oops. The user might click the “Cancel” button - we need to handle that too:

1 2 3 4 match download url with | `success data -> process data | `network_failure msg -> show_error_dialog msg | `aborted_by_user -> abort ()

When registering an extra feed to an interface we want to download it first to check it exists:

1 2 3 let add_feed iface (feed_url:feed_url) = download_feed feed_url; (* Error! *) register_feed iface feed_url

Error: This expression has type feed_url
 but an expression was expected of type [< `remote_feed of url ]
 The second variant type does not allow tag(s) `local_feed

Oops. The user might specify a local file too:

1 2 3 4 5 let add_feed iface (feed_url:feed_url) = begin match feed_url with | `remote_feed _ as feed_url -> download_feed feed_url | `local_feed path -> check_file_exists path end; register_feed iface feed_url

Java makes you do all the work for static type checking, but manages to miss many of the benefits. No matter how much care you take with your Java types, there’s always a good chance you’re going to crash with a NullPointerException. By requiring correct handling of None (null) and ensuring pattern matching is exhaustive, OCaml’s type checking is far more useful. As with Haskell, when a piece of OCaml code compiles successfully, there’s a very good chance it will work first time.

And, of course, static checking makes refactoring much easier than in Python. For example, if you remove or rename something, the compiler will always find every place you need to update.

Data type definitions

OCaml makes it really easy to define new data types as you need them. The types are always easy to see, and you know that OCaml will enforce them (unlike comments in Python, which may be incorrect). Here’s a record type for the configuration settings for an interface (an optional stability level and a list of extra feeds):

1 2 3 4 type interface_config = { stability_policy : stability_level option; extra_feeds : Feed.feed_import list; }

And here’s a variant (enum / tagged union / sum) type for the result of a download:

1 2 3 4 type download_result = [ `aborted_by_user | `network_failure of string | `success of filepath ]

Polymorphic variants

The OCaml labels tutorial describes polymorphic variants as a way to use the same name (e.g. Open) for different things (e.g. opening a door vs opening a lock) and says:

“Because of the reduction in type safety, it is recommended that you don’t use these in your code”.

This is quite misleading (and I was quickly corrected when I repeated it). Their real purpose is to support subsets and supersets, which are useful all over the place. Some examples:

The “0install” command-line parser accepts a large number of options. The “0install run” subcommand accepts a subset of these. That subset can be further subdivided into common options (present in all commands, such as --verbose), options common to selection commands (e.g. --before=VERSION) and those specific to “0install run” (e.g. --wrapper=COMMAND).
The GUI code that handles dialog responses (OK, Cancel, etc) must handle the union of all the action button responses it added and the always-present window close icon.
The download code only handles the subset of feed URLs that represent remote resources.
Users can only register local and remote feeds to an interface, not distribution-provided virtual feeds.
Cached feeds contain only remote implementations, local feeds contain local and remote implementations, and distribution feeds contain only distribution implementations. All three types get combined together and passed to the solver.

Here’s an example, showing the run command dividing its options into sub-groups, with the compiler checking that every option will be handled in all cases:

1 2 3 4 5 6 7 let select_opts = ref [] in Support.Argparse.iter_options flags (function | #common_option as o -> Common_options.process_common_option options o | #select_option | `Refresh as o -> select_opts := o :: !select_opts | `Wrapper w -> run_opts.wrapper <- Some w | `MainExecutable m -> run_opts.main <- Some m );

Without polymorphic variants, OCaml’s exhaustive matching requirements mean you’d have to provide code to handle cases that (you think) can’t happen. That’s tedious and your program will crash if you get it wrong. Polymorphic variants mean you can prove to the compiler that only the correct subset needs to be handled at each point in the code. This is fantastic, and I can’t think of any other major language that does this (though I’m sure people will suggest some in the comments).

Immutability

In OCaml, all variables and record fields are immutable by default. This is far saner than Java (where the default is mutable and you must use final everywhere to override it). Immutable is a better default because:

Typically, you want most things to be immutable (in any language).
If you forget to mark something as mutable, the compiler will quickly let you know, whereas forgetting to mark something as immutable would be missed.

With mutable structures, you are always worrying about whether one piece of code will mutate a structure that another is relying on. For example, I originally made the XML element type mutable, but I found I was writing comments like this:

1 2 (** Note: this returns the actual internal XML; don't modify it *) val as_xml : selections -> element

After removing the mutable annotations from the element declaration, the compiler showed me each piece of code I needed to modify to make it work again. Then, I was able to remove those notes.

[ The main difficulty in this conversion was handling XML namespace prefixes. Previously, each element had a reference to its owning document, which held a shared (mutable) pool of prefix bindings. Now, each namespaced item holds its preferred prefix, and the output code builds up a set of bindings before writing out the tree. ]

There is one case where Python and Java do better than OCaml: OCaml strings are mutable! The convention is to treat them as immutable, though.

Abstraction

OCaml makes it very easy to hide a module’s implementation details from its users using abstract types.

I gave one example in the bugs post, where hiding the fact that a sorted tree is really the same type as an unsorted one prevents bugs due to mixing them up.

Here’s another example. In the Python code, we would parse a selections XML document into a Selections object, like this:

1 2 3 class Selections(object): def __init__(self, root_element): ...

I found all this parsing and serialising complicated things and so in the OCaml rewrite I decided to use the plain XML element type everywhere. That did simplify things, but it also removed some safety and clarity from the APIs. Consider the Selections.create function (which now does nothing unless the document is in an old format and needs to be upgraded):

1 2 3 4 let create root = ZI.check_tag "selections" root; if is_latest_format root then root else convert_to_latest root

It’s nice and simple, but it just returns an element. It would be easy to pass some other XML element to a function that only works on selection documents (or to pass a document that’s still in the old format). We can solve this simply by declaring an abstract type for selection documents in the interface file (selections.mli):

1 2 type t val create : element -> t

(note: it’s an OCaml convention for a module’s main type to be called t; other modules will refer to this type as Selections.t)

I think this gets the best of both worlds. Internally, a selections object is just the XML root element, which is simple and efficient, but code using it can’t mix up the types. And, of course, we can change the internal type later if needed without breaking anything.

This isn’t a particularly novel idea (you can do something similar in C). However, Python and Java would require you to write a wrapper object around the object you want to hide, and Python makes it easy for users of the API to access the internal representation even then. If you’re writing a library, OCaml (like C) makes it clear when you’re changing the module’s interface vs merely changing its implementation.

( There is another interesting feature, which I haven’t used yet: you can use the “private” modifier to say that users of the module can see the structure of the type but can’t create their own instances of it. For example, saying type t = private element would allow users to cast a selections value to an XML element, but not to treat any old XML as a selections value. )

I did experience one case where abstraction didn’t work as intended. In the SAT solver, I declared the type of a literal abstractly as type lit and, internally, I used type lit = int (an array index). That worked fine. Later, I changed the internal representation from an int to a record. Ideally, that would have no effect on users of the module, but OCaml allows testing abstract types for equality, which resulted in each comparison recursively exploring the whole SAT problem. It can also cause runtime crashes if it encounters a function in this traversal. Haskell’s type classes avoid this problem by letting you control which types can be compared and how the comparison should be done.

Speed

Python is well known for being slow, but much of what real programs do is simply calling C libraries. For example, when calculating a SHA256 digest, C does all the heavy lifting.

Despite this, I’ve found OCaml to be fairly consistently 10 times faster in macro benchmarks (measuring a complete run of 0install from start to finish). Also, although I’ve added a lot of code and dependencies since the initial benchmarks, it still runs almost as quickly. The 0release benchmark took 8ms with June’s minimal version, compared to 10ms with the final version.

When doing pure calculations (e.g. a tight loop adding integers), OCaml is typically more than 100x faster than Python.

Even so, OCaml is probably not a great choice for CPU-intensive programs. Like Python, it has a global lock, so you can’t have multiple threads all using the CPU at once. But if you’re writing small utilities that need to run quickly, it’s perfect.

No dependency cycles

Perhaps I’m making a virtue of a flaw here, but I like the fact that OCaml doesn’t allow cyclic dependencies between source files. I think this leads to cleaner code (back when I was writing Java, I wrote a script to extract all module dependencies and graph them so I could find and eliminate cycles).

What this means is that in any OCaml code-base, no matter how complex, there’s always at least one module that doesn’t depend on any of the others and which you can therefore read first. Then there’s a second module that only depends on the first one, etc. For example, here are the modules that make up 0install’s GTK plugin (note the lack of cycles):

Cycles can be a problem when converting existing code to OCaml, though. For example, the Python had a helpers.py module containing various high-level helper functions (e.g. get_selections_gui to run the GUI and return the user’s selections, and ensure_cached to make sure some selections are cached and download them if not). That doesn’t work in OCaml, because the helpers module depends on the GUI, but the GUI also depends on the helpers (the GUI sometimes needs to ensure things are cached). The result is that I had to move each helper function to the module it uses, but I don’t mind because the result is a clearer API.

Another example is the Config object. When I started the Python code back in 2005, I was very excited about using the idea of dependency injection for connecting together software modules (this is the basis of how 0install runs programs). Yet, for some reason I can’t explain, it didn’t occur to me to use a dependency injection style within the code. Instead, I made a load of singleton objects. Later, in an attempt to make things more testable, I moved all the singletons to a Config object and passed that around everywhere. I wasn’t proud of this design even at the time, but it was the simplest way forward. It looked like this:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 class Config(object): @property def fetcher(self): if not self._fetcher: from zeroinstall.injector import fetch self._fetcher = fetch.Fetcher(self) return self._fetcher @property def trust_mgr(self): if not self._trust_mgr: from zeroinstall.injector import trust self._trust_mgr = trust.TrustMgr(self) return self._trust_mgr ...

OCaml really didn’t like this design! config.py depends on all the other modules because it calls their constructors, while they all depend on it to get their dependencies.

Note that this design isn’t very safe: Fetcher’s constructor could ask for config.trust_mgr, and TrustMgr’s constructor could ask for config.fetcher. In Python, we have to remember not to do that, but in OCaml we’d like the type checker to prove it can’t happen.

In most places, I fixed this by passing to each constructor just the objects it actually needs, which is cleaner.

Another approach, which I used when lots of objects were needed, is that instead of requiring a config object, a class can take simply “an object with at least fetcher and trust_mgr methods”. Then we know statically that it will only call those methods, even though we actually give it the full config object.

The result of all this is that you can look at e.g. the fetch.mli interface file and see exactly which other modules it depends on, none of which will depend on it.

GUI code

Converting the GTK GUI to OCaml (using the LablGtk bindings), I replaced 5166 lines of Python (plus 1736 lines of GtkBuilder XML) with 4017 lines of OCaml (and no XML). I’m not sure why, but writing GTK code in OCaml just seems to be much easier than with Python.

I used GtkBuilder in the Python code in the hope that it would make it easier to modify the layouts, and to improve reliability (since the XML should always be valid, whereas Python code might not be). However, it actually made things harder because Glade (the editor) is constantly trying to force you to upgrade to the latest (incompatible) XML syntax, and I ended up having to run an old OS in a VM any time I wanted to edit things.

In the OCaml, the static type checking gives us similar confidence that the layout code won’t crash. Also, with GtkBuilder you name each widget in the XML and then search for these names in the code. If they don’t match, it will fail at runtime. Having everything in OCaml meant that couldn’t happen. [ Note: I later discovered that LablGtk doesn’t support GtkBuilder anyway. ]

Here’s an example of some OCaml GTK code and the corresponding Python code. This shows how to build and display a menu (simplified to have just one item):

OCaml 1 2 3 4 let menu = GMenu.menu () in let explain = GMenu.menu_item ~packing:menu#add ~label:"Explain this decision" () in explain#connect#activate ~callback:(fun () -> show_explanation impl) |> ignore; menu#popup ~button:(B.button bev) ~time:(B.time bev);

Python 1 2 3 4 5 6 7 8 9 10 11 global menu # Fix GC problem with PyGObject menu = gtk.Menu() item = gtk.MenuItem() item.set_label('Explain this decision') item.connect('activate', lambda item: self.show_explanation(impl)) item.show() menu.append(item) if sys.version_info[0] < 3: menu.popup(None, None, None, bev.button, bev.time) else: menu.popup(None, None, None, None, bev.button, bev.time)

Some points to note:

LablGtk allows you to specify many properties at once in the constructor call, whereas in PyGTK we need separate calls for each.
The Python API broke between Python 2 and Python 3, so we have to make sure to use the right one. It’s not sufficient to test the Python code using only one version of Python!
The Python bindings have always suffered from garbage collection bugs. If we don’t store menu in a global variable, it may garbage collect the menu while the user is still choosing - this makes the menu disappear suddenly from the screen!
Actually, I see that Python’s MenuItem takes a label argument, so maybe I could save a line. Or maybe that doesn’t work on some older version. It’s not worth the risk of changing it.

Update: I used the original layout for the OCaml above as I was comparing line counts, but it’s a bit wide for this narrow blog and some people are finding it hard to read. Here’s an expanded version which uses less special syntax:

1 2 3 4 5 6 7 8 let menu = GMenu.menu () in let explain = GMenu.menu_item ~packing:menu#add ~label:"Explain this decision" () in let callback () = show_explanation impl in let _signal_id = explain#connect#activate ~callback in menu#popup ~button:(B.button bev) ~time:(B.time bev)

Here, ~ indicates a named argument and # is a method call. So menu_item ~packing:menu#add ... is like menu_item(packing = menu.add, ...) in Python.

However, I did still have a few problems with the OCaml GTK bindings:

There’s no support for custom cell renderers. I used three of these in the Python version and had to find alternative UIs for each.
Various minor functions aren’t included for some reason. The ones I wanted but couldn’t find were Dialog.add_action_widget, Style.paint_arrow, MessageDialog.BUTTONS_NONE, Dialog.set_keep_above, icon_size_lookup and Selection_data.get_uris.
You usually don’t need the result of creating a label or attaching a signal so you need to use ignore, which can cause silent failures if you forgot an argument (it will ignore the partial function rather than the widget or signal result). Probably I should make ignore_signal and ignore_widget utility functions.
I had to work around a bug in the IconView support (reported, but with no response).

API stability

This is a community thing rather than a language issue, but OCaml and OCaml libraries seem to be very good at maintaining backwards compatibility at the source level. 0install supports the old OCaml 3.12 and libraries in Ubuntu 12.04 up to the latest OCaml 4.01 release without any problems. The only use of conditional compilation for compatibility is that we don’t define the |> operator on 4.01 because it’s already a built-in (this avoids a warning).

On the other hand, binary compatibility is very poor. You can replace the implementation of a module with a newer version as long as the public interface doesn’t change (good), but any change at all to the interface requires everything that depends on it to be recompiled, and then everything that depends on them, and so on.

For example, if the List module adds a new function then the signature of the List module changes. Now any program using the new version of the List module is incompatible with every library binary compiled against the old version. Even if nothing is even using the new function! This means that distributing OCaml libraries in binary form is effectively impossible.

Summary

OCaml’s main strengths are correctness and speed. Its type checking is very good at catching errors, and its “polymorphic variants” are a particularly useful feature, which I haven’t seen in other languages. Separate module interface files, abstract types, cycle-free dependencies, and data structures that are immutable by default help to make clean APIs.

Surprisingly, writing GTK GUI code in OCaml was easier than in Python. The resulting code was significantly shorter and, I suspect, will prove far more reliable. OCaml’s type checking is particularly welcome here, as GUI code is often difficult to unit-test.

The OCaml community is very good at maintaining API stability, allowing the same code to compile on old and new systems and (hopefully) minimising time spent updating it later.

Keybase.io

Thu, 13 Feb 2014 15:54:50 -0800

Comments: "Keybase.io"

The Day We Fought Back by the numbers

Thu, 13 Feb 2014 15:58:50 -0800

Comments: "The Day We Fought Back by the numbers"

URL: https://thedaywefightback.org/the-results/

Moon | Typeset In The Future

Thu, 13 Feb 2014 18:30:52 -0800

Comments: "Moon | Typeset In The Future"

URL: http://typesetinthefuture.com/moon/

I’ll tell you exactly where we are now. We are in the FUTURE; and we are on the MOON.

After studying 2001: A Space Odyssey in intimate detail, Duncan Jones’s Moon was the logical choice for my second foray into sci-fi typography. As this opening shot illustrates, Moon is a bleak, lonely, and above all beautiful love-letter to classic sci-fi typography and design. It’s also one of my favorite sci-fi films of all time.

UPDATE: I forgot to mention it originally, but this article contains some pretty massive spoilers about the plot of Moon. In my defense, you’re here to read about typography, and typography is always essential to the plot. Nonetheless, if you haven’t seen Moon, go and watch it first. The future will still be here when you’re done.

The searching question above is the opening shot of an infomercial for Lunar Industries. Their moon-based mining of Helium-3 is providing a plentiful source of energy on Earth. (On the evidence of the advert, they sound like a great company to work for. I’m sure they have strong ethical values.)

Sounds like science fiction, right? I’m afraid that’s where you would be wrong, sir or madam. As of January 2014, NASA is accepting applications from companies who want to mine the moon. This film isn’t science fiction – it’s inevitable science fact.

Fittingly, the infomercial finishes with an animation of the words ‘sun / moon / earth / energy / future’…

…and a transition to the Lunar Industries logo:

As introductions go, it doesn’t get more Eurostile Bold Extended than that.

And that’s where I have bad news, I’m afraid. It’s not Eurostile. It’s not even Eurostile’s daddy, Microgramma. According to conceptual designer Gavin Rothery, it’s actually Microstyle.

You’re probably wondering how to tell the three apart, right? Don’t worry – it’s trivially easy, as the big number two below will illustrate. Eurostile is in blue; Microgramma is in red; Microstyle is in green:

I’m glad we’ve cleared that up. (For ease of blog post cross-referencing, I’m going to put my fingers in my ears and continue to call it Eurostile anyway.)

Of course, sci-fi typography isn’t just about the Eurostile, you know. Eurostile has a long-standing competitor; a rival, if you will. And the yang to Eurostile’s tenuous yin is none other than our rectilinear friend, Bank Gothic. Moon uses both fonts in quick succession for maximum futuristic effect.

A clever 3D layout of Bank Gothic Medium is used for much of the film’s opening credits, superimposed in-and-over each of the positioning scenes:

One of these scenes introduces us to the film’s central character. Look! It’s Sam Bell, running on a treadmill to keep fit during his three-year stay on the moon. But what’s that on his t-shirt?

Haha! It’s a meaningless, throwaway gag about running. Because ‘runners don’t quit’, do they? Unless this comedy 80s typography has a deeper, hidden meaning…

One of my favorite things about Moon is its constant foreshadowing of the movie’s central twist. Wake Me When It’s Quitting Time is a pretty damn blatant forward reference to what happens later in the film. (And I hate to say it, type-fans, but that’s not Helvetica on Sam’s t-shirt. It’s Arial. You can tell from the Q and the G.)

This opening sequence is also the first time we discover that the moon base is called ‘SARANG – 사랑’:

This is essentially a duplication – the word ‘sarang’ is an English rendition of the Korean word ‘사랑’, which means ‘love’ or ‘affection’. (‘Sarang’ is also etymologically and graphically close to ‘saram’ or ‘사람’, which, ironically, can mean either ‘person’ or ‘people’.) In either case, it’s an unfortunate choice of name for Sam’s permanent home on the Moon, many thousands of miles away from his beloved wife and daughter.

Next up is Moon‘s title card, an inverted homage to Apollo 8‘s famous Earthrise photograph:

…albeit with more Bank Gothic than the original:

(The gradient fill effect on the MOON logotype is a popular sci-fi trope for adding drama to typography. We’ll revisit this in future blog posts.)

Moon uses an interesting angular typeface for its location-establishing shot:

This typeface is OCR-A, which was designed in 1968 for use in optical character recognition systems. It’s actually an ISO standard for character recognition. Moreover, it looks like THE FUTURE, and so it makes a perfect choice for on-screen interstitial positioning shots. (Matthew Skala has very kindly made a modern implementation of OCR-A available for free on his web site.)

Back inside the base, Sam notices that one of the four HE3 harvesters has a full load ready for collection. The on-screen display is classic Bold Extended sci-fi, with bonus points for yellow-and-white text and barber’s-pole patterning:

Also notable in this screenshot is a juxtaposition between the precision of the harvester’s on-screen display, and Sam’s hand-written customization of the surrounding fascia. This human customization of a clinical surrounding is something we certainly didn’t see in 2001, but it’s a common theme in Moon. Sam’s boredom keeps finding creative and subversive outlets within the strict design confines of the moon base environment.

Sam’s pressurized suit has three fabric mission patches:

Although Sam’s patches follow the Apollo mission patch trend for Bold Extended typography, they are very different from those of the Apollo era. NASA astronauts always had creative input into the patches for their missions. In his fantastic autobiography Carrying The Fire, Command Module Pilot Michael Collins describes the design process for the Apollo 11 mission patch typography:

I also penciled APOLLO around the top of my circular design and ELEVEN around the bottom. Neil didn’t like the ELEVEN because it wouldn’t be understandable to foreigners, so after trying XI and 11, we settled on the latter and put APOLLO 11 around the top.

Sam’s patches are the exact opposite of those from the Apollo era – rectangular rather than round; corporate and branded rather than personal and decorative.

Sam records a message for Lunar Industries central back on Earth. He’s unable to speak to them live, due to an ongoing problem with the long-range comms:

The on-screen interface is once again all about the Bold Extended. One of the buttons to the right of the screen has an interesting label. I wonder what it does?

Sam isn’t happy about the fact he’s been on the base for nearly three years. “I’m talking to myself on a regular basis”, he says. (Be careful what you wish for, Sam.)

Sam’s attempts to customize and personalize his environment continue in the living quarters. He’s keeping count of his days on the Moon with a dry-wipe marker on the bathroom wall. By my reckoning, this is 146 days and counting – not quite the nearly-three-years mentioned in the plot.

UPDATE: Patrick Devine has commented on Hacker News that there are actually 156 smileys, not 146. Turns out I miscounted. Patrick also notes that three years is exactly 156 weeks. Good spot, sir!

It’s not clear whether the facial emotions represent Sam’s mental state day by day. Nonetheless, they are a neat parallel to the simple expressions seen on the front of GERTY, Sam’s AI companion in the base:

GERTY communicates his emotions to Sam via the GERTY Unit Primary Emotional Interface. He has a stock set of emoji faces to draw on, based very closely on the standard emoticons that any Internet user will be familiar with. I’ll reference a few of them throughout this article, along with their official emoji names.

Sam has drawn a window and pastoral scene on another metal wall, and surrounded it with family photos…

…and the functional Eurostile of a harvester status screen is similarly customized with family photos and postcards:

Just readable amongst the photos is a Post-it note detailing Sam’s cheeky plans for starting the day:

07.30: Have a wank 07.32: Clean up mess 07.33: Nice cup of tea and a bickie

Sam might be American, but the film’s director and designer are definitely British. (To explain for Americans: ‘bickie’ is British slang for ‘biscuit’; ‘biscuit’ is British slang for ‘cookie’; and you can work out ‘wank’ for yourselves.)

This Post-it is particularly ironic in light of the quotation we see as the camera pans down the wall of photos:

‘Abstinence is a good thing, but it should always be practised in moderation.’ – Anon

I’m delighted to report that the card is set in popular sci-fi font Swiss 911 Compressed.

Next up is a gratuitous shot of Sam having a shower. If this isn’t a ‘backside of the moon’ gag, I’ll be sorely disappointed.

I’ve had a closer look, but I can’t see any typography:

If there’s no typography, then I’m afraid we’re just not interested.

After his shower, Sam gets his hair cut by GERTY:

GERTY is using a futuristic vacuum-based device to suck up Sam’s hair and cut it to the perfect length. You can just make out a Lunar Industries logo on the device’s hi-gloss handle:

Here’s how the device looks in close-up:

HANG ON A MINUTE. What’s that written on the transparent plastic tube?

Crop. Zoom in. Move left. Zoom in again. Enhance.

That embossed text appears to say:

www.haircut.com 2″ EXTENDER

Is this the Moon production team saying that at some point in the future, haircut.com will be purchased by a futuristic hair-cutting company who make robotic trimming devices?

No, dear reader, it is not. Because that future is already here. TODAY.

It turns out that haircut.com is the present-day home of RoboCut Inc.:

…and that RoboCut Inc. are the makers of the RoboCut DIY:

That’s right – the product you see in Moon is a product you can buy yourself today, for only $59.99 (including free shipping).

Want to know how the RoboCut works? Here’s RoboCut founder and inventor, Dr. Alfred Natrasevschi, to explain:

Haircut completed, Sam expresses his annoyance that LunarSat (a.k.a. ‘long range comms’) still hasn’t been fixed. GERTY notes that it’s fairly low on the company’s priority list right now. His passive role in this scene is fascinating. If you watch Moon after 2001, you can’t help but be suspicious of GERTY, in the same way that Ripley can’t help but be suspicious of Bishop in Aliens after her experience with Ash in Alien. GERTY is a calm, relaxing AI voice with a glowing camera lens and corporate logo on the front. Moon certainly isn’t afraid to play on our expectations. After all, what could possibly go wrong?

Sam listens to a message from his wife and daughter. He can’t talk to them live due to the LunarSat mishap, but he’s delighted to see them nonetheless. Just look at the joy on his face:

No. Don’t look at his face; look at his clipboard. That’s where the typographic interestingness is most likely to be found.

Crop. Zoom in. Rotate ninety degrees. Zoom in again. Enhance.

Hold on a minute – it looks like this clipboard was signed on the 16th January 2008. But that’s not THE FUTURE – indeed, it’s not even after Moon was released!

Moreover, who’s the signatory? It looks remarkably like the name of the film’s line producer. (Damn you, high-definition film releases.)

Later on, Sam watches some TV. Specifically, he watches Bewitched – or as it’s also known in the future, ‘SUPERNATURAL COMEDY LADIES’:

As he makes a cup of tea, he spots a strange girl sat in his chair:

While he’s distracted by the plot, let’s take a look at two typographically-interesting posters in the background:

This poster, with its square iconography and cheery SELECTION EVENT – HEARTY FEEDING – PLEASE ENJOY – ORGANIC SUSTAINANCE MENU, makes you wonder if the Lunar Industries catering team learnt their trade at Aperture Science.

Even more interesting is a sign just visible beneath the pre-packed food boxes:

Organic Material – Soylent Storage

Soylent. That name sounds familiar. Surely this isn’t the same foodstuff made by the Soylent Corporation in 1973′s Soylent Green? Because if it is, that would be remarkably worrying. Soylent Green is People!

At least we can be confident that this Soylent isn’t made from people. After all, where would you get them from? There’s only one person on this moon base, and that’s Sam Bell.

Sam burns his hand in all the kerfuffle. GERTY fixes him up in the infirmary. Just visible behind Sam’s head is a screen with the comforting advice to “TRUST ROBOTIC ASSIST”:

ROBOTIC ASSIST is the official name for GERTY 3000, Sam’s AI companion in the base (voiced by Kevin Spacey doing his best HAL impression). And why not TRUST ROBOTIC ASSIST? I’m sure the 3000 series has a perfect operational record.

Sam goes to bed. Beside his bunk is issue 15 of Take Off Magazine:

Take Off was a 132-issue aviation partwork published in the 1980s by Eaglemoss. This issue features V-Bombers and Biz-Jets:

Sam has a sexy dream about his wife. It does not, however, contain any typography.

Sam is rudely awoken by his Karlsson Digibell Alarm Clock, which features a classic LCD font. Sam’s future version of the clock has a bonus feature compared to the one you can buy in shops today. That’s right – Sam’s version plays Chesney Hawkes songs.

Specifically, it wakes him up by playing Chesney’s number one hit The One And Only. “I am, the one and only”, croons Mr. Hawkes.

Yes, you heard me right – the filmmakers are foreshadowing upcoming events via the lyrics of a Chesney Hawkes song.

Over food, we see more of Sam’s Post-it reminders:

Water plants!! 11:30 Gaze at earth from window

…and a picture of the Earth, with Wish You Were Here on it. He really does want to go home, you know.

Sam also has an IKEA instruction booklet tucked behind his Lunar Industries sheet. The reasons for this are somewhat unclear.

Sam heads out to retrieve some more HE3 from Matthew, one of the harvesters. He sees a second vision of the mysterious girl, and crashes into the side of Matthew, damaging both himself and his rover. The rover’s emergency sign once again asks him to TRUST ROBOTIC ASSIST:

There’s a clear message coming through from these on-screen text displays: if things go wrong, ROBOTIC ASSIST should be TRUSTed to make things right. Let’s fade to black with that sensible advice in mind.

Fade up from black. Why hello, Sam! You’re looking remarkably fresh-faced for someone who’s just nearly killed himself in an accident. How’s the head? Fuzzy? Tell you what, why not let GERTY run some tests to check for brain damage:

This test is a Concentration-style matching pairs game. (Unlike Concentration, these cards all have Lunar Industries logos on the back.) Some of the card symbols are reminiscent of Zapf Dingbats – such as the HEAVY GREEK CROSS, and a rotated BLACK DIAMOND MINUS WHITE X – but I do wonder if I’m reading too much into things, and the design team haven’t just drawn some pretty symbols in Illustrator. (I had hoped that the asterisk card might be my own personal favorite, the HEAVY EIGHT TEARDROP-SPOKED PROPELLER ASTERISK, but alas it is not to be.)

“Let’s try another test” says GERTY, like some kind of benevolent GLaDOS. The analogy makes me trust him even less.

Shortly thereafter, we overhear GERTY saying something to Lunar Central about “the new Sam”. What’re you going on about, GERTY?

I’ll tell you exactly what he’s going on about. This is a brand new Sam. He’s a clone of the original Sam Bell, and he’s just been woken up by GERTY. This might get confusing rather quickly, so let’s lean on our good friend ‘typographic convention’, and use subscript to refer to him henceforth as Sam2. Trust me, it’ll make life a damn sight easier once you know where we’re going with this.

Sam2 spots that Matthew has stalled. He asks GERTY to unlock the doors so that he can go and fix it. GERTY says “I’ll pass on your message”. (This is Lunar Industries speak for “I’m sorry Sam, I’m afraid I can’t do that”.)

Sam2 has a dream about his wife. This time, it’s not sexy at all – it’s psychologically and typographically disturbing. There’s a scary Sam1 under the bedclothes, and he works for RANUL SEIRTSUDNI, on the 랑사-GNARAS base:

Lunar Central tells Sam2 to stay put. GERTY apologies for being under strict orders not to let him outside. He’s acting like some kind of… benevolent HAL 9000. This analogy actually endears him to me slightly. Sam2 tricks GERTY into letting him go outside anyway. He finds Sam1 in the rover, and brings him back to the base. GERTY tells him that the person he’s found is also Sam Bell.

GERTICON: OHMYCRIKEYFACE

A considerable amount of plot happens. It’s all fascinating, but if I’m honest, it’s rather light on typographic action. I feel I’d be doing us a disservice to document the whole lot, but I will report an important bit of GERTY dialog when he’s explaining events to Sam1:

“I’m here to keep you safe, Sam.”

That sounds a lot like the First Law Of Robotics to me. If GERTY is programmed to follow the Laws, his robot AI is about to have some interesting choices to make.

A message comes through from Central to say that the crew of the Eliza are on their way to ‘rescue’ Sam2:

They look like a friendly bunch, don’t they? The instructions next to their mugshots say:

Lunar Industries rescue crews have your best intentions at heart. Please try not to panic until they arrive. Remain on-station and make sure you obey their instructions no matter how strange they may seem. After all they’re here to help!

What could possibly go wrong with a crew called Rothery, Ward and Shaw?

Sam2 realizes he must have come from somewhere on the base, and starts hunting around. He and Sam1 have a fight. Sam1 looks in a topical multi-mirror. (Don’t worry, that’s definitely Sam1 in both mirrors. After all, it’s not like there are any more Sams around, right?)

GERTY and Sam1 have a Big Chat. When pressed for the truth about his wife, GERTY says “I can only account for what occurs on the base”.

GERTICON: NOTHINGTOSEEHEREFACE

After some cajoling, GERTY finally tells Sam1 the truth. He’s a clone, and his memories of his wife and daughter are memory implants.

GERTICON: SADTRUTHFACE

The Sams find some signal-blocking antennas outside the base. Sam1 goes back to look at his old video logs. GERTY enters a special password on a Windows Keyboard to unlock all of the video diaries:

Eagle-eyed viewers will notice that the password begins with KLGAN. Unfortunately, the security-conscious Duncan Jones cuts away from the keyboard just before we can hack into his email.

Sam1 learns that Lunar Industries seem to view the words ‘termination’ and ‘contract’ in much the same way as the Mafia. Sam2 finds yet more long-range-comms-blocking aerials. He prints out their coordinates on a handy receipt printer:

I very much doubt those are going to become significant later on.

Now: I’m sure you’ve already noticed that these are Selenographic coordinates, as used to refer to locations on the Earth’s Moon. Specifically, the co-ordinates are:

LAT 034°23′01.2″S, LONG 124°56′67.6″E LAT 121°09′56.2″S, LONG 045°34′56.4″E

We’ll gloss over the fact that one of these numbers has sixty-seven seconds in a minute. However, it’s a little hard to ignore the latitude value of 121°S, given that the maximum value that latitude can take is 90°S at the pole. (Maybe Sam mistyped it due to the stubby fingers you get on space gloves.)

Back at base, the Sams discover a hidden room. Look at that: it’s Sams all the way down!

Trays 0001 through 0006 have already been opened. Sam0005 and Sam0006 (for it is they) decide to open tray 0007:

It’s the small details of this scene that I find most disturbing. Every pre-packed Samn has a pre-packed Wake Me When It’s Quitting Time t-shirt. The person at Lunar Industries responsible for this whole macabre set-up not only had the temerity to subject Sam Bell to his own personal Groundhog Day; they also had the gall to leave cynical in-jokes for every iteration to endure.

Once they’re out of the Tunnel Of Clones, Sam1 asks GERTY why he helped with the password. “Helping you is what I do”, says GERTY.

See, Sam1? I told you you should TRUST ROBOTIC ASSIST – he’s definitely read the Laws. (Although he may have skim-read the bit about not killing people in a space-coffin.)

Sam1 drives the rover a very long way from the base, and re-establishes video contact with Earth. He calls his house. Turns out his wife is dead, but his now-15-year-old daughter has a chat with him anyway. She ends by shouting off-screen to her Dad to say that someone is asking about Mom. We hear the distinct voice of a certain Sam Bell in reply. Sam0 is still alive! (Sam1 closes the video call device quickly, and looks sad and pensive.)

The known-to-be-alive-ness of Sam0 gives an an alternative, cheerier way of looking at the pre-packed t-shirt dilemma for the less depressingly-morbid of you. Presumably Sam0 is complicit in the fact that thousands of Sams are in cold storage on the Moon. If he’s complicit in the scheme, could it be that these small personal details – even that seemingly-sinister t-shirt – have been chosen with Sam0‘s involvement, as homely comforts to soften the terror of Samn‘s reality?

I don’t think that’s the truth of it at all. But if helps you sleep at night, you’re more than welcome to run with it.

We’re back at the base. Sam1 continues to fall apart. Sam2 sees Sam1‘s video call with his daughter, and realizes that he and Sam1 will be killed when the Eliza crew arrive. He convinces GERTY to wake up a new clone. Let’s call him Sam3.

This section of the film gives us a rare close-up of the base’s OPERATIONAL NOTIFICATIONS board:

Amongst other things, we discover that Lunar Industries Ltd. is a Registered Company Of The United Kingdom No. 06346944. Delightfully, this is indeed the case. In addition to the scrolling Eurostile, there’s also a Big Countdown Clock (in a pseudo-LED font), reinforcing the time pressure of Eliza‘s imminent arrival. (It was conveniently fixed at 88:88 before the Eliza plot point came into effect.) Notable on both font displays is an illustrative dot-matrix resolution substantially lower than the actual resolution of the fonts displayed thereon.

Sam1, who is by now in a right state, wakes up to Chesney Hawkes. (See? I told you it was foreshadowing.) He discovers Sam3 in the infirmary.

Talking of foreshadowing: on the movie’s original one-sheet poster, there are three copies of Sam Rockwell’s name, as shadows behind the main one:

Strictly speaking, this is an accurate cast list. The film now stars Sam Rockwell0, Sam Rockwell1, Sam Rockwell2, and Sam Rockwell3.

To put it another way – the movie’s poster is four-shadowing the foreshadowing.

Maybe I should go and have a lie down for a bit, and come back when the conspiracy theories have subsided. It’s a shame sci-fi films don’t have intermissions these days. Let’s transplant the one from my 2001: A Space Odyssey post, and go and have a cup of tea while the Sams work out what to do next.

And we’re back. After much debate, it is decided that Sam1 will go back into the crashed rover; Sam2 will take the HE3 launcher back to Earth; and Sam3 will stay in the infirmary to be discovered by the crew of Eliza. He’s still unconscious, so he doesn’t know this yet, but it’s a decision made by a quorum of hims, so he can’t really complain.

Once he’s dropped Sam1 off, Sam2 loads up the HE3 launcher. GERTY suggests turning him on and off again so that he can’t provide any incriminating evidence to Eliza. “I’m here to keep you safe, Sam. I want to help you.”

Sam2 asks GERTY if he’ll be okay. “Of course. The new Sam and I will be back to our programming as soon as I’ve finished rebooting.”

GERTICON: YAYPROGRAMMINGFACE

Sam2 replies: “Gerty, we’re not programmed. We’re people, you understand?”

As GERTY turns round, we see that Sam1 has stuck a KICK ME Post-it on GERTY’s back side:

Sam2 turns GERTY off. After a short, poignant pause, he removes the Post-it from his back.

GERTICON: REBOOTINGFACE

Sam2 gets ready to launch back to Earth. At the very last minute, he realizes he can use the harvesters to knacker the signal-blocking aerials, and avoid Sam3 suffering a similar fate to his own. He grabs the piece of paper with Chekhov’s Coordinates on it:

…and types a long series of numbers into a keyboard with great precision despite wearing spacesuit gloves. The harvester coordinates update. Go Sam2!

This screen prompts a couple of interesting questions:

1) What’s an ‘OLD MAN OVERIDE’?

2) Is that an infinitely-looping BASIC program?

20 GOTO 10 10 GOTO 20

The Eliza arrives. Sam3 wakes up and asks GERTY what’s going on.

GERTICON: DONTASKMEJUSTREBOOTEDFACE

Eliza crew members find Sam1 in his rover. Sam2 has a psychedelic trip through space. A harvester crashes into an aerial. Long Range Comms finally start working. And if you look closely, you discover that the contents of the portrait-orientation comms monitor is actually being played from a DVD player plugged into a landscape monitor turned 90 degrees clockwise:

All of which is a timely reminder that this frankly awesome sci-fi film was made on a budget the size of a postcard. Which just makes its amazing styling and design all the more impressive. Indeed, Moon is one of my favorite examples of sci-fi storytelling through design – and, to my mind, a worthy successor to Kubrick’s masterpiece.

- @daveaddey

Postscript: Apart from the note about Microstyle, and a nod to the DVD players, I deliberately wrote this post without referring to Moon Conceptual Designer Gavin Rothery‘s superb ‘They Never Went To The Moon’ blog. I much prefer to write these posts as a response to just what I see on screen – it’s more fun that way. You should totally go and read his blog in its entirety, however, as he explains or expands upon much of what I’ve written above. The blog also includes images of many of the computer screens and stickers from around the base, along with Gavin’s original renders for the base design. Good work, sir!

Flappy Space Program by corpsmoderne

Thu, 13 Feb 2014 21:30:25 -0800

Comments: "Flappy Space Program by corpsmoderne"

URL: http://corpsmoderne.itch.io/flappy-space-program

The Economics of Star Trek — Editor's Picks — Medium

Thu, 13 Feb 2014 22:04:25 -0800

Comments: "The Economics of Star Trek — Editor's Picks — Medium"

URL: https://medium.com/editors-picks/29bab88d50

I promise this is about Star Trek. Sort of. Bear with me a moment.

I’ve been reading a lot about robots lately. When I read about robots, and the future, I can’t help but think about it in economic terms. And that inevitably turns my mind to the branch of economics called post scarcity economics. Traditional economics, of course, deals with the efficient allocation of inherently scarce materials. Post scarcity economics deals with the economics of economies that are no longer constrained by scarcity of materials — food, energy, shelter, etc.

The thing that never sits quite right with post scarcity economics, though, at least the very little that I’ve read, is that it’s always sort of an all or nothing affair: you either don’t have enough of anything or you have enough of everything. Thinking of this as a mental exercise is kind of fun, I think, but in reality it seems to me that getting from point A — a scarcity economy — to point B — post scarcity — is going to be a long, complicated journey as some things become more abundant in some places, while other things are still scarce.

What is needed is some sort of interim-, or proto-post scarcity economics.

More and more I find myself thinking we are, as a race, constrained by the economic models we have. We have capitalism, of course, the proverbial worst model except for every other one that dominates much of our planet right now. It’s definitely a scarcity-based system. Then we have the centrally planned systems of Communism and Marxism, not particularly effective, as it turns out. We have European-style socialist capitalism, but that’s still capitalism, and scarcity-based, albeit with a much more robust safety net than we have here in the US. Some Americans seem to think that a robust safety net somehow nullifies the distributed planning of capitalism. I’ll listen to them again when our schools are decent and our life span starts increasing again magically.

The key here, to me, is to start thinking about how economics would work when we decouple labor from reward. Does that make a system inherently communist? I don’t think it does. People work. They get paid. It is market driven, and not centrally planned. In reality,the market already basically dictates this, for who can claim that a Wall Street banker works more than a teacher? The only thing we really need to do is take this to a logical extreme: that people can still get paid doing zero work. This fear seems to be at the heart of most people who say that Europe is communist: if we give people so much welfare, some of them might stop working! Quelle Horreur!

It seems to me that with the rise of machines and robotics, advances in mining technology, energy technology (both fracking and green energy technologies), the obesity epidemic in the US, etc., that there are plenty of reasons to believe that we may be at the beginnings of a post scarcity economy. We have a surplus, no doubt. Of course, we still have legions of people in the world that are starving, and even people still here at home. But we actually have the capacity to feed them, to feed everyone, even now, even if we don’t have the will. It’s not a matter of scarcity; it’s a matter of the organization of labor and capital.

Take a mental journey for a moment with me: what if, one day, technology reaches the point that a small number of humans — say, 10 million — can produce all of the food, shelter and energy that the race needs. This doesn’t seem like insanely wishful thinking, given current trends. There’s no rational reason why the advances in robotics, factories, energy and agriculture could continue unabated for long periods of time. Of course I’m not saying they will, but rather, they could.

So, then, take that journey. What, then, of labor? In today’s terms, a ‘healthy’ economy now is one at or near full employment. A healthy economy now is one where everyone has a job. But in our mental exercise, those jobs are actually unrelated to a healthy economy, at least from strict economic terms. Everyone’s fed and housed and tons of people simply don’t need to work. Right now, we have them working making shit we don’t need. Is that any better than them not working?

I give you we’re in some fringe areas of economics here, but I have always wondered: is there any economic proof that we need full employment to reach full satisfaction of needs? To my knowledge, there isn’t. There’s a body of economics that goes into standards of living, and the increased standard of living. And here we get to our shitty world of unabated consumerism,Thorstein Veblen’s conspicuous consumption and George Battaille’s accursed share — the inevitable destiny of all economies to eventually produce more than they need, and, thus, waste it.

Seems to me that if we could think beyond capitalism and think of a new model, we could break out of this pointless cycle of more and more consumption of shit we don’t need and model things in another way.

Yes yes, of course. We all know that. The problem seems to me that the minute we leave capitalism behind, we only look at the past alternatives of communism, marxism and pure socialism and pooh pooh them. Few people seem to be able to look beyond capitalism without regressing to the other failed economic models of the early 20th century, as if they are the only alternate possibilities for man.

Yet there have been some other attempts to leave capitalism behind that attempt to also leave the baggage of communism, marxism and socialism behind. The most notable is participatory economics, or parecon. This is a worthwhile attempt, I think, but to me it doesn’t quite pass the smell test of being sufficiently un-communist, what with its workers councils and lack of any sort of ruling class. All very un-American, and in any case, a bit preoccupied with “workers” and “individual need” to really work in any post-scarcity economy where the very concept of a laborer is iffy. When you start thinking this way you start getting into the dodgy world of heterodox economics and, well, that’s a world of a lot of crackpots. Some good ideas, sure, but a lot of crackpots, and more to the point, it’s a world devoid of empirical research, which is a serious problem. Economics is really at its worst when it’s just making up theories. It’s a lot more noble when there’s some real data to back it up.

Parecon does have some awesome concepts, though, by the way. I don’t hate it completely. I especially like that people’s say over any issue is proportional to the amount that issue affects them. It also has some states’ rights-ish aura similar to “laws being made at the level closest to those affected.” It’s a worthy school of thought to consider when looking for a pure alternative to capitalism in a vacuum, though probably not very practical in reality for reasons similar to communism (despite not being centrally planned, it still very much hinges on some third party deciding the relative worth of each job — a messy business). More to the point, it doesn’t help us in thinking about our mental picture: a world where a small number of people can produce enough for everyone.

Then I got to thinking. Screw the dodgy world of heterodox economics. Let’s go full-on fantastical and look at sci-fi. There IS actually a model out there that deals fairly realistically with a post scarcity economy. Not only that, it actually takes into account the difficulties of migrating from a capitalist society to a post scarcity society incrementally. It’s not just a theory in a vaccum.

It’s called Star Trek.

Stay with me here.

Star Trek and Economics

The Previous Theories

When looking at the economics of Star Trek, there have been three broad approaches in the past:

Trying to shoehorn Star Trek’s economics into the model of parecon. This is problematic because of the obviously hierarchical society of Starfleet, with Admirals, captains, commanders, chancellors, governors and whatnot, and the clear existence of a relatively strong Federation president, who is democratically elected. Plus we never once see a labor meeting, and it’s pretty obvious personal freedom and enrichment are important to society.Calling the Federation Communist, based on comments from Kirk in Star Trek IV on not having any money in the future and Picard’s speech about the economics of the federation being significantly different than 21st century economics and people pursuing personal enrichment rather than the accumulation of wealth. The problem with this definition is it’s lazy — just because they don’t pursue the accumulation of wealth, it does not mean the Federation is communist. There is absolutely, obviously, still private property in the Federation: most obviously Joseph Sisko’s restaurant in New Orleans and Chateau Picard, evidencing that not just small possessions are allowed but that the land itself is still privately owned. One could argue that these aren’t really Sisko and Picard’s to own, but they are routinely referred to as “his” restaurant and vineyard so we gotta go with Occam’s Razor here and assume they do, in fact, own them.A sort-of guessing game based on the various mentions of Federation Credits and trying to glean the system from every single mention of money or payments within the series. This is always a pain in the ass, especially given the original series sometimes did things that were pretty out there according to later firmly established canon, and later firmly rejected by Roddenberry himself before his death. Additionally, many of the assumptions about Federation Credits seem iffy: are they really currency? Do they have to be? Are they scrip? Rations? We simply don’t know. And in any case, trying to define the entire economy of the Federation — and perhaps even learning something from it — should be more than a matter of resolving obscure trivia references (though of course it’s fun).

None of them seem correct. None of them seem realistic. And yes, let’s go for realistic here, why not?

Let’s take a different approach here.

What we know

Let’s start with the facts.

The Federation is clearly not a centrally planned economy, and therefore obviously not communist. Individual freedom of choice is very obvious. Everyone chooses their careers, and there are many mentions of this throughout the series — witness every single time someone waxes nostalgic about why they chose to enter Starfleet. Witness Bashir going on about why he wanted to be a doctor instead of a tennis player. Witness Wesley dropping out of Starfleet. Witness Vash being an archeologist and Kasidy Yates being a cargo ship captain.

Private ownership still exists — the biggest examples, to me, are Sisko’s restaurant and Chateau Picard, but many other examples abound from all the trinkets everyone owns in their quarters. Crusher’s family owns a (haunted) cottage on some old-Scottish settlement planet. The Maquis routinely refer to “our land,” which they presumably owned, and while an individual tribe may have collectively owned the land through a corporation, like the Alaska Native Land Claims Settlement Act, or through a co-op, they clearly “owned” the land, just like anyone else owned land, while the Federation was the superseding government that could give that territory away to another sovereign party, much like the ceding of the Sudetenland or Guam. Any alternative situation (the government owning the land and renting it to the settlers?) is never alluded to and in any case the words stated (“our land”) clearly indicate private ownership is still very much part of the cultural zeitgeist. Then we have JJ Abram’s Star Trek and it’s pretty unlikely that, what? The Federation owned that shack Kirk grew up in, that sweet Corvette or that roadhouse bar? Those items sure looked privately owned. Some spaceships were privately owned. Finally, let’s not forget Star Trek: Generations when Kirk says in the Nexus “This is my house. I sold it years ago.”

Next: The Federation is not true post scarcity economy: famines routinely still exist, transportation lines are vital in moving around goods within the Federation. Transportation is a whole grey area in most post-scarcity economic works, at least the few I’ve read. The Federation might have enough food, but at any time some planet may well be starving or in need of medicine that needs to come from somewhere else.

It seems pretty clear cut that jobs are optional. They explicitly state on many occasions that the Federation is based on a philosophy of self improvement and cultural enrichment, and in any case we sure do run into a lot of “artists” in the Federation. I particularly love those hippies in TOS. The Federation seems a bit like Williamsburg — a lot of artists who don’t need to work. Or maybe more like the UK at the height of its social programs supporting artists. Let a million JK Rowlings bloom. It’s a bit weird, to me, that we’ve never seen people who sit around and literally do nothing, but then why would we? And, of course, we’ve certainly seen more than a few societies that are all chilled out and not doing much (Risa, etc).

Next: The Federation doesn’t use money. This is basically absolute. Kirk says it in Star Trek IV. Picard says it several times. Quark mocks it to Rom. Roddenberry put it down as a hard and fast rule. No theory of Star Trek economics can be real while ignoring this fact. It has to be addressed. It is the basis of all confusion and, honestly, interest in figuring it out at all.

Money still exists, so do banks. Crusher buys fabric at Farpoint. DS9 makes mention of theBank of Bolias, on a Federation planet. Nog loans Jake latinum.

We also know there exists such a thing as the Federation Credit. This presumably causes some confusion since they are routinely referred to like money (Kirk mentions that the Federation has invested 122,200 credits in Spock), and things are purchased for credits (Uhura buys a tribble, Quark occasionally accepts them at his bar).

This would seem to be a giant contradiction to the lack of existence of money. We’ll get to that in a bit.

There is still a ruling class, or classes — it is not perfectly-egalitarian in a communist manner. We have admirals and presidents and governors and colony leaders. There are enlisted personnel in Starfleet and officers. Some are elected, some are appointed. Some Federation members were even hereditary nobilities.

There is still commerce (and even Vulcan commerce), trade, trading vessels, and, we can assume, corporations, in some form (though this may not be 100% definite — Dytallix is mined for the Federation. It isn’t 100% clear it is in the Federation).

Some thought exercises

Let’s do a couple thought exercises.

First: if you eat a meal at Sisko’s Creole Kitchen, do you pay? It seems almost definite that you don’t pay. If you paid, with anything, including Federation Credits, that would be money. You could barter, but it seems if the entire economy was a barter economy, we’d hear it. No, it seems almost certain that you go to eat at Sisko’s, you don’t pay, and Joseph Sisko doesn’t pay for his supplies, and his suppliers probably don’t pay for theirs.

Next: Can everyone have anything? Anything at all? Is the Federation a perfect post scarcity society? The answer seems almost certainly no. If you went to a replicator, or a dealer, or the Utopia Planatia Fleet Yards and asked for 10 million star ships, the answer would be no. More concretely, when the Borg attacked, and during the Dominion War, the Federation suffered from a serious starship shortage.

Next: Imagine there’s some level of welfare benefits in every country, including America. That’s easy. That’s true. Imagine that, as the economy became more efficient and wealthy, the society could afford to give more money in welfare benefits, and chooses to do so. Next, imagine that this kept happening until society could afford to give the equivalent of something like $10 million US dollars at current value to every man, woman and child. And imagine that, over the time that took to happen, society got its shit together on education, health, and the dignity of labor. Imagine if that self-same society frowned upon the conspicuous display of consumption and there was a large amount of societal pressure, though not laws, on people that evolved them into not being obsessed with wealth. Is any of that so crazy? Is it impossible?

I think that is basically what’s going on on Star Trek.

A Theory of Star Trek Economics

I believe the federation is a proto-post scarcity society evolved from democratic capitalism. It is, essentially, European socialist capitalism vastly expanded to the point where no one has to work unless they want to.

It is massively productive and efficient, allowing for the effective decoupling of labor and salary for the vast majority (but not all) of economic activity. The amount of welfare benefits available to all citizens is in excess of the needs of the citizens. Therefore, money is irrelevant to the lives of the citizenry, whether it exists or not. Resources are still accounted for and allocated in some manner, presumably by the amount of energy required to produce them (say Joules). And they are indeed credited to and debited from each citizen’s “account.” However, the average citizen doesn’t even notice it, though the government does, and again, it is not measured in currency units — definitely not Federation Credits. There is some level of scarcity — the Federation cannot manufacture a million starships, for example. This massive accounting is done by the Federation government in the background (witness the authority of the Federation President over planetary power supplies).

Because the welfare benefit is so large, and social pressure is so strong against conspicuous consumption, the average citizen never pays any attention to the amounts allocated to them, because it’s perpetually more than they need. But if they go crazy and try and purchase, say, 10 planets or 100 starships, the system simply says “no.”

Citizens have no financial need work, as their benefits are more than enough to provide a comfortable life, and there is, clearly, universal health care and education. The Federation has clearly taken the plunge to the other side of people’s fears about European socialist capitalism: yes, some people might not work. So What? Good for them. We think most still will.

However, if they so choose they can also get a job. Many people do so for personal enrichment, societal pressure or through a desire to promote social welfare. Are those jobs paid? I would assume that yes, those jobs are “paid,” in the sense that your energy allocation is increased in the system, though, again, your allocation is large enough that you wouldn’t even really notice it. Why do I say this? The big challenge here is how does society get someone to do the menial jobs that cannot be done in an automated manner. Why would anyone? There are really only two options: there is some small, incremental increase in your hypothetical maximum consumption, thus appealing to the subconscious in some primal way, or massive societal pressure has ennobled those jobs in a way that we don’t these days. I opt for the former since it grounds everything in market economics, albeit on a bordering-on-infinitesimal manner, and that stands to reason, since that’s how people talk in Star Trek. They talk about individual fulfillment, buying, selling, etc. No one was ever guilt-tripped into joining Starfleet, save by maybe their family.

There is almost zero mention of central planning. It’s a capitalistic society, it’s benefits are just through the roof. Also, market economics = crowdsourced. That is, it’s not centrally planned. It’s democratic. It’s the only mechanism we know of to allocate resources that isn’t centrally planned. The alternative is that all allocations are done algorithmically through a computer and the economy is completely decoupled from market forces, but that’s still basically central planning, and infinitely more complex than assuming there is still some semblance of market underpinning, much like we stayed on the gold standard for far longer than we needed to and we still have pennies even though we don’t need them. It’s a vestige of the past. It’s the constitutional monarchy.

Either way, presumably, you take whatever job you want, and your benefits allocations are adjusted accordingly. But by and large you just don’t care, because the base welfare allocation is more than enough. Some people might care, some people might still care about wealth, such asCarter Winston. More power to them. They can go try and be “rich” in some non-Federation-issued currency. But most people just don’t care. After all, if you were effectively “wealthy” why would you take a job to become wealthy? It pretty much becomes the least likely reason to take a job.

So, behind the scenes there is a massive internal accounting and calculation going on — the economics still happen. They just aren’t based on a currency unit, and people don’t acquire things based upon a currency value. People just acquire things from replicators, from restaurants such as Sisko’s or coffee shops like Cosimo’s, or, presumably, get larger things from dealerships or (more likely) factories. This could still be called “buying,” as a throwback.

Two points here: first, the accounting is done in energy units, so that there is no need for currency. And why not? Resource allocation is mainly about energy anyhow, doubly so if it’s only robots building most things. And secondly, if you never had money, never saw it, and it didn’t physically exist to measure things, you’d pretty much tell people, like a certain 20th century oceanographer, that you don’t have money in the 24th century, regardless of some automated accounting. This jibes with Federation people knowing what money is — because other societies have it — but saying they don’t use it. Because they don’t.

However, you could still buy and sell things. You could take a thing from a replicator and go to someone else and “buy” something else with it. Why couldn’t you? It’s a free society. It’s essentially barter. Kirk may well have sold his house for a year’s supply of Romulan ale.

Or Federation Credits.

It is tempting to argue here that the massive accounting system uses a unit called the Federation Credit, but i don’t believe that’s the case. If it were, the credit would be too much like money because a) accounting is done in it, b) it is issued by a governing body (like a fiat currency) and c) it is fungible, i.e. you can already buy things with it and if you could buy things with it AND a and b were true, it would pretty much be a currency. This would fly in the face of Roddenberry’s absolute diktat that the Federation has no currency.

I’m gonna make a bold new theory here. Federation Units are “Federation” the same way that American Cheese is American. It is simply descriptive. Currency was invented long before capitalism as a means to disintermediate trades: you wanted my grain, I didn’t want your cows, I wanted farmer Ted’s grapes. Rather than make every trade a 3, 4 or 5 way trade, we made a little certificate we all agreed was worth something to us and us only. This need would still occasionally crop up in the Federation, even without money. I believe the Federation Unit is aprivate currency, developed by third parties to facilitate complex trades or trades outside the Federation. I believe that the Federation Unit is not actually underwritten or issued by the Federation. I think it is more akin to the Calgary Dollar or the Chiemgauer. Or bitcoin. This would solve so many problems. It would make it unequivocally true that the Federation doesn’t use money. It would give people a unit to use as reference when they say things are expensive. It would be a thing citizen’s could acquire, if they wanted to, through barter originally, then allowing them to use them to purchase things (like Tribbles or Holosuites) from people who elected to take them, since taking them is optional (witness Quark’s vacillations on whether he accepts them or not). It would make a nice proxy for talking about investment levels, such as when Kirk said how much the Federation had invested in Spock.

Foreign Reserves

Additionally, I believe that the Federation acts like any current sovereign nation state and holds foreign reserves of currencies of other nations. It’s assumed that not all foreign trade is done through barter. The federation itself probably holds foreign reserves in foreign currency just as China holds US dollars and England keeps a reserve of Euros. Sisko at one point tells Quark he could have charged rent for the bar, but he chose not to. Presumably that would have been paid in latinum. Presumably the Federation would have just held onto it as foreign reserves. All evidence, in fact, points to the fact that the Federation operates as a nation and uses foreign reserves exactly as we do now. The Chinese government holds US Dollars but you don’t here a Chinese person say “we use dollars.” This is a bit confusing by the episode in which the Federation offers 1.5 million Federation Credits for use of the Barzanian wormhole, but it doesn’t have to be contradictory. Federation Credits had value to the Barzanians, so the Federation could simply procure them from the issuer with its foreign reserves of other currencies at market rate.

The Individual Can Have Money

An individual of the Federation can procure latinum by barter for goods, labor or, presumably Federation Credits, if they had them. I assume that there’s probably some black market value for Federation Credits just like any other currency, sovereign issued or not (you can buy aLewes Pound on eBay right now for $7.98). Perhaps its more legitimate and the Units are traded on a commodities exchange. It really doesn’t matter. As a Federation Citizen I can have gold pressed latinum, Federation Credits, Frangs, Darseks, Isiks, Leks, or Quatloos in my wallet. I can have a wallet. I can buy things with Self Sealing Stem Bolts if I want. But none of that is in conflict with the fact that the Federation has no unit of currency, has no money, and my society is predominantly concerned with societal good and self improvement.

Then there’s the matter of Quark’s bar. What’s up with that? He never seems to charge anyone for drinks, but is obsessed with money, and you can buy holosuites in latinum or Federation Credits, and you can bet on the Dabo table with Latinum. At first I thought there was a whole complex thing where Quark doesn’t charge Starfleet personnel because he made the mental calculation it was cheaper to give them drinks for free and keep accepting free rent from Sisko, but then I realized that doesn’t really work because he charges them for the Holosuites and Dabo tables. Then I realized: Quark’s is like any other casino. The drinks are free: they are a loss leader against the higher profits of the Dabo Table and Holosuites.

The Proto Post Scarcity Economy

The thing I love most about this theory is that it seems plausible for our future. Tom Paris said that a new world economy takes shape in the 22nd century. That might be a smidge optimistic but we already have a world economy, in one sense, so the new one could be something only incrementally different from this one. Money went the way of the dinosaur, he said, and Ft. Knox was turned into a museum. Most of us are already off the gold standard, and it’s certainly not inconceivable in another 180 years we don’t use paper money at all, and a single currency has dominated the planet — the Dollar is already close — and it slowly fades into the background.

From there, perhaps a cultural shift takes place as we realize that “everyone in a job” isn’t the same as a full economy, and we start to look for models beyond capitalism that aren’t all communist hoo-ha.

I sort of love that Star Trek forces us to think about a society that has no money but still operates with individual freedom and without central planning. I love that democracy is still in place. I love that people can still buy and sell things. It’s real. It’s a more realistic vision of post-capitalism than I have seen anywhere else. Scarcity still exists to some extant, but society produces more than enough to satisfy everyone’s basic needs. The frustrating thing is that we pretty much do that now, we just don’t allocate properly. And allocating properly cannot be done via central planning.

The only real “out there” requirement in all of this is a governmental layer higher than the nation, and indeed, higher than the planet. This doesn’t seem insane, I suppose, if we were to suddenly find ourselves not alone in the universe. And indeed we already have some measure of international government now. Moreover, the Federation clearly adheres to the “laws made as close to home as possible” routine, since as far as we can tell the Federation president really only has authority over Starfleet, Foreign Relations and power allocation and accounting. Virtually every other law we encounter in the Federation happens at the individual planet or colony level.

It’s interesting to me because these are things we’re going to have to reckon with, I believe, in my lifetime. If robots do all the dirty work, and the US is hugely rich, does every single person really need a job? Are we going to let all of that money pile up in the 0.1% ruling elite, or can it be distributed to everyone? Does wealth being distributed to the people in an equal manner mean communism absolutely? Of course it doesn’t. The US isn’t communist. The UK isn’t communist. Denmark isn’t communist. What happens when the surplus is more than enough?

Servo Layout Engine: Parallelizing the Browser // by Paul Rouget

Fri, 14 Feb 2014 01:59:01 -0800

Comments: "Servo Layout Engine: Parallelizing the Browser // by Paul Rouget"

URL: http://paulrouget.com/e/servopres/

During FOSDEM, Josh Matthews talked about Servo. You can find his slides here: joshmatthews.net/fosdemservo

Video:

http://mirrors.dotsrc.org/fosdem/2014/UD2218A/Saturday/Servo_building_a_parallel_web_browser.webm

10 Feb 2014

A new take on an old industry., Drop the ego and prove "I could make that so much better"

Fri, 14 Feb 2014 04:17:48 -0800

Comments: "A new take on an old industry., Drop the ego and prove "I could make that so much better""

URL: http://blog.lawnstarter.com/post/76623724133/drop-the-ego-and-prove-i-could-make-that-so-much

I mow lawns for a living, what do you do?

Well, if you’re working on a startup it probably sounds a helluva lot cooler than that. Whether it was my ego, or insecurity I came very close to never even starting LawnStarter simply due to the industry it was in.

After dropping out I was ready to work on something that sounded revolutionary. What I didn’t expect was to stumble upon an industry that needed a full system overhaul.

In the very beginning of working on LawnStarter I almost always told myself and other people that this was a segue to make some cash until my “real” startup began. I mean, come on, clearly I wasn’t going to make some lawn care company!

As weeks turned into months and progress became rather substantial I started to realize the only thing holding us back was my ego. I felt uncomfortable telling friends what we were doing (in my defense they gave me tons of shit), I undersold the opportunity, and worst of all I was honestly a bit embarrassed.

Looking back, it couldn’t be clearer that I had fallen into a mindset that is all too familiar in our community.

Why the hell not do a startup to try to change the lawn care industry? It’s a massive industry that could really use some help!

For some reason, I bought into this premium that our community puts on companies that produce gimmicky technology and raise massive rounds without having any real revenue streams. I got so caught up in this mentality that I was literally embarrassed to be working on a company that was making money in a less than sexy industry. Seems ass-backwards to me, and if I had simply ignored my ego for 10 minutes, I would have recognized how ridiculous I was for being embarrassed.

Over the past 5 months I’ve realized just how many opportunities there are to make really shitty industries better. People are willing to pay for a better experience, think Homejoy and Moveline (hopefully one day somebody puts Lawn Starter in that group). Even with money to be made and awesome companies to be built it seems to me like our egos and insecurities are holding us and progress back in dozens of not so attractive industries. Why not ride the wave of awesome customer experience and try to put your stamp on an industry that hasn’t changed in decades?

While everyone is trying to make the next facebook or snapchat - which is cool, I use both of these a lot - you could be trying to reinventing an industry nobody else wants to touch.

We all have a list of things or industries that “we could make so much better”. For me it was lawn care since high school. I’m finally putting my money where my mouth was and I look forward to seeing if I was right.

Me, I’ll be mowing lawns in Fairfax and dozens of other places. Where will you be?

-Steve

De La Soul to Make Entire Catalog Available for Free for 25 Hours | Music News | Rolling Stone

Fri, 14 Feb 2014 05:32:23 -0800

Comments: "De La Soul to Make Entire Catalog Available for Free for 25 Hours | Music News | Rolling Stone"

URL: http://www.rollingstone.com/music/news/de-la-soul-to-make-entire-catalog-available-for-free-20140213

De La Soul

Robbie Jeffers

February 13, 2014 4:35 PM ET

In honor of next month's 25th anniversary of their debut album 3 Feet High and Rising, De La Soul are making their entire catalog available for free download for 25 hours on the group's website. The download bonanza will begin on Friday, February 14th at 11 a.m. EST until Saturday at noon.

Check out 10 Unseen Photos Included in De La Soul's Album Downloads

"It's about allowing our fans who have been looking and trying to get a hold of our music to have access to it," De La Soul member Posdnuos tells Rolling Stone. "It's been too long where our fans haven't had access to everything. This is our way of showing them how much we love them."

The same things that made 3 Feet High and other De La albums so influential — its creative, if not fully licensed, use of a myriad of samples — has also prevented the group's work from appearing on many digital platforms. "It's been a trying journey," admits Posdnuos. "We've been blessed to be in the Library of Congress, but we can't even have our music on iTunes. We've been working very hard to get that solved." The rapper points to frequent personnel changes at record labels and hazy language in early contracts that have led to long delays in properly clearing the group's catalog.

The release of the group's catalog is the first of numerous upcoming projects. In a few weeks, they'll post new songs to their site, with You're Welcome, their first album since 2004's The Grind Date, expected to be released before summer. Next month will also see the release of Preemium Soul on the Rocks, a six-song EP with three beats each from DJ Premier and Pete Rock. The group is also planning a visit to Detroit to work on an unreleased beat from J Dilla, the prolific producer who passed away in 2006. "Dilla was the Tupac of producers," says Posdnuos. "He has so many unreleased things that no one has heard. His family knows how vital and important an ingredient his music was to our work."

Where does 3 Feet High and Rising rank on Rolling Stone's 500 Greatest Albums of All Time?

Asked about the status of You're Welcome, an album originally scheduled for release last year, Posdnuos says that it's "coming along amazingly," but points to self-criticism in its delay. "We have tons of music, but we're our own worst critics," admits the rapper. "Certain groups have too many 'yes men.' In our group, we have too many 'no men.' When we look back on some of the stuff we have, we're like, 'Yo, we need to just put this out.' The album is still called You're Welcome, but we also have this whole other album that we're working on that…Wooo, I wish I could talk about it."

Twenty-five years into their career, the group is ready, if somewhat cautiously, to adopt the more-is-more release schedule of its younger peers. "We're just getting in the mode of constantly giving people new music," says Posdnuos. "I'll be the first to say that not everyone can do it. You can put out a new mixtape every week, but it can dilute what you're putting out because you haven't had enough time to see what's going on with your life to write something from a different angle. With us, we've sat a long time without releasing an album. It's high time we start releasing a bunch of stuff because it's there."

To read the new issue of Rolling Stone online, plus the entire RS archive: Click Here

Mark Shuttleworth » Blog Archive » Losing graciously

Fri, 14 Feb 2014 05:32:23 -0800

Comments: "Mark Shuttleworth » Blog Archive » Losing graciously"

URL: http://www.markshuttleworth.com/archives/1316

Friday, February 14th, 2014

With Bdale Garbee’s casting vote this week, the Debian technical committee finally settled the question of init for both Debian and Ubuntu in favour of systemd.

I’d like to thank the committee for their thoughtful debate under pressure in the fishbowl; it set a high bar for analysis and experience-driven decision making since most members of the committee clearly took time to familiarise themselves with both options. I know the many people who work on Upstart appreciated the high praise for its code quality, rigorous testing and clarity of purpose expressed even by members who voted against it; from my perspective, it has been a pleasure to support the efforts of people who want to create truly great free software, and do it properly. Upstart has served Ubuntu extremely well – it gave us a great competitive advantage at a time when things became very dynamic in the kernel, it’s been very stable (it is after all the init used in both Ubuntu and RHEL 6 and has set a high standard for Canonical-lead software quality of which I am proud.

Nevertheless, the decision is for systemd, and given that Ubuntu is quite centrally a member of the Debian family, that’s a decision we support. I will ask members of the Ubuntu community to help to implement this decision efficiently, bringing systemd into both Debian and Ubuntu safely and expeditiously. It will no doubt take time to achieve the stability and coverage that we enjoy today and in 14.04 LTS with Upstart, but I will ask the Ubuntu tech board (many of whom do not work for Canonical) to review the position and map out appropriate transition plans. We’ll certainly complete work to make the new logind work without systemd as pid 1. I expect they will want to bring systemd into Ubuntu as an option for developers as soon as it is reliably available in Debian, and as our default as soon as it offers a credible quality of service to match the existing init.

Technologies of choice evolve, and our platform evolves both to lead (today our focus is on the cloud and on mobile, and we are quite clearly leading GNU/Linux on both fronts) and to embrace change imposed elsewhere. Init is contentious because it is required for both developers and system administrators to understand its quirks and capabilities. No wonder this was a difficult debate, the consequences for hundreds of thousands of people are very high. From my perspective the fact that good people were clearly split suggests that either option would work perfectly well. I trust the new stewards of pid 1 will take that responsibility as seriously as the Upstart team has done, and be as pleasant to work with. And… onward.

This entry was posted on Friday, February 14th, 2014 at 1:01 pm and is filed under ubuntu. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Comments are filtered through Akismet for spam detection. Please follow the Ubuntu Code of Conduct - your opinions are welcome but please keep them polite and constructive.

Schneier on Security: The Insecurity of Secret IT Systems

Fri, 14 Feb 2014 06:50:38 -0800

Comments: "Schneier on Security: The Insecurity of Secret IT Systems"

URL: https://www.schneier.com/blog/archives/2014/02/the_insecurity_2.html

A blog covering security and security technology.

« GOPHERSET: NSA Exploit of the Day | Main | My Talk on the NSA »

February 14, 2014

The Insecurity of Secret IT Systems

We now know a lot about the security of the Rapiscan 522 B x-ray system used to scan carry-on baggage in airports worldwide. Billy Rios, director of threat intelligence at Qualys, got himself one and analyzed it. And he presented his results at the Kaspersky Security Analyst Summit this week.

It’s worse than you might have expected:

It runs on the outdated Windows 98 operating system, stores user credentials in plain text, and includes a feature called Threat Image Projection used to train screeners by injecting .bmp images of contraband, such as a gun or knife, into a passenger carry-on in order to test the screener's reaction during training sessions. The weak logins could allow a bad guy to project phony images on the X-ray display.

While this is all surprising, it shouldn’t be. These are the same sort of problems we saw in proprietary electronic voting machines, or computerized medical equipment, or computers in automobiles. Basically, whenever an IT system is designed and used in secret – either actual secret or simply away from public scrutiny – the results are pretty awful.

I used to decry secret security systems as "security by obscurity." I now say it more strongly: "obscurity means insecurity."

Security is a process. For software, that process is iterative. It involves defenders trying to build a secure system, attackers -- criminals, hackers, and researchers -- defeating the security, and defenders improving their system. This is how all mass-market software improves its security. It’s the best system we have. And for systems that are kept out of the hands of the public, that process stalls. The result looks like the Rapiscan 522 B x-ray system.

Smart security engineers open their systems to public scrutiny, because that’s how they improve. The truly awful engineers will not only hide their bad designs behind secrecy, but try to belittle any negative security results. Get ready for Rapiscan to claim that the researchers had old software, and the new software has fixed all these problems. Or that they’re only theoretical. Or that the researchers themselves are the problem. We’ve seen it all before.

Tags: air travel, disclosure, economics of security, obscurity, secrecy, security engineering

Posted on February 14, 2014 at 6:50 AM • 28 Comments

To receive these entries once a month by e-mail, sign up for the Crypto-Gram Newsletter.

In Europe they use quantum key cryptography in their voting systems:

http://www.idquantique.com/news-and-events/...

In Europe, the struggle to create a public realm out of the monarchy's private government extends back to populist movements in the Middle Ages such as the Ranters and Diggers and Bretheren of the Free Spirit; they seem more likely to view their government as something that really belongs to them, with the potential to work for them.

In the United States, our struggle to create a public government really begins with the 14th Amendment. Between then and the civil rights era is when we obtained universal suffrage. The franchise was highly exclusive in the Revolutionary era -- so much so that "WE, the People" probably only represents the will of 5-7% of the population at the time. Since then, the conservative battle cry has been "smaller government" and "privatization." Notwithstanding that we had private government once before -- when we were owned by the Britain -- we have this myth of obtaining self rule by fighting tyranny, when, in fact, the road to self rule has been a much more complicated struggle. But the myth prevails over history.

The 522B is ancient - at least 5-10 years old. http://aerodetection.com/rapiscan-522b/ says the units it has are 8-10 years old.

I'd be much more interested in the Rapiscan 620DV which appears to be the model used at major airports in Europe.

I still remember the first time I ran smack into security-by-obscurity. My boss at the time put me in charge of the most secure system we had at work. It was kept behind a heavy locked door and I was admitted only after a strong lecture on how important it was to keep it as secure as possible.

Even behind a closed door, he felt it necessary to whisper: "and the password is 'secret', which of course we can't tell anybody and they would never guess..."

The idea of continual improvement driven by the hostile nature of the operating environment seems very similar to evolution in the natural world. I don't mean analagous to, I mean another form of evolution, survival of the fittest. Obscurity, obfuscation, and political lobbying are all attempts to exclude a product from the security evolutionary process but even at the very highest levels - government printed currency, NSA information - it is impossible to isolate anything from security evolutionary forces.
It is better to embrace this process and to continually iteratively evolve and grow stronger and wiser during the process than to try and hold the driving forces of evolution at bay. The driving threat forces of security evolution themselves evolve, adapt and get stronger over time. You can hold them at bay temporarily through obscurity and obfuscation, but it then become only a matter of for how long? e.g. Sony Playstation 3
The Playstation 3 is an interesting example in that it was finally cracked due to sloppy cryptography implementation ( what was meant to be a random number generator was implemented as a constant ). I say interesting because peer / open review would have exposed the obvious flaw and it would have been fixed. In this case it was the obscurity and obfuscation that ultimately was responsible for the security being broken.
https://www.schneier.com/blog/archives/2011/01/sony_ps3_securi.html

OK, so if Kaspersky or Symantec or TrendMicro or Norton doesn't hand over all their source code I shouldn't buy the product... right?

OK, go ahead and post all your passwords and SSN's and your credit card info.

@beatty Well ... yes, although not just because you don't have source code. Virus scanners typically only search for known exploit code (and perhaps a few variations on it). They're fundamentally reactive and in my view not worth the effort.

@vincent You jest. The difference between secret paswords and secret algorithms has been explained to death already.

@vincent: not really. That is yours, your privacy for your own usage/protection, but when somebody offer for public usage some kind of security product, it should be available for public scrutiny on potential security threats/weaknesses and invasion of privacy (like recently Samsung on smart TV agreed to provide led indicator when camera is on. I hope that activation is hardware, not software).

The meta-problem here is that many (in my experience most) software "engineers" are incompetent and do not qualify as engineers. The result is that the typical software system sucks badly.

This machine is just a standard example. However though Windows (no matter what version) was suitable as an embedded OS has no business working on software or surrounding systems.

@beatty -- you shouldn't buy their products if you don't trust them. I don't and the systems my family uses haven't had an problems. But they don't hide what they're doing and they have some pretty vigorous competition.

@vincent -- I hope you don't think there is anything especially "secure" about your SSN. But hiding secret information used to access a system is different than hiding or obscuring information about the the system itself, which is what this post is about.

No, this just another cherry-picked instance of failed product development that is being used for self-serving purposes. It's easy, isn't it, to swoop in after the fact and point out everything that went wrong. Do you even know what the original threat model was? For all you know this failure was the RESULT of anal security engineers that suffocated product development until the project collapsed. You don't know. You pick up on all this pop news junk and fling it anyway you want.

Oh, it gets better. I keep thinking that if you control the software, you can probably control where the X-Ray beam is at. And observe that TSA agents walk back and forth through the scanner all the time.

So you keep it ON bouncing back and forth scanning at roughly chest level when not actually scanning the full body. When you see something metallic (say, a TSA badge), you immediately drop the emitter down to crotch level for the next 10 seconds...

"For all you know this failure was the RESULT of anal security engineers"

Win98 and plaintext passwords are not the result of anal security engineers. Unless, of course you mean actual anal security engineers, in which case it is not surprising, as they only know stuff about how to use a variety of rubber corks.

I somewhat disagree that engineers try to hide their bad designs through obscurity. I don't believe they even think about security.

There seems to be this idea of "Well, why would anyone attack that?" that is prevalent far too often. It's why we have empty passwords on internet-facing SCADA stuff, hopelessly outdated operating systems that can't be updated on embedded systems, etc. Nobody thinks like attackers. Closed systems are perfectly acceptable to people who don't think they will ever be a target.

It would be like me inventing my own door lock, and because no one has seen one before, I can assume I'm protected. (Of course this could also be said - Since no one has attempted to break into it yet, I hope I'm protected)

Rather than the alternative

Buying a door lock that has been proven in the real world. Paying attention to security bulletins so that if an exploit is found, I can replace it with a fixed version.

@vincent - Either way, I'm not giving you my key.

@Jason. All you said is valid when you are random target making you more protected than next target in the phishing scheme. Just to bring some relax mood: "Two young ladies were in the jungle and spotted lion. One start running, another asked is she really could run faster than lion. Nope, she reply. Just faster than you..." If you are NOT random target, all depends on the actor's resources available to break your security (psychical or informational): local thugs, organized crime, LEA local or state, LEA federal, foreign agents, etc.

@vas pup:

Well, now we're just getting deeper into security concepts. I don't think the threat of a targeted attack is a reason to abandon tried and tested methods. I think it's a reason to bring in additional expertise, add some additional layers of protection. (Instead of just a good door lock, add a surveillance system, alarm system, maybe a stronger door, bars on windows). The security system you use, no matter what you're protecting has a cost that must be weighed against the risk, and to be effective almost certainly will be layered.

@ Bruce,

The truly awful engineers will not only hide their bad designs behind secrecy, but try to belittle any negative security results

That statment is a little unfair, because when it comes to hardware the closer you are to the metal, generaly the more competent you are as an "engineer".

The problem generaly starts and ends with managment, because,

1, Like quality, security has to be there fully functional from project day 0.

2, Security processes, training etc "cost".

You have to be an "old engineer" to remember the days befor quality processes were considered part and parcel of the job. And unfortunatly the area quality processes are least frequently found is "software engineering". Just take any modern software methodology and find the bits that are actually about "Quality Assurance"...

The simple answer is all you will find is an illusion or mirage paying lip service to any real quality process. It's also the reason grizzled old vetrans of software coding will tell you that most software development methodologies are at best "make work" and that you will get better results where team members share a common non adveserial goal and thus trust each other.

And when you look back at the development of QA systems it was the teams who bought into it and trusted the others that the most benifit was seen.

The reason QA actually got going was two fold,

1, Managment saw the financial benifit before the factory door.

2, Those who saw benifit used QA as a part of purchase decision.

Neither of these conditions is true currently for "security" thus managment treat it as "a non productive inefficiency" and thus "managment mantra" says it should be ruthlessly expunged from the work process "to increase productivity"

The way to get security into the design process as a norm is by making having it the most profitable path to walk, that way as with QA "managment mantra" will change.

Untill that time blaiming other people for "keeping their jobs" is a little unfair.

I think the article is dead wrong about the threat projection system being a big issue.

The purpose of this system is to keep the screener alert. In a normal airport, a contraband item like a bomb, gun, etc. might occur at most once a day. Rare contraband like a bomb is probably less than once in a lifetime. Hence it would be natural for a screener to simply 'pass' all luggage, even if they are being diligent. Adding these "false positives" gives the screener something to do, and increases security by "impedence matching" the task at hand to the psychology of the operator.

It's true that an attacker could have the system inject innocuous items, or perhaps have it inject items at a very high rate. I suspect that either of these new behaviors would be quickly noticed.

Actually, in any airport, contraband like this would occur at most 24hrs/airport-lockdown-time per day.

Looks like I was wrong. The other article gives more details about the system, and it is pretty crappy.

It's one thing to superimpose false images that are removed after alarming on them. It's another entirely to allow some other person to choose the time when the false image will be shown, and to replace rather than modify the image.

Hacking is illegal. Selling crappy secured soft- / hardware isn't (wearing my black and white glasses now).

"Upon seeing a weapon on the screen, operators are supposed to push a button to notify supervisors of the find. But if the image is a fake one that was superimposed, a message appears onscreen telling them so and advising them to search the bag anyway to be sure. If a fake image of a clean bag is superimposed on screen instead, the operator would never press the button, and therefore never be instructed to hand-search the bag."

If the training software assumes that the .bmp images have simulated contraband, one would think that the training software would do something if the operator doesn't press the button when a .bmp is displayed. Or does the attacker who introduces a "clean" .bmp file also modify the software?

Thank you, Bruce... "Obscurity means insecurity" is exactly what I've always meant, when I said "closed source by definition is insecure".... only open source can be secure (which doesn't guarantee that it is, only that it's at least possible).

What really worries me is that we haven't really learned a lot.

The 1983 movie wargames could happen today. Maybe not in the US (although I doubt that), but there are more countries in the world. The problem with security by obscurity is that you just don't know whether there is a WOPR that has a backdoor with the login "Joshua".

How secure are these nucleair platform systems? Just look at the stoxnet virus. Is "the west" capable of protecting itself against this kind of things? I don't think so. (looking at this news item)

And is the JSF/F-35 capable of dropping a nuke? It also contains 20 mln lines of C++ code.

I think this is way more worrying than any "terrorist attack".

I don't know. Maybe it's just BS that I am talking about. I am not a security expert. But I do know that you can't trust computers. Not yesterday, today or tomorrow.

Bruce,

Saw you at SAS, thanks for speaking!

Did you notice on the way out that all the machines in the Punta Cana airport were the make and model Billy and Terry evaluated?

Mike

Your comment about engineers (smart vs awful) was unfortunate. Well-established companies such as Diebold produce software with a workforce that is salaried and university-educated. The software produced usually conforms to management's priorities. If QA isn't isn't part of the software process, the software produced will tell the tale. The company with good management and a weak engineering staff is a rare beast. Unicorn rare. If Diebold has crappy software, then Diebold is to blame, not some mythical bumbler.

To say nothing of the procurement process.

True of Diebold and Rapiscan!

Schneier.com is a personal website. Opinions expressed are not necessarily those of Co3 Systems, Inc..

IP Address Details - ipinfo.io

Fri, 14 Feb 2014 08:30:18 -0800

Comments: "IP Address Details - ipinfo.io"

solo.im - a single founder peer group

Fri, 14 Feb 2014 08:39:18 -0800

Comments: "solo.im - a single founder peer group"

Stephen Law: How the US Treasury imposes sanctions on me and every other "Stephen Law" on the planet - my letter to OFAC

Fri, 14 Feb 2014 12:26:16 -0800

Comments: "Stephen Law: How the US Treasury imposes sanctions on me and every other "Stephen Law" on the planet - my letter to OFAC"

URL: http://stephenlaw.blogspot.com/2014/02/how-us-treasury-imposes-sanctions-on-me.html

Right, here's another thing I am getting off my chest - email letter to OFAC (edited slightly from version sent).

Dear OFAC

This correspondence is copied to my UK Member of Parliament The Right Hon. Andrew Smith. Please copy him into your reply.

My name is "Stephen Law". The name "Stephen Law" appear on OFAC's "specially designated nationals" list:

Here is the actual OFAC listing for "Stephen Law", alias of "Steven Law"

https://ofac.data-list-search.com/Entities/ByName/steven-law

This person is Burmese and is suspected by US Treasury of drug trafficking. He is the son of Lo Hsing Han (dubbed by US Treasury as "The Godfather of Heroin") and has a Singaporean wife. His addresses, as listed by you, are all in Burma and Singapore. None are in the UK.

I have discovered that, as a result of this listing, US Customs block shipments of goods to me here in the UK. Also when people try to wire me money from abroad (not just from the US, but from anywhere), for e.g. occasional travel expenses for academic conference attendance, the payment is interrupted and various checks are made before the funds are released. This became so bad during one period (a series of payments every single one of which triggered a block) that I had to switch to a different bank account. At no point was I told why this was happening (i.e. that you, OFAC, are responsible). The banks concerned believe they must keep this information from me (I was told this by my bank branch). Hence it took me many months to figure out what the source of the problem was: OFAC/US Treasury.

It appears any "Stephen Law" anywhere in the world will suffer this same treatment, as indeed will anyone who merely happens to have the same name or alias as one of your "specially designated nationals". This has proved frustrating, time-consuming and also costly to me personally. E.g. I have paid US$77 postage for goods it turns out I can never receive because they are returned by US customs to the US vendor because my name is listed. As a result of the OFAC listing, I cannot now order goods from - or receive gifts from friends and relatives in - the United States.

Can you inform me: given I am very obviously NOT the Burmese Stephen Law:

(i) how I can avoid having all goods shipped to me from the US to my UK address being blocked and returned to sender by US customs?

(ii) how I can avoid my own bank repeatedly asking me who I am (and requesting information including my DOB, which they already possess) before unblocking any payment from abroad?

My bank knows who I am, and they know I am not the Burmese "Stephen Law" on the specially designated nationals list, but still I have to go through this same rigmarole every single time money is wired to me. How do I avoid this please?

Yours faithfully

Stephen Law

PS Ofac-caused delays to payments to me can run into weeks. On one occasion I ran up overdraft charges as a result of not receiving funds blocked by OFAC.

PPS I was interviewed by Foreign Policy magazine about all this a short while ago.Also interviewed by News Hour on BBC World Service.

Why I Dropped Out Of YC | wikichen

Fri, 14 Feb 2014 17:52:28 -0800

Comments: "Why I Dropped Out Of YC | wikichen"

URL: http://wikichen.is/writing/why-i-dropped-out-of-yc/

I bought my first Moleskine by sheer coincidence while shopping for craft supplies that a design course had required; I had avoided buying a notebook for the first two years at university by resourcefully (for which some might argue was born out of necessity from the lazy frugality of your average college kid) removing stacks of printer paper from the student center under a well-rehearsed, convincing ruse that I was somehow heroically fixing a “paper jam”.

By my third year I was making enough money from being a resident advisor and scrapping by with a delightful hodgepodge of research jobs that I could finally renounce my thieving ways and afford some decent stationeries, one of which was a proper notebook on which I could keep notes. I’m going to be taking actual notes in an actual notebook like a Shakespearean boss, I thought as I browsed the notebook section in the art supplies store. I picked the most conventional looking one from the pile: a large Moleskine Classic Notebook with a hard cover and plain, white pages. I knew not what a Moleskine was then or of its more hipster connotations (I swear) or why it cost an arm and leg for one, but I was pleased to have something I could, at long last, take notes on that’s not single sheet and A4-sized.

Naturally, those notes turned out to be nothing but the wild swirling caricatures of a sleep-deprived procrastinator nodding off every few minutes as the professor droned on. But over time that first Moleskine served as an outlet for my creative self, a blank canvas I painted with my thoughts and ideas and dreams. It captured the moments of blinding clarity and paroxysms of fleeting epiphany that so often defined checkpoints of my personal growth. More importantly, the notebook embodied me at my best and dutifully provided a record from which I could reflect and study.

I filled up that very same Moleskine at the end of last year, around the time when I decided to join the startup as she made the transition to the Bay Area after being accepted into Y Combinator. The fortuitous signs of a full notebook greeting a new opportunity in a new year got the best of my superstitious tendencies, so I ordered a brand-spanking new one off Amazon. In the five or so weeks since Y Combinator commenced in full swing, I haven’t written anything in the Moleskine. In fact, it’s still sitting on my shelf, wrapped in its original packaging.

Earlier this week we met with our advisor and YC partner Aaron over what seemed to be a divergence in the direction of the company and the underlying disagreement on how to best solve problems. In the hour we talked, what surfaced instead was an irreconcilable working relationship rooted in fundamental differences in personalities and misaligned expectations of roles and responsibilities. There were no hard feelings, just hard lessons learned.

Maybe I signed up for the wrong thing or for the wrong reason, as well-intentioned as I was from the start. Maybe it was more about Y Combinator, the subject of numerous pages in my Moleskine, and that was wrong. Maybe in the wake of a failed startup I deluded myself into thinking great teams are predicated on great friendships, a truism that no longer holds absolute truth.

It was also the acceptance that the startup would never be something I could call my own, not just in stake but in equal say, that cemented my decision to walk away. A sense of ownership can be a powerful motivator for a startup; in this case the lack thereof tore it apart. But if you asked me for the one reason why it didn’t work out, the truth is that it just wasn’t fun anymore, and when you’re grinding away on an early stage venture with no pay, the enjoyment you derive from working with your partners is all you have.

Egos clashed and convictions shaken, we came to the consesnsus that it’d be best if I part ways with the startup. To some extent I was relieved, freed from an incompatible environment that brought no party joy, and it hardly mattered that, to those like me who held the institution on a pedestal, this is Y Combinator we’re talking about. Aaron put it this way, “Stop thinking about YC. YC is an artificial construct, a forcing function that augments a startup’s chances to succeed. But if you can’t work together, then none of it matters. If someone tells me I can do YC but I’ll have to wear hot pants the whole time, I wouldn’t do it.” And I wholeheartedly agree with him: life’s too short to wear hot pants.

It’s been a few days since my departure, and I’m as uncertain as ever regarding my next steps, largely given that I’m still digesting what I’ve learned from this experience, and partly because I’ve grown jaded and disillusioned with Silicon Valley at large. What’s certain is that I won’t be touching startups for a very long time, and I might even stay away from Hacker News for a while. If anything, when the time comes and that startup itch manifests itself again, I’ve taken to heart that it will be of my own creation, with a team I can entrust equal ownership, and on my own terms, Y Combinator or not.

Taking the new Moleskine from the shelf and removing it from its plastic wrap, I turned to the first blank page and picked up my pen.

Here we go again.

Hacker News 100

Chris's Wiki :: blog/linux/SystemdWhyItWon

5 Ways To Burn Out Programming

1. Think about your project and only the project

2. Have a negative attitude toward everything.

4. Switch jobs often

5. Work long hours, ignore your life

Summary

The Day the Internet Didn’t Fight Back

IBM layoffs strike first in India; workers describe cuts as 'slaughter' and 'massive' :: Editor's Blog at WRAL Tech Wire

IBM layoffs strike first in India; workers describe cuts as 'slaughter' and 'massive'

Hemingway

I challenged hackers to investigate me and what they found out is chilling | PandoDaily

Introducing Wit Speech API

Turning speech into actionable data

How does it work?

Consuming the API

FlapMMO -- flapmmo.com

Chris Hates Writing • The anonymity I know

The anonymity I know

Elasticsearch.org 1.0.0 Released | Blog | Elasticsearch

Rails 4 Engines - TechRabbit

Rails Engines

Versus Many Apps

Versus Single App

Engine Usage

Admin

Shared Code

API Server

Strategies

Migrations and Models

Admin

Assets

Routes

Tests

Memory

Folders and Files

Interaction Between Engines

Summary

Scientific method: Statistical errors : Nature News & Comment

Out of context

What does it all mean?

Numbers game

CandySwipe Open Letter to King regarding trademark.

Scaling Asana.com - Asana Engineering Blog

Everyone learns Markdown; all content is in Markdown

We put the Markdown in Statamic, a file-based CMS

We make our changes in Github

Our staging server is realtime

The work in Github is linked to Asana

Before deploying, we test the site with CasperJS

Deploy daily

Saying Goodbye To Python

Scientists Say Their Giant Laser Has Produced Nuclear Fusion : The Two-Way : NPR

Obama Sued by Rand Paul Over Phone Surveillance Program - Bloomberg

Found Legal

‘Minimal’ Usefulness

D.C. Decision

Evan M Rose. - Last Thursday, my brother, Stephen Rose fought...

robsheldon.com: Hello, world.

I have a theory

Hueman, how are you?

Apple,Have a Little Huemanity

Comcast Acquiring Time Warner Cable in All-Stock Deal Worth $45 Billion - WSJ.com

The Facebook Comment That Ruined a Life - Page 1 - News - Dallas - Dallas Observer

Dug.js — A JSONP to HTML Script — Rog.ie

Dug.js Setup

Dug.js Parameters

Download Dug.js

License

Why Your Python Runs Slow. Part 1: Data Structures – Blog ~ Saulius Lukauskas

Technical Details Behind a 400Gbps NTP Amplification DDoS Attack | CloudFlare Blog

NTP Amplification 101

Not Just Theoretical

Globally Distributed Threat

Time to Clean Up the Problem

Making Remote Work Work - Learning - Source: An OpenNews project

Christopher Groskopf’s Tricks for Going to the Office without Going to the Office

Staying organized

Staying in communication

Apple,
Have a Little Huemanity