David Hu

Khantemplations: A Khanpilation of thoughts on life and work

2015-01-14T00:00:00-05:00

From:	david@khanacademy.org
To:	Khan Academy Team <team@khanacademy.org>
Date:	Sat, Aug 9, 2014 at 2:23 PM
Subject:	Say what you wanna say, and let the words fall out

Dear colleague,

Hi, I’m David. I just graduated from college earlier this year, and I just started full-time a month ago.

As a student, life has always been structured with a clear next step and short-term goals. Get good grades. Get into university. Get a job.

But what now? I have no one to answer to now. No one to enforce rules or set expectations. No clear directions or structure.

Should I work hard at work? Make new friends or maintain existing ones? How do I find a soulmate? Should I climb our corporate breadmaster hierarchy ladder, and strive for the coveted Assistant Breadmaster title?

In the process of figuring out priorities and structuring my life, I’ve found our 1:1s and AMAs very helpful. But I want to know more about everyone here, and so I have a few questions for you:

What is one thing you wish you knew when you first started at Khan Academy?

What is one thing you wish you knew when you first graduated?

Is there anything you think you did right? What worked well for you?

What advice do you have for new grads or interns?

… and anything else that you want to say.

Whoever you are — intern or full-timer, junior or senior, young or 26-and-married, developer or content producer, De-Facto Software Kingpin or Director of Wellness, human or dancer — I want to hear from you! I would love to get a broad range of differing perspectives.

Be specific or general. Talk about work or not. Anything from “Small and frequent code reviews” to “Announce ‘bread’ loudly because people are wearing headphones” to “If you’re not happy, be happy.”

Say a little or a lot. Maybe just one cryptic word to puzzle over, or make this your magnum opus. Whatever it is, I will read it and think about it.

I also think this would be a great opportunity to share with everyone — not only the new grads and interns among us, but also existing and future employees. So, I’m planning to compile this into an internal publication for everyone’s edification. By default, I will anonymize your contribution. But,

If you use the codeword fearless somewhere, I will attach your name next to your thoughts.

If you don’t want me to share your thoughts at all with anyone, use codeword bread.

If you’re allergic to emails or have more to say, I’d be happy to have a 1:1. Respond with codeword Tea-Rex.

If we’ve already had this conversation before, reply with the standard company codeword for this situation, tortoise.

Thank you so much. I, and I hope others, will appreciate this.

In the words of the highly respected CEO, educational philanthropist, and role model to many of us here,

I don’t think you should wait. I think you should Speak Now.

(Note: The spinning hand-tree asterisk thing denotes an inside joke at Khan Academy.)

So I sent the above annoyingly long email to err’body at work. And they didn’t fire me — far from it, my colleagues turned out to be really helpful people:

Received responses from 25 individuals
Responses with codeword Tea-Rex: 2
Responses with codeword bread: 2
Responses with codeword fearless: 11
One response used every codeword (I ended up cancelling out bread with fearless and went on a 1:1 with this mischievous individual)
Responses from young and old, intern and full-timer, and across departments

Khantemplations Khanpilation

I had enough responses that I decided to compile this into a book made from dead trees. To add insult to injury to my environmentalist friends, I ordered 150 of these (to distribute. I promise I’m not smoking them):

Here are the table of contents:

And here’s a photo of one of the section dividers because I like how fancy they look:

Thanks to Marcos for knowing all the things about printed book design — from suggesting Adobe InDesign for layout to giving me ideas for cover designs. I used BookBaby for the printing.

Butchered paraphrasing and merciless editorializing of responses

I don’t want to distribute this book publicly, because I told my colleagues this would be distributed for internal publication only (and I love them). So instead, here is a summary with anonymized quotes mixed in (I hope. If I leak PII, you are free to sue me). It’s a hodge-podge of advice, regrets, and lessons from my co-workers. By condensing it into a dry list of bulleted points, I've managed to suck out all the anecdotes that once weaved each response into a beautiful flowing narrative with story and character. Well beauty and soul be damned — I'm an emotionless engineer!

My proofreader friends encouraged me to actually de-wall-of-text it. So I did. Thanks to you, I’m no longer emotionless and am now capable of love.

1. Take risks, make mistakes, and learn from them

This is the single most unanimous and uncontested life advice from my colleagues. Take chances, go on adventures, be bold, and be outgoing, especially when you’re young, energetic, and unburdened:

“Take risks — it will only get harder to take them the older you get.”
“I wish I could have been less nervous and more outgoing in work and social life. There’s not much to gain by being shy.”
“Taking chances always pays off. Tackle unknowns, not knowns.”

It’s not always rainbows and butterflies. And you’ll make mistakes. But do it anyway, because it is through mistakes that we learn the most.

“Don’t be afraid to struggle through difficult experiences.”
“Oh and don’t be afraid to make mistakes. Go looking for mistakes.”
“It is okay to make mistakes. In fact it is preferable.”

I hear this very often, but it takes courage to take risks:

“I live my life by two rules: 1) Do the right thing. 2) Do the courageous thing. Courage means doing something in spite of our fears. Because we all have fears, and not being afraid of something isn’t hard. Being afraid of something and doing it anyways is.”

It’s all too easy to do what we’ve always done before. Go home. Play video games. Facebook stalk. Catch up on /r/taylorswift. (Wait… what do you mean not everyone does this?) Having lived in the real world for only 7 months, I’ve found that doing things outside of my comfort zone — flying to a random desert with a bunch of lunatics I just met who wear onesies through the airport (I love you all) — have allowed me to grow as a person.

2. Invest in people

“One of the biggest takeaways from my year abroad is to value and prioritize my relationships more.”

So much of finding a career is just knowing the right people.

“I wish I knew the value of a network. Jobs, internships, and general career opportunities are so heavily based on who you know and how you can get in.”

Multiple respondents expounded on the importance of mentorship in growing at work.

“Find role models and mentors. Invest in good people, personally and professionally. Be proactive in this — reach out.”

But also make an effort to reach out to people outside of work. For one coworker, work consumed their conversations, social activity, and life. And then,

“When it came time for that job to pass, and for me to go join something else, I felt a very strange hole in my identity.”

Transitioning from school to work means transitioning from seeing classmates everyday to maintaining those friendships, but is also an opportunity to make new friends:

“After college, maintaining friendships takes a lot more effort. You have to actually schedule things! … Just accept the extra overhead because it’ll be worth it when you look back.”
“For the first 3 years out of school, I formed a lot of close friendships that still last today. We were all single, enjoying independence, and learning a lot about growing up — work, money, dating, traveling, cooking, etc.”
“Your health, the health of the people you care about, and the relationships you have with your close family and friends are the only things that really matter.”

3. No, really, invest in people

People and relationships were discussed in so many different ways that I split this into two points! (But really, it’s so I can title this blog post “10 Things You Must Know After You Graduate Or Be A Regretful Miserable Lonely Wreck Forever” if I wanted to.)

This section is more about how and what you can do.

Perhaps the best way to grow is by surrounding yourself with the sort of people you want to become:

“Find the role models who can be your true mentors, and do your best to work with them.”
“Don’t wait for people to come to you. If you see a person you really respect or is doing something cool or seems like they have their head on straight or could be someone you can lean on (or could lean on you) reach out to them! Build the bridge yourself and make the first move.”

Love.

“Care and be helpful. You will never (NEVER) regret going out of your way to do something helpful to or nice for someone else. You will regret times when you could have gone out of your way and chose not to.”
“I tried to have lots of love and respect for those around me.”
“Hone the art of really good listening.”
“Tell your parents you love them.”

Stay in touch and just spend time:

“Identify the bright and kind people in your life, and make sure you stay in touch with them.”
“Keep a list of the contacts you value; use recurring tasks to ensure you keep in touch; learn from them and network through them; return the favor.”
“One of my favorite things was a weekly supper club where we’d all get together and try to cook a home cooked meal.”

Put yourself in different social circles:

“Finding a soulmate: There’s no science to this. Lots of my friends are in happy relationships through friends. Just have to make time and space for this to happen. Surround yourself with interesting people you want to be around, and put yourself in different circles.”

But, at the end of the day, you have to accept that nothing is permanent:

“People around you will come and go. People around you will change and you will too.”

4. Make yourself happy, because impressing your grandmother won’t

Listen to yourself and do what makes you happy. Make happiness something you control, and not something dictated by the approval of others:

“You are the only one you have to impress and be accountable to at the end of the day. An amazing resume, a prestigious job, a diploma from the best school, stays at the coolest hotels, first class tickets, basically anything that is about impressing others will not necessarily make you happy and satisfied.”
“Don’t ever sacrifice your happiness or sanity for money, power or some ill-fated glory. All that stuff fades away and at the end of the day you’re left with yourself.”
“Don’t make decisions based on how you think you’ll be perceived by others.”
“I followed my heart and went with my gut, rather than being influenced by others’ opinions.”

How to be happy? I don’t know. But here are some ideas:

“Take some time off after graduation to just focus on yourself. Do those things you love. I travelled, spent time with friends, and played guitar. Reevaluate things.”
“Live and work in the moment.”
“Practice gratitude every day.”

I do the latter by writing three different things/people I’m grateful for in my journal each day.

5. Work hard, take a break, work hard

Many of my colleagues credit where they’re at now to hard work:

“Guard your time for creating things. This doesn’t mean sacrificing family or love — there are lots of other things people waste time on.”
“Work hard. Be present and proactive. Do not underestimate how much impact you can have on the world if you consistently work hard. Consistently working hard will have more impact than any single decision.”
“I worked (and still work) really hard. That’s a hard thing for a lot of people to do, to actually commit yourself 100% into something no matter what the pay, outcome or gain for you is.”

But what does it mean to work hard? Work 25 hours a day? Clone yourself? Blast Taylor Swift while you work?

Perhaps, focus only on the important things and drop the rest:

“You — and only you — are in charge of your own attention and focus. They are very finite resources which you spend every second. Spend them with care.”
“If what you are doing is not important, and if you don’t think it is going to lead to something important, then why are you doing it?”

And maybe you’ll be able to work hard in the long run by not working hard all the time:

“At past jobs I have worked too much and it led to burnout and feeling pretty miserable. I’ve found that by just working hard when I can and not being upset at myself for when I can’t, I can keep a much more sustainable and productive cadence.”
“If college is a series of sprints, life after college is a marathon. … Pace yourself, and you’ll accomplish more and be much happier.”

6. Assume you’re stupid so you can always be learning

I’ve struggled with this — I’ve always felt I had to prove myself. But I’ve found that ultimately, being selfless and working towards shared goals is for the best. And that means if there’s something you don’t understand, if there’s something that will deter you from getting the job done, speak up, even if you think you’ll look foolish for it.

“Don’t worry about being smart. It just causes you to stress out too much about your mistakes and occasionally even decide not to ask for help when you might need it.”
“Don’t be afraid to be honest with your mentors and managers — they want honesty above all else and will do whatever they can to help.”
“School didn’t teach you as much as you think it did. Assume you are stupid going in and just listen. Learn from others.”
“If you see every experience as a learning experience, you’ll come to value every moment.”
“Don’t just practice — deliberate practice.”

7. Take the lead; be proactive

Paraphrasing a response: Be grateful if you work in a good environment, but at the same time, it is your responsibility to speak up, voice dissent, and improve the work culture, for that is how a good environment is made and preserved.

“Company culture is comprised of individuals stepping forward and taking the lead on shaping it — from board game nights to karaoke nights to hiking trips.”
“Don’t wait for others to tell you what to do. Nobody knows what the fuck is going on in this world… they look for others to tell them what is next. Be that person.”
“Aim high and take ownership.”

8. Figure out what you want to do, and do it… maybe

Should you follow your passion? Many of my colleagues are at Khan Academy because they’re so passionate about education:

“You have to find what you’re doing personally meaningful. If you’re like me, spending 8+ hours a day on something you don’t believe in just isn’t possible.”
“There were moments when I told myself I was an idiot to leave… just for a chance at KA. There was a period of a few months where I was definitely following my feelings rather than thoughts about which route was the most secure, but I’m tremendously glad I did it.”

But what if you don’t know what you enjoy? How can you find out?

“One of the best things that I did was reading blogs and looking for companies that appeal to me on more than just a monetary level, but on some sort of spiritual or philosophical level.”
“One thing I’ve found helpful is to take a few minutes at the end of every weekday to list the things I did at work. I then mark my favorite and least favorite things. Over time, I can see what I’m enjoying and what I’m not enjoying, and I can work towards doing more of the things I love and less of the things I don’t.”
“I tried to take on a variety of projects… which really paid off as time went on — in part, because I discovered what kind of work I enjoy doing”.

But, be pragmatic. Like the answer to so many questions, it depends. Where are you now? What do you enjoy? Will you grow? Can you can support yourself?

“You’ll have time to follow your passion your whole life, build a reputation of great work and trust to enable you to do whatever your passion may be.”
(paraphrased) “Financial success is not insignificant, and I don’t deride people who choose financial well-being. It allows you to take care of your family. What matters most to me is you weigh that proportionately relative to your other criteria.”
“When choosing a job, I would have been much better off optimizing for the right team to work with over function…. They were very nice people, but unfortunately, they were not inspirational, and I didn’t realize how much I needed that.”

One respondent noted that earning experience put them in a position to choose rather than beg for any job they could get.

9. Climb the corporate bureaucracy by using this 1 weird old tip

Sorry, I don’t really have a neat theme to tie all this together. It’s just a collection of work-related thoughts.

At your first job, you may initially feel overwhelmed. But things are going to be OK. You’ll figure them out. It just takes time.
One colleague learned to be pragmatic instead of stressing out about doing everything the right way, and in so doing, developed a bias towards doing and shipping things.
Priorities shift and projects change. So, “developing skills is what really counts — learning how to work well with others, communicate ideas, manage stress, and balance your life.”
“Take time every week to reflect, set goals, and work for yourself. Be your own ‘academic advisor.’”
From a team lead: “It is absolutely critical that you become a person who does not drop things on the floor. When you say you’ll do something, when an idea occurs to you, when an issue is raised in a meeting and isn’t resolved, you have got that thing. It is recorded in some safe place, and you know that you will follow up on it. If you’re not going to do something, that is a conscious decision, not an accident.”
You can accomplish a lot more with a team, and build something bigger than yourself.

10. How I learned to stop worrying and accept that no one really knows what they’re doing

But it’s OK, because we’re young and we’re reckless. We’ll take this way too far. It’ll leave you breathless, or with a nasty scar….

“If something is likely to make your life better, you should do it. If it turns out poorly, you can change that decision later.”
“I think most of the best things that have happened to me in my life have been not totally my own choice. Looking back on them, I made some proximal decision that put me in a position to experience that thing, but very rarely was I primarily responsible for the best things that have happened to me.”
“Most decisions you’re going to make are not going to ruin your life. It may seem so scary at the time, but the things that I look back on most fondly are the times when I was pretty scared.”
“That thing you are losing sleep over because you are worrying about it? Get over it, it’s not worth that kind of time and energy.”

Takeaways

Okay, that was a long list of stuff. And you’ve probably heard a lot of it before. Yeah yeah, take risks. Yeah yeah, be happy instead of chasing approval. Yeah yeah, love. Yeah yeah, yeah yeah, yeah yeaaaaahhhh!!!!! It’s a damn cold night, tryna figure out this life….

So what’s going to be different this time? Frankly, I don’t know. That’s up to you to figure out. But I hear ya — I don’t remember myself the last time a blog post changed my life.

I’m not saying I want to change your life. But on the other hand, I’d feel bad if you read this far and just ended up wasting 10 minutes of your life, so actually yes, maybe I do want to change your life. Let’s frame it as, I hope you can derive some tangible benefit from this.

So here’s an idea. Pretend this is an experiment. Try to live by just one of these principles for a day. If you didn’t die, great success! Maybe try for a week next time, and push yourself to go even further. You can stop at any time if it isn’t working and you’ve given it a serious honest effort.

So I’d love to know if you found this worthwhile — if there’s something you read that challenged your beliefs, if you chose to do something differently because you were successfully brainwashed by my Silicon Valley tech company friends’ words, if you become a billionaire one day (and it was all because of this post, right?), or if you just have Taylor Swift (or Avril Lavgine or Sara Bareilles) stuck in your head right now. Any of these outcomes would make me happy.

You can tell me when it’s over, if the high was worth the pain. Something something Starbucks lovers.

Acknowledgements

Credits to Ben Kamens for the gif.

The acknowledgments from Khantemplations is reprinted below.

Thanks to Marcia for giving me that little nudge to actually send out the initial email. Thanks to Marcos for suggesting InDesign for typesetting this compilation and conceiving and laying out various book cover concepts. Thanks to Jessie for reading drafts, providing feedback, and keeping me on track to finish this. Thanks to Ian for providing feedback on cover designs. Thanks to Marcos and Tabitha for giving last minute feedback on the content typesetting and layout.

And of course, thanks to all you Khantributors whose responses made up this book. Thank you for taking the time to send me your words or go on walks. Thank you for being brave and letting your thoughts be heard. You have made this possible. So from the bottom of my heart, thank you. Your thoughts have given me much to think about and to act on.

Did you not have a chance to have your voice heard? Well, I hope this is just the beginning. May this be the kindling that fuels future discussions!

Thanks to Eddie Du, Jessie Duan, Mary He, Nigel Pynn-Coates, Owen Wang, and Jamie Wong for ~~proofreading~~ making me realize the first draft of this blog post was a piece of turd. Especially thanks to Jessie for bringing up thoughtful points that got me to rewrite this blog post, and Eddie for the quotes formatting. I’d let any of you teach English to of my future unborn baby(ies). I love you all.

Open-Sourcing UW Flow

2014-02-28T00:00:00-05:00

We’re really excited to be open-sourcing uwflow.com — a project that three friends and I have been working on for the past year and a half.

Here’s some highlights:

Flow lets University of Waterloo students plan courses with friends.
Some numbers:
- Last month, we served 40K pageviews to 9K unique visitors.
- Last year, users made 480K searches.
- In total, users have contributed 47K ratings and reviews.
Who made this:
- Development: Mack, Sandy, Jamie, and me
- Marketing: Terrance and Shubham (Sept. 2012 - April 2013)
We’re open-sourcing with the permissive MIT license because we want anyone to be able to fork and host this for their school.
Check out the code on GitHub, and come talk to us in our public chat room!

Read on for how we started and why we’re open-sourcing.

How we started

Two years ago, I met up with Mack and Sandy to hack on a side project with these goals:

Launch fast and iterate
Make something useful
Be startup viable

The idea was to also use this as our engineering capstone project [1], as well as potentially be a business that we could continue after we graduate. We could get course credit for doing a startup while still in school!

Over the next few months, we came up with about 300 ideas. But, nothing stuck. Nothing met all our criteria and that we’d be dying to work on everyday for the next two years.

Four months later, we finally agreed on something. We all hated planning courses — choosing the classes and professors that would make up your next term’s timetable. Taken from Sandy’s standard pitch:

It’s like doing taxes. Each time, it’s inevitable, and each time, it’s a pain. Right now, students use the undergraduate calendar to look through thousands of courses before even knowing what’s out there. And that doesn’t even include opinions from people who had taken those courses before. For that, we rely on friends and word of mouth. But that’s both incomplete and inconvenient.

So we thought, it’d be awfully useful to have a website where we could see what our friends are planning to take, what they’ve taken, and what they thought about those courses. Unfortunately, nothing like that that existed (or was widely used) at Waterloo to our knowledge.

Perfect. We’ll build it.

Development

We started coding in September 2012. Much of the work was done in the first two months. In an effort to launch early, we made sacrifices:

Skipped class
Stayed up all night
Aggressively cut features
Used up all of our assignment grace days

Of these, I regret the all-nighters. They helped with hitting deadlines (such as course enrollment periods), but were harmful to our health, happiness, and long-term productivity.

Our tech stack and codebase came to be heavily influenced by where we had interned – in particular, Khan Academy and ContextLogic. We borrowed practices, technologies, tools, and coding styles. We owe Flow to those internship experiences.

We were fortunate to have some awesome people join us. Two students in the Accounting faculty whom we met at the VeloCity residence, Terrance and Shubham, volunteered to help us out with growth and marketing. Two months after we started coding, our classmate and good friend Jamie asked to join us to form our engineering capstone project team. [2]

Launch

We first launched after two weeks of coding. It was an embarrassing MVP to just our Software Engineering class of 80 students. A few weeks after that, we launched to the 1500-strong Accounting faculty, and then a few weeks later, to the entire campus.

To get the word out, we tried all sorts of things:

Told all of our friends
Posted on Reddit, Facebook, blogs, and forums
Emailed residence dons and professors
Walked up to students and told them
Facebook ads (Sandy had credits as a former intern)
Posters

The most effective channel was social media – Facebook and Reddit. Posters and other physical media were among the least effective – we saw no discernible impact the week we had posters like this all over campus:

The first few months after launch were a really exciting time. We had 450 sign-ups the first day, and got lots of positive feedback and ideas for features. We ran a focus group, and one keen user typed up all of this of her own initiative:

(Today, a year after, most of these are still TODOs. Yep, we’re lazy bums.)

To seed our ratings and reviews, we scraped sources including official, publicly-released course critiques. We then hosted a “beer and review night” at the VeloCity residence, where we gave out beer in exchange for ratings and reviews.

We also ran a raffle contest, giving out a Nexus 7 and other goodies. This effectively doubled our ratings and reviews. If you’re a Flow user, that’s why you (still) get those mysterious points for reviewing courses.

We’ve found that our biggest traffic gains came from announcing new, useful features, such as our launch of class sections info.

Business?

Though we had some good traction, Flow didn’t really work out as a monetizable business. We were fortunate to have won a pitch contest from Waterloo’s Velocity startup incubator, and we are thankful for all the assistance and mentorship they’ve provided us. We also interviewed at YCombinator, but didn’t get further. [3]

We realized we weren’t enamoured with selling to universities – we wanted to build things our users would love. We didn’t want Flow to turn into mandate-ridden enterprise bloatware that would alert you every 20 minutes to ask if you don’t want to be logged out. Or sell that in Powerpoint presentations to “decision makers” who wouldn’t be the end-users.

We thought about possibly serving the exploding online courses market, but our core userbase was college students on campus. We did not have the heart to pivot and abandon them.

Neither did the idea of squeezing money from students greatly appeal to us – whether that’s through ads (that we never even look at ourselves), or through textbooks (that we haven’t bought since being clueless first-years).

We did, however, all agree on one thing. We wanted Flow to continue to be used and maintained long after we graduate, whether as a sustainable business or some other means.

We then realized that we much preferred to see Flow sustained as a community-supported, open-sourced project, than to submit to the whims of the highest bidder.

Open-source

So, that’s why we’re open-sourcing. Explicitly, the reasons are:

We’ll (hopefully) be graduating in two months.
We hope Flow continues to help students long after we graduate.
We want to see Flow continue as a student-driven project for students.
We have lots of features and improvements we wanted to make, but didn’t get the time. For example, analyzing prerequisites, recommending courses, building a conflict-free timetable, and auditing degree requirements.
We’ve always wanted to bring Flow to other universities (and others have asked us), but our hands were full with just Waterloo. Now others can do that for us. :)

So over the past few days and months, we’ve prepared the codebase to be open-sourced:

Wrote developer docs.
Ensured tests were in good order.
Wrote a style guide and set up a linter.
Tested setup on fresh installs of Mac and Linux (Ubuntu 12).
Picked the permissive MIT license so you can fork, modify, and host this for your school.

Want to contribute? Check out the code on GitHub.

Not sure where to start? Perhaps you can tackle one of these issues.

Have questions or just want to talk? Come hang out with us in our public chat room.

Want to fork Flow to be a white-labelled platform for your school? Awesome – go for it. Just be wary that the data scrapers are tied to Waterloo’s APIs.

What’s next?

Although we’re graduating, we’ll still be the primary maintainers of Flow for a while. But, we’ll be on the lookout for active contributors to whom we can gradually hand off responsibilities – review pull requests, ensure the site stays up, ensure Flow stays up to date, and, eventually, take over as primary maintainers.

Oh, and a few things coming up soon:

Email logins! We’re thinking of you, non-Facebook friends.
We’ll be boothing at the SE Capstone Symposium on March 21 in DC. Drop by and say hi!
Secret surprise.

Thanks to everyone who gave us a shot. Thank you for rating courses, writing reviews, making searches, sharing schedules, giving us feedback, or just reading this far.

Footnotes

[1] Also known as the Fourth Year Design Project, this is a 3-course sequence where Software Engineering students at Waterloo work on a single project in teams of 3-4 over the span of 20 months. Instead of a purely educational school project, we thought of this as the perfect opportunity to do a startup while we were still in school (and get course time and credit at the same time).

[2] Although Jamie is one of the most competent programmers I’ve worked with, and we’ve pair-programmed in ACM contest practices, we just had to ask him Fizz Buzz anyway. You know. Just in case. We also asked him his favourite interview question ever: “What’s your greatest weakness?” If you ever interview him, make sure to ask him questions like that.

[3] Here’s the email Harj sent us:

I’m sorry to say we decided not to fund you guys. It was a tough decision because we enjoyed meeting you and it’s clear you’re good programmers. Our concern is that while you’ve build good software, we couldn’t see how you’ll be able to acquire enough users to grow this into the kind of large, independent company that investors are hoping to fund. We think it would be best if you finished college and in the meantime worked on projects because they were interesting rather than trying to come up with a startup idea right now. That approach actually often leads to better startup ideas anyway and we’d encourage you to reapply again in the future, we’d be happy to hear from you.

We agree with pretty much all points.

Also, here’s a picture of us at YCombinator:

… fixing things we broke the previous night.

Start with Mockups

2013-09-25T00:00:00-04:00

My friends and I are working on a side project — a course planning website for our university. We decide to add a new feature that involves some UI work. My friend goes off to hack on it. 300 lines of code later, he’s done part of the UI and shows it to us. Here’s what it looks like:

It lets you add a course (done) for a specific term (not yet done).

We discuss this for a few minutes, and realize that the extra information in the dropdown isn’t necessary in this context. At this point, the user has a course in mind and just needs to find it, so all that’s needed is the course code and name. Something like this (with Select2):

Being smaller, we can place this dropdown next to term headings — we get the “for a specific term” part for free without designing more UI.

So, we agreed to scrap the first prototype and redo most of the UI. We were regretful of the throwaway code, but it was better than writing more throwaway code.

How could we have averted this?

Start with Mockups

We didn’t need 300 lines of code to figure that out. With a rough 5-minute sketch, we could engage in the same level of discussion.

As a programmer, I’ve found it incredibly helpful — before coding any piece of UI — to first draw a mockup. Just like a lean startup, the goal is to get feedback, fail fast, and iterate quickly. Experiment with multiple ideas before settling on one.

This has been a lesson we’ve learned during my internships at Sift Science as well — particularly since our awesome designer just joined in August. So, we now operate with this rule:

For any major additions to the UI, please always draw some quick mockups on paper first. And by draw, I mean scrawl, scratch, barf graphite onto paper, whatever you want. This is not art. You will not be judged on how pretty it looks — I promise you. Mockups are purely meant to define interaction and layout — not visual styling/design.

The purpose of a mockup is to be able to very quickly iterate on UI concepts, before it gets much more difficult to change once committed to code. It’s also much easier for someone to give you feedback for big changes on a 5-minute mockup than on beautiful shiny working code and UI.

Another advantage of starting with low-fidelity mockups is that it focuses time and attention on trying different layouts and interactions, instead of refining one and being stuck at a local maximum.

A pencil-and-paper mockup also makes it easy to communicate to other team members what it is that you’re building, concretely (heh).

Finally, just to emphasize that a mockup is a communication and ideation tool — not the end result — here are some of my brilliant masterpieces that you can mock at:

Seven concepts of a tagging widget for an unreleased side project. We ended up going with #4. On other mockups where I heavily relied on hover states, my friend gave good reasons to avoid them and suggested alternatives (while still showing relevant info without clutter).

Here’s a series of three mockups depicting the UX to get human training data to improve a customer’s machine learning model for Sift Science.

Notice the misbehaving, untrained puppy in the upper-right.

(The “share on Facebook and Twitter” button is a joke.)

The puppy has been trained into a fraud-fighting dog!

I hope that shows you can be devoid of artistic skills and still draw mockups.

Why Interns should Blog

2012-09-14T00:00:00-04:00

At the University of Waterloo in Canada, all engineering students enroll in a co-op program (“co-op” is Canadian for internship). We alternate between four months of work and four months of study, graduating after five years with eight study terms and up to six co-op terms (two years of work experience)! Consequently, there’s usually Waterloo interns at many tech companies in Canada and the USA.

Now, for software engineering students in my class, we’re required to write technical work reports that usually ends up being 20 to 30 pages or longer, for four co-op terms. The goal is to assess judgement and writing ability, but the format is very rigid and unsuitable for many development internships.

One day, it occurred to me what a great win it’d be for everyone if even a portion of this effort could be turned into public blog posts. To illustrate the potential, of the 80ish students in my class, just three blogged about their internships to my knowledge. Of those three, all of them made the front page of Hacker News! In fact, one was even invited to present a TEDx talk and got his next co-op through a blog post.

With ideas from some of my classmates and my mentor Ben Kamens of Khan Academy, I composed a proposal arguing why we should let students write public blog posts for their work reports:

Dear Software Engineering administrators,

I’ve received some feedback from students in our class regarding the existing work report format. As an academic rep, I’d like to communicate this to you.

I’ve been told that the format feels contrived. Ideally, when one encounters a problem during a co-op term for which one needs to make a choice, one would conduct an analysis, write a report based on it, and then implement the solution. However, what many of us usually do is to just hold an informal discussion and make a quick choice after that. When it’s time to write the report, students may “exaggerate the problem and pretend the choice was made after a full study.”

There are also students who resort to submitting self-study reports or confidential reports if their work term did not produce an appropriate subject that will satisfy the required format. Further, in many startup environments, there is little time to write a full report analyzing alternatives; it is preferable to just pick one and ship it, replacing/fixing it later if required. Fast iterations and user-facing work is emphasized, and it’s just not practical to write a report of this size and scope for purely internal purposes.

I’m not recommending reducing or eliminating work reports altogether. Rather, I would like to propose an alternative:

For one work report submission, Software Engineering students can choose to write and publicize a technical blog post instead of a synthesis/analysis work report.

Why a blog post?

Blog posts are an important new form of media to complement and broaden a student’s writing repertoire. Many startups have accompanying blogs that increase community engagement and drive traffic to their product. These blog posts could bring publicity for Waterloo’s co-op students and Waterloo’s co-op program. Because these blog posts will be public, students will be motivated – not to satisfy a checklist of requirements – but because their co-op employer, their colleagues, professionals in industry, and potentially a future employer will be reading it (this has actually happened).

Further, not only do blog posts help increase publicity for the student, but they may actually be helpful to the community. For example, I have a friend writing a self-study report comparing various server push technologies and analyzing which are the most appropriate in certain scenarios. He was also considering writing a report on which mobile platform, Android or iOS, to develop for first. The amount of work and research put into a self-study work report would serve as a fantastic community resource if made publicly available as a blog post or a series of blog posts.

My co-op mentor and lead developer at Khan Academy, Ben Kamens, repeatedly encourages all full-timers and interns to blog, even during company hours. On the topic of interns writing blog posts, he has also said:

raises the market value of interns
great practice for shipping work into the real world – blogs are mini products, and getting familiar with moving past the “it’s not ready to ship” inertia is invaluable”
possible that a community will spring up around the post – might be invited to speak at a conference as a result

Here are some examples of students in SE 2014 who have written blog posts based on their work terms and reaped significant benefits:

Jamie Wong wrote a blog post about everything he did at Khan Academy. This found its way to the front page of Hacker News with 102 points and subsequently attracted 12 000 visitors over 2 days. Jamie received a response from Peter Norvig, AI researcher and director of research at Google, congratulating him on his work.

Another student’s blog post on a co-op project received more than 200 points on Hacker News, was top post for a day on the programming community on Reddit, and was viewed by more than 20 000 visitors the day it was posted. As a result of this blog post, this student received multiple employment offers (one of which will be his next co-op term) and was also invited to present at a TEDx event.

Wen-Hao wrote about his project at Square, PonyDebugger, which attracted 260 points on Hacker News.

From the perspective of work report markers, enforcing consistent marking might be an issue. Whereas the existing work report template is very well defined with unambiguous checklists, marking a blog post will leave more to the marker’s discretion and judgement. Instead of specifying a rigid structure, for a public blog post, we may just want to set out some guidelines. As a starting point, here are some possibilities:

be on a technical subject
have a minimum length requirement in number of words (can split up into a series of blog posts)
be publicly accessible
make an effort to promote to the relevant audience?

Grading could just be pass/fail; let the public nature of the blog post serve as the incentive to make an effort.

If there are no other issues that we can’t work around, I think this would be a great thing to try at least once.

I truly wish this could be something very useful, not just for students themselves, but for the technical community in general.

Thank you for your time and consideration!

I’ve received a response that the old work report format is being reworked to be less rigid, which is great to hear. However, there’ll have to be more work done to have the blog post proposal fleshed out and accepted, but at least one professor likes the idea. We’ll see where it goes.

Leaving public artifacts is a great way to not only provide natural incentives, but to allow for unbounded creativity and enrich the world at the same time. I sincerely wish that this would encourage more students who might have been thinking about blogging to blog.

If you have any anecdotes about blog posts, or what I could do to convince my school, I’d love to hear them!

Also, I’m overdue myself for another blog post about Khan Academy (I blame working on my 30-page work report :P). Stay tuned!

Announcing Numbers API

2012-03-05T00:00:00-05:00

tl;dr: Check out Numbers API for interesting facts about numbers.

Back when I was interning at Khan Academy a few months ago, my intro project was to create a dashboard for exercise statistics. Among the variety of graphs and widgets was one that reported a factoid about the number of shipped exercises, such as “We now have more exercises than the number of years Harriet the Galápagos tortoise lived (1830–2006).”

People seemed fascinated by their tidbits of random trivia. Sal Khan even suggested hosting a web service for these facts at numbers.khanacademy.org, but nothing came of that. That is, until now.

My friend Mack and I were deciding on a side project to distract our minds from Bode plots and Jacobians (if only Sal had some videos on Control Theory…), and narrowed down a crapload of ideas to just one. The winner was an API for interesting number facts, based on the criteria of being novel, completable in a few weekends, and interesting to us.

After a few weekends of on-and-off work, including a demo at our class’s student-organized hackathon, we’re finally launching numbersapi.com!

We built Numbers API with Node and Express for the back-end and Sass + Compass to make CSS bearable. We designed the landing page and the web service together, and then Mack put together the first iteration of the landing page while I set up the server and wrote the API + docs. We then sort of swapped roles, and I had the chance to prettify the landing page to my tastes. I also yet again self-plagiarized from my work at Khan Academy and did 20% of the work required to convert the rolling numbers counter into a jQuery plugin (the other 80% is more testing, performance tuning, code clean-up, and greater extensibility).

Mack, on the other hand, started hunting for content. I believe he used LXML to scrape Wikipedia and a Python natural-language processing toolkit to process sentences into a consistent grammatical form. This form was specifically chosen to be flexible enough to be used in a sentence like “42 is <fact string>.” as well as “We now have more exercises than <fact string>.” (I leave the exact form as an exercise to the reader.) This turned out to be non-trivial for trivia facts, and we even resorted to manually combing for interesting facts on cardinal numbers (which were far outnumbered by numbers used as names and codes) and phrasing them appropriately.

This is just the beginning. Time-permitting, we are also thinking of extracting and paraphrasing interesting facts from the Guinness World Records, almanacs, numbers in nature, statistics, number of works by famous artists, etc.

Have fun, and we can’t wait to know how you’ll be using these number facts! We ask only that you be gentle to our server.

Numbers API

My Experiment in Daily Idea Generation

2012-01-05T00:00:00-05:00

I’ve long had the fear that, should I ever want to do a startup, I wouldn’t have an idea that I wanted to passionately pursue. So, for about a year, I’ve been passively maintaining a list of any idea that pops into my head as a Google Doc. With about 40 items on that list, there wasn’t anything that hadn’t been done that I thought was particularly compelling. Then, one day, a meta-idea popped into my head: what if I spent 10 minutes every morning actively generating ideas, writing down any thought I get in a stream-of-consciousness, unfiltered fashion? I could just cut back my morning Reddit + Hacker News + RSS reading.

Would there be anything interesting? How many would I get? Would the volume trickle off after a few days? Would it increase the rate of passive ideas that pop into my head? Would it nurture my creativity and engender greater right-brain activity (whatever that means)? I was curious.

So, I started this experiment hoping to do it every day for about a month. I didn’t care how crappy the ideas were, as I kept in mind:

I’m more interested in the results of this experiment than for anything worthwhile to come as a bonus side-effect.
I don’t care what anybody thinks. I did not need to show this to anybody.
Have fun. If not, then I’m doing it wrong.

By the end, I accumulated a total of 286 ideas over 29 days:

All ideas

This is the raw list of my stream-of-consciousness writing, with some points edited to be clearer. (See below for my favourites.)

Tue Nov. 15
- Dream or sleeping patterns (this was just the first thing on my mind when I woke up to get me writing)
- Anonymous love messages - FB app?
- Open-sourced UserVoice (feature requests & questions forum)
- Are you working? Monitor computer usage: # of lines of code written, what sites visited, breakdown, what apps you used
- Vim plugin: show how a document or code was created. Show all insertions / deletions, commands used. (Show off your Vim skills.)
- Browser history analytics: time spent, histogram, scrolling behaviour, # of words read
- What to buy friends? FB app? Public wish list.
- What should I build? Voting. Private / public. Champion ideas.
Wed Nov. 16
- ”I’m sleeping” notifications – chat, email, phone
- Public shaming service for productivity. I will X or owe everybody $20.
- Computer usage analytics: hours per program, file sizes, interactions, etc.
- Git analytics: commits / day, which hours, lines of code, etc.
- Learn a language morning cards / alarm clock
- Wiki: summary (tl;dr) of everything: articles, books, movies, shows, news (with multiple granularity: days, weeks, months). For the impatient.
- Meme tracker: keep updated on new memes (eg. for Reddit) or ways of saying things (eg. “redundant X is redundant”, “y u no”, “le *”, etc.)
- Top news or Reddit distilled to a newsletter
- Learning protocol / specification; universal learner’s profile. API specs. (like OpenSocial for students)
- Who are you most compatible with? FB app?
- Programming wiki: JS, CSS, any language. Good examples, usage, advice, links to references, StackOverflow questions, collaboratively created, threaded comments w/ upvotes? Seeded (eg. with MDN)?
- Open-source / wiki code snippets, recipes, etc. Eg. iterate over each line of file, in every language.
Thu Nov. 17
- (4 Khan Academy ideas (where I was interning at the time))
- Global ideas; people submit their issues / problems they would pay to have solved
- “You’re going to be late” automatic notifier on calendars
Fri Nov. 18
- Nutritional info daily: scan products’ barcodes (or enter manually) what you’re eating in a day. Stats, recommendations, are you healthy?
- What similar people are doing / reading / working
- Real-time Reddit: stories being submitted, upvotes generated
- Am I wasting my time? Daily email usage, chat, Reddit, terminal, gaming. Histograms.
- Mall / store item locator. Insert what you want. Gives short Hamiltonian path. Recommendations, ads.
- Learn from coding master. Get paid to coach.
- Virtual pair-programming over internet.
- What are my friends doing? eg. Visiting LA, watching which movies, seeing which concerts, go to Startup Weekend. Recommendations. Meetups. Organize trips or bring together groups going to same event.
- directory / wiki of failed attempts / ideas
Sat Nov. 19
- Where did you buy that / what is that product? Take picture or scan barcode. Gives links.
- Mobile app: need to be where at what time (checks with GPS). If not, server publicly shames you (posts on your twitter, FB, etc.).
- Something between PaaS and SaaS: you give data schema, can just write client-side app. Server only for data persistence + external requests.
- Restaurants: where do I eat today? Recommendations (proximity and favour unvisited places), ads.
- API: translate a word into multiple languages and make pretty Wordle-like motif
- Google image search with upvotes
Sun Nov. 20
- Simplified / pretty UI for weather. Single line graph for temperature & precipitation; zoom in and out
- Self-diagnose: answer quiz about lots of health-related signals (eg. age, sex, conditions, etc.). Tells probability of various forms of cancer, depression, various diseases.
- JS plugin: quote expander for long text. Replace long / less important text with intro sentence or a pre-written summary; click to expand
- Chrome extension for reading level of Reddit / HN articles
- Person info gatherer: that person’s Wikipedia article, LinkedIn, FB, blog, what known for, phone #s, locations, etc.
- How arrogant is this writer / blog? ML or voting
- Particle filter algorithm demo in JS
- Public shaming: web scraping or write script. “David’s blog did not have a new article by date.” API for GitHub, web, social, etc.
- Culture wiki: travelling to / migrating to new country? Learn social norm, fairy tales, popular movies, people, how to act, etc.
- Article recommendations based on what you like / dislike. Use Reddit info?
- Beginning {programmer,engineering,electrician,doctor,painter,etc.}: what to learn? What skills are most useful in industry? What tools are used?
- Goals web: Bayes net. eg. “I want to work at Google”. Lists possible prereqs (what did other people do before accomplishing certain goal?) Click prereqs to recursively get their prereqs. Get your own personalized path to goal. Infer structure and probabilities from user input. Eg. Are you a prof? What did you do before / to become one?
- What do similar people do? Eg. What do HN readers, standing-desk users, Cubans, GEB readers, rock climbers, terminally-ill, etc. like to do? What are their age groups, ethnicities, hobbies, other characteristics? (eg. Reads Reddit? Have a blog? Contributing to StackOverflow? Doing a startup? Working at tech company? Have read book X? Age histogram? Played piano?) You fill out detailed profile / answer questions like 20Q. Recommends what you could do. Recursively see what people doing each recommended item like to do.
- Wallpaper rotates between top daily submissions to /r/wallpapers or /r/earthporn or other source
Mon Nov. 21
- Alternatives to: software, other things. Function clusters. Ratings, Feature charts. Reviews. Network. Use X if doing Y, etc.
- Problem: need help finishing what I started
- Commentary on non-fiction books. Criticisms, summaries, wiki errata. Community annotations.
- Atheist, evolution, etc. arguments, common rebuttals/responses, appropriate counter-rebuttals. Rebuttals to common misbeliefs. Arguments flowchart to convince somebody of X.
- Online UML creator
- Online (JS) data visualization: dumb simple straight copy+paste, or input data in cells or chart. WYSIWYG configuration. Output appropriate charts, get permalink to output, produce HighCharts (or other JS charting software) code.
- Pair picture slideshows to music: slide transitions match sounds in music; slide mood matches tempo/pitch/loudness; find appropriate music for givens slides; find appropriate pictures for given music
- Wikipedia road map: most linked articles, visualize links as web
- Distinctions Venn diagram: enter two keyword / phrases. Uses Wikipedia / Google / some API to find similarities and differences
- Online PDF editor: annotations, form fields, etc.
- HN / Reddit auto-submit. Go to sleep; will automatically submit article at optimal pick-up time
- Map of HN readers, Redditors, etc. for each article. Links between stories commented by the same user
Tue Nov. 22
- Code for academic papers: open-source implementations
- App feedback / usage: Post screenshots, videos, playback IDK! etc. (I’ve forgotten what I meant when I initially wrote this now.)
- Do I have bad breath meter
- Blue light filter (software or physical): For reading phones at night, etc. (Flux)
- JobVite for seekers: Recommendations, where to apply, resume hosting, checklists, what other jobs seekers are doing, common qualifications / keywords of other seekers, etc.
- Given some website, outputs a packaged theme: colour scheme, font scheme, templates, CSS, design elements, widgets, etc. Make it easy to copy another website’s design
- Dumb simple one-click easy-to-use program for SSH forwarding to access Pandora, other US-only services elsewhere
- Reddit picture viewer (with captions): Collage of pictures
- Q&A or discussion forum on any webpage or Wikipedia page. Common misconceptions? Browser extension?
- LaTeX-as-a-service on a server: will compile LaTeX for you; API (because LaTeX is a pain to setup)
- Design critiques: people can comment on webpage. Possibly automatically check against design guidelines? Lint / page-speed for design.
Wed Nov. 23
- Mistake database: document every mistake you’ve made. People upvote. Good for learning a new language, technology, skill, coding contests, etc. (eg. Python’s default args evalled once; JS closure-in-for-loop gotcha)
- Joke database: upvotes, etc.
- Talk topics: enter what you like / have talked about; suggests other ones
- Re-word sentences: Here’s what I want to say. What’s the best way of wording it?
- Number range trivia app: Estimate a certain number (eg. # of lions). See who’s the closest. Histogram of other users’ estimates, mean, etc.
- Apps for common board games: apples-to-apples, Pictionary, etc.
- Ping-pong ball-collecting robot (for training on a bucket of balls)
- Can I get into university X? ML? Enter marks, extracurriculars, etc. to determine which programs / schools suitable and probability of acceptance. User voting?
- Career choice. ML (do a quiz; answer 20Q questions) suggestions. Ask existing professionals.
- What tools / libraries / framework is it built with? Tries to automatically detect? Wiki?
- Aggregate karma: GitHub, StackOverflow, Reddit, HN, Twitter followers, etc. Total followers, following, etc. Another layer of game mechanics, interesting challenges, and badges! Follow your friends! Leaderboards.
- Share interesting code / one-liners (like Commandlinefu for any language)
- Share how you solved technical challenge
- Open-source or user-contributed API dictionary
- Make Google search results link directly to result instead of redirect
- Critique platform: Upload any media (eg. pictures, videos, links, text, etc.). People annotate with critiques (eg. more whitespace here, change to this font, this colour, redesign like this, etc.). StackOverflow-like game mechanics. Offer monetary bounties?
- Decision-making platform: What are your criteria, budget, needs, etc. ML or user-contributed w/ voting and StackOverflow game mechanics.
- Bike-sharing for interns
- Share browser bookmarks + aggregate stats, recommendations, etc.
Thu Nov. 24
- Warning, prompts, reminders to go to sleep: alarm clock for sleeping.
- Quiz you on knowledge of a programming language. Warns you if dangerously lacking; shows percentile? Suggests articles / books / gives tips to fill you in on crucial gaps in your knowledge.
- Random app ideas generator? Eg. {social,photo-sharing} for {neighbours,music lovers}. Markov chains?
- Improve this line of code / how to write this more idiomatically / code reviews
- what’s the term for “concept” (like /r/tipofmytongue) or fill in the blank: “blah blah ____ blah”. ML or user-contributed w/ game mechanics
- Check for updates on page. If there are, refreshes or just replace with new content with minimal visible reflowing or show some indication
- Lure theists / beliefs in certain things: Asks questions. User does quiz. Question by question, a contradiction is built up, or slowly lead to more ridiculous things.
- Write comments in code. Will find related StackOverflow posts. Prevent you doing stupid things such as parse user-provided HTML with regex.
- What apps + tools should I use? For programmers, musicians, designers, etc. Aggregates what people use. Create your own usesthis profile, find similar ppl, have nice page to link to from your blog, etc.
Fri Nov. 25
- Haiku-ify: Take sentence (or paragraph or article), convert to Haiku or other poem
- What can I get for X dollars?
- Pandora client for Mac - free version
- Lint all - lint a directory (detect file types)
- Mercurial / Git pre-submit linter
- Tutorial or wiki for beginning programmers on everything: introduces Linux, Vim, Git, JS, Firebug or Chrome dev tools, Python or Ruby, C or C++, a functional language, Makefile, Apache or Nginx, DNS, setting up a server, a web framework, where / how to ask for help, how to debug, etc. (things not always taught in school but useful in industry)
- Travel app: checklists, itineraries, games, weather, etc.
- Someone lonely / wants conversation near you
- Go eat lunch with somebody close by (for those who don’t want to eat alone)
- Create fake FB wall posts for screenshotting (to submit to Reddit (like /r/atheism))
- Mistake database with regexes to capture it. Lint existing or new code.
- Malware / virus to force-upgrade IE users or install Chrome frame
- Browser upgrade for old people: Chrome but with IE icon (or look?), auto-import settings and bookmarks, really really easy setup.
- Pro-tips around your neighbourhood / city. Where to eat, best place for haircut, when garbage truck comes around, where to get X, where Y hangs out, etc.
- Jimmy Wales “A personal appeal” banner parody / creator
Sat Nov 26
- Am I sleeping? Mobile app + alarm. If sleeping notifies others and updates chat client statuses
- Arbitrarily-detailed math proofs: click each step to expand into more steps, and so on
- Game: take short sentence and recursively replace words with definitions. Guess the original word / phrase
- Get real-world colour in RGB (mobile camera app)
- Find top colours: Compare any two (or two similar colours): ask which one is prettier. Or ask which one is the more canonical “green”. Ask which colour goes better with another one. Game mechanics to create aggregate info for colour designers.
- Comparison test of prettiness: Side-by-side display two different versions, ask people which one they prefer.
- Idiom linter: linter replaces common code constructs with their more idiomatic or more efficient forms (or more correct ways, but prompts first): eg. if (a) return true; else return false; –> return a;
- Happy birthday personalized Google / homepage
- Mobile app for ml-class or ai-class or other Stanford online courses
- Mandatory knowledge every person must know: eg. instinctive drowning response, Heimlich manoeuvre, etc. Quizzes you
- “Hip” tracker – what’s trendy right now (Google search trends or Twitter or upvotes). Comparisons of common things, eg. languages or frameworks
- Wiki for most idiomatic way to do a common task X in every language (eg. file I/O, print a line, regex test, etc.)
Sun Nov. 27 (at Yosemite today :)
- Yelp-like tips (eg. best menu item, do X) (take highlights / common parts) of reviews for everything: eg. Places, tourist attractions
- Simultaneous wake-up: alarm keeps ringing if all parties are not awake (or just notifies if other party has hit snooze or has not cancelled alarm)
- Bear / wildlife detector / mapper
- Documents for every location / place to visit: maps, rules, brochures, etc.
- Common mapping software for parks / indoor places / fairs
- Travel itinerary for different places
- Get to sleep app: plays a boring lecture with monotonous voice; boring book; soothing sounds
- Interesting stories about the places / things near you
- App: extremely simple and easy-to-understand open-source / wiki instructions for everything: microwaves, cars, various hardware, etc.
- Remote-activated middle finger lights on side / back of car
- Next rest stop app (for road trips)
- Mechanical / animatronic dog to scare away burglars
- Synchronize car stereos (or phones or other devices): like lots of boomboxes / speakers
Mon Nov. 28
- Water or calorie consumption estimate: walk this far / do this exercise this long. How much energy depleted?
- Number of deaths at certain locations. Mobile app? Closest death to you.
- Motivational stuff: pictures, videos, etc.
- Mobile app: voice notes
- Dictation software: write by dictating. Shows you guessed text along with audio recordings, make it very very easy to edit misguesses. Voice commands for editing.
- Am I healthy? Enter sleeping hours, exercise types & times, eating, drinking, washroom breaks, etc. Tells how healthy you are compared to others, what you should be doing to improve, what you are at risk of
- Infographic / size viewer creator. I can’t visualize extreme numbers. Express numbers in other things / forms, such as number of Plutos to make up mass of Earth, probability of death from wolf attack, number of books to stack to how high, number of times around the Earth, etc.
Tue Nov. 29
- Productivity: only allow certain apps to run for a specified time. Blocks Reddit, HN, etc. after a while. Enforced rate of text entry or something to force you to do work.
- Build-a-web app TopCoder / contracting. Modularize components, eg. contest to design logo, contest to write code for this tiny component, this API, etc.
- Safety tips website with micromorts info (probability of death)
- Scan product barcode (or take picture) at store; get reviews
- Watch TV show together over internet with live chat, polls, predictions, etc.
- Enforce reasonable tab order on forms: left-to-right to bottom-right
- Tool updates: new tools, plugins, reviews, etc.
- What software do you use? Updates you on software your friends starts using or similar people, or trending new software or alternatives replacing incumbents
Wed Nov. 30
- Post your standing desk setup picture (or just desk + monitor setup). List products / hardware + price
- Life hacks website: number of people doing it, upvotes / downvotes, reviews, cautions, related profiles to you, related hacks
- jQuery Earth globe plugin
- Motivational Growl messages: On Reddit too much? Reminds you to work on day’s most important tasks. When you open computer, shows you what you should be doing today (from your task list) in very big text in middle of screen. Occasional reminders throughout the day
- Who do I look like? Faces database or closest matching celebrity
- Drag + drop tangrams app (probably exists)
- What is it like in place X? Temperature, web cam, traffic, weather, etc.
- Markov chain twitter messages
- “What hipsters are using now” descriptions of new technologies, used by whom, switching from, comparisons, etc.
Thu Dec. 1
- Little known facts about me
- What does it sound like / media library
- Gift picker: answer quiz about person, suggests gifts
- Brainstorming tool: As you enter ideas, will show images / word associations to inspire you
- What would you rather do? 2 difficult scenarios: eg. Break arm or leg
- Anonymous email sending (definitely exists)
- Shared logins: Use service X together, split the bill
- Discussion aggregator: all comments about a site / page from HN, Reddit, Twitter, etc. (So blog authors don’t need to update with links to discussion on HN, etc.). Plugin so blog authors can put it on their page, separated by section. Or browser extension to see comments on other sites on any webpage.
- HN / Reddit Wordle, aggregate stats, time trends, interesting visualizations, etc.
- Anonymous love messages / compliments – anybody can create a page (or add to existing one) for any real person, and write positive things for that person. Donations
- Upvote anything at any granularity: highlight chunks of text or quotable sentences and just upvote those, or pictures or parts of pictures. Discuss anything on any webpage
- Public webpage / website like / upvote directory.
Fri Dec. 2
- Word games online: write one word / three words or 1 sentence of story. Other continue it
- Conventional way / idioms of doing X in Y
- Translate code into story (literate programming)
- CSS generate from WYSIWYG editor
- Real-life snakes and ladders (use slides?)
- Mobile app: Pro-tips at certain locations. eg. Avoid that side trail
- Which usernames / emails are mostly taken? Suggestions. User-contributed ideas?
- Better diff (more context / side-by-side) for GitHub
- Gifts exchange (some illegible word)
- Secret Santa for everything, eg. birthdays
- Checklists for everything. User-submitted, customize (or fork) for self or company, etc. (eg. Web service pre-launch)
Sat Dec. 3
- New website / web app launches
- jQuery animation generator. WYSIWYG UI.
- Draw stick figures; they will battle and interact with each other. Different sizes / weapons have different characteristics (but limited ink?). Draw and save your army; battle with others
- Beautiful CGI landscape + music (eg. glider through game like Skyrim) People can contribute to beautiful landscape
- jQuery element explosion / text letter-by-letter explosion plugin
- Blog article –> game: different letters / words are different enemies with different characteristics. Certain letters / features are obstacles / part of the map. Browser extension?
- See what other people (or aggregate history) are doing on the site: their cursor positions, clicks, etc. Live chat with other visitors? (eg. on article that have just hit HN / Reddit)
- Video comments to articles: easy and painless, one-click record with webcam
- Live article discussions: live chat and Google+ hangouts? Popular HN / Reddit articles
- Game mechanics for life (Make real life as addictive as WoW or Second Life)
Sun Dec. 4
- Visualize your bill / spendings: Histogram, categorize, timeline, etc.
- Company that helps people live a nomadic lifestyle: arrange lodgings, language mentor, etc.
- HN / Reddit daily paper newspaper, delivery
- Short story / article writing contest (like TopCoder)
- Multiple collaborators writing by expanding: Click a word or sentence, make that into a story (flesh it out)
- Food roulette: Random restaurant near you
- Stay in touch: Randomly connects you with a friend you haven’t talked to for some time, updates you on what they’re up to
- Online virtual browsing of real stores. 3D, look like actual store?
- Behind the scenes website: How article was written (with actual edits and such), how was web app made, course prepared, etc. Show granular steps.
Mon Dec. 5
- Git rebase GUI (inspired from one of Mack’s ideas)
- Game: everybody contributes to a drawing. Add in details, colour, etc. (Optional penis / obscenities filtering)
- Pictures on timeline + music with text experience: customize & send to loved one (like very customized e-card)
- Reliable thorough useful modular formula sheets. Allow forking, customizing by changing notation, collaborating, upvotes. Cheat-sheet creator for a class.
- Audio / voice journal
- Make time predictions about tasks. Eg. finish blog post by 9 pm / in 3 hours. Occasional reminders, see histogram, prediction accuracy / biases. Convert to time card with hours worked. Google Calendar integration
- Disputes / comfort levels / satisfaction in relationships. Tell your partner complaints or choose common ones, fill out survey, etc.
- Public “I love you” messages: lots of customization: theme, background, uploads, font, etc. (like about.me)
- “My greatest wish is to …” (fill in blank) with upvotes, FB likes, etc.
Tue Dec. 6
- App that monitors your bank transactions, emails, etc. for anomalous or fraudulent activity
- TopCoder for designers (or just use upvotes)
- What words come to mind for this book / movie / show / etc. ? Tagging, Wordle. Find similar things by tags?
- Submit your fantasy. Upvotes. Actors will act them out
- Aggregate everything you need to catch up on: eg. HN, Reddit, chat logs, email, Twitter, FB, code review, various app notifications
- Learn from an expert: screencasts of professionals doing / making something with audio commentary
- Draw shapes + colour –> generate CSS 3
- Platform for free educational videos made by professionals in field. Upvotes.
- Word this sentence in a certain style: Victorian England, cowboy, gangsta, Southern, etc. How would they say it? ML or upvotes or both?
Wed Dec. 7
- Reddit link bait title generator
- Intuition for equation: graphs of it, adjust values of parameters and see effect; incrementally applies transformations and see what graphs / table of values looks like at each stage
- Reddit joke explainer / updates you on new memes / inside jokes (possible duplicate Nov. 16)
- “Concretely”: What are practical applications of various theoretical tools / equations? Why am I learning this / what concretely is this useful for?
- Is that design stolen / what other sites share that design / code
- Site to replace interview phone screens: three coding questions, interviewer can watch sped-up version of how code was written later
- Interview technique: give a few commit messages. Have them code those
- Problems database: what bothers you in life? Upvotes
- Shared, anonymous FB, twitter, G+ accounts. Use to comment anonymously on public posts
Thu Dec. 8
- Alarm for going to sleep: audio reminders, air strike sirens, etc. Interrupts computer eventually
- Opposing viewpoints comment site or plugin. Rebuttals to arguments. Show how many support or oppose given topic; show most upvoted argument, most upvoted rebuttal, etc.
- Conversation topics: enter what you plan to talk about. Suggests others, top news, etc. Recommendations based on other users
- Line-by-line commenting on other sites. Plugin or central site. Like MS Word or Google Docs commenting (possible duplicate Dec. 1)
- Programming language “pop quizzes”: contests, vote on which questions you liked, winners get karma?
- Bike-mounted radar / laser-rage finder. Warns you of incoming traffic
- Age-guessing game. Upload your photo + age, see what others think. Guessers get karma for close guesses
- Mnemonics: eg. “Every Good Boy Deserves Fudge” but for every subject. Eg. anatomy, every formula
- Set bash prompt to emoticon (eg. table-flipping) or informative indication depending on current directory, # of files, some custom script, presence of version control directories, etc.
Fri Dec. 9
- Users flag telemarketers’ phone #s, software in phone blocks or warns of them
- Alternative uses for common household items. Eg. pipe cleaners can be used for cleaning pipes, cylindrical CD holders can be used as bagel holders
- “What I should have said” post original conversation, but add what would have been much better response / comeback. Upvotes, other suggestions
- Bike-mounted smartphone holder
- “Challenge Accepted” - people pay to see stunts / dares (eg. eat X chili peppers, hop on one foot while X and Y for Z distance). Acceptors upload video proof and judged by community or moderators? Money refunded if no acceptors.
- “Thank you”: public thanks or shaming people
- Video Q&A: upvote questions, then go directly to part of video where it’s answered
Sat Dec. 10
- Debate anything: hangouts / chat rooms just for debate
- Tracking / sharing with neighbours: list what you have / can offer
- What tools do you use at work? For hobbies? Aggregate stats
- Think you can improve something? Upload “before” and “after” images. Improve designs, UI, etc. Commenting & voting
- What is your P(death | some time range)? Enter your activities. Tells at what times / doing what you are most likely to die. Also enter any activity, eg. 5 hours flying, and equate that to something else, eg. X time in car
- Get large / small #s in terms of other things. Eg. lifetimes of the universe, something vivid / shocking (possible duplicate Nov. 28)
- Review any item in restaurant menu: query where is best burrito? Sort by most popular item at any restaurant
- Crowdsource decision making: state your scenario, lists pros & cons. Others can add discussion + voting (comments are colour-coded to show which side you support) (possible duplicate (similar) Nov. 23)
- Your quirks, and who else shares it (eg. smelling books, touching tongue)
Sun Dec. 11
- Revision control blame that finds when line of code was originally added, ignore displacements, whitespace, etc. Or shows timeline of that line of code through history and every change that was made to it.
- Vim plugin that finds documentation of function or language feature (on ctrl-k?)
- Find similarities between you and someone else –> gives talk topics (mobile NFC app)
- Mutual / collaborative brainstorming. Real-time, upvotes? Time limit? Winner? (Most upvotes)
- Enter Git commit message before working. Get analytics (eg. time taken), automatically sets that as status message to teammates
- Which is better: anyone can upload 2 images, set a custom question, customize # of views / demographics, etc. (possible duplicate Nov. 26)
Mon Dec. 12
- Culture characteristics. Venn diagrams
- Bash prompt creator: apply colours to ranges, special fields (current directory, hostname, time, etc.), useful Unicode symbols. WYSIWYG UI.
- Collaboratively created CSS / JS bugs and quirks test pages with demos. Upvotes, comments for workarounds.
- Place wagers on show outcomes. Polls.
- Post-Christmas gift bartering / trading (possible duplicate Dec. 2)
- Creative gift ideas wiki
- Image –> auto slice, HTML, CSS
- Simulated “Cash Cab” in taxis just for fun (just trivia game, no prize money)
- Ghost you: See your jogging, biking, race progress (where you were at that time) on previous day / best day. Mobile app with GPS
- Mobile app that finds strangers around you that you might want to talk to. Finds common interests, or pre-defined criteria for matches
Tue Dec. 13
- Take article, write it in style of some author or other article. (eg. Zed Shaw) ML or crowdsourced
- Software to implement “stop in the middle” technique to avoid procrastination. Gives you 5-min warning, lets you write reminders of how you were planning to finish, then stops you (less barrier to returning to work on it)
- Work bartering: I’ll program something in exchange for you designing or drawing something. More efficient allocation of resources
- Visiting a city? Enter time in city, finds what’s interesting, etc. Gives TSP-tour of places you would find more interesting
- Recommendations for places to visit. Attractions, landmarks, cities, countries, etc. Traveller’s profile
- What are people getting each other? Enter gift + recipient details. Power recommendations, ads, etc. (possible duplicate (similar) Dec. 1)
- Public ideas for A/B testing. Write description, people submit ideas, upvotes, and receive karma or money based on # of conversions of suggested alternative
- Get IP address of all sites in browser history, or as you visit sites
- FB automatic location changer

Frequently-occurring words:

My favourites

I picked some items that particularly interest me, whether it’s because I would find it fun to work on or wish it existed:

Challenge Accepted: users pay to see stunts / dares (eg. eat X chili peppers, hop on one foot while X and Y for Z distance). Acceptors upload video proof which is judged by community or moderators. Money refunded if no acceptors.
Productivity: Did I work today? Report computer usage: # of lines of code written, # of commits, what sites visited, what apps used, email usage, Reddit, terminal, chat, gaming, etc. Generate report and graphs of all this, compare with previous days. Set quotas; will warn and eventually block usage of time-wasters. Enter your commit message before coding — automatically sets this as your status message for teammates to see.
Karma: Aggregate all of it! GitHub, StackOverflow, Reddit, Hacker News, Twitter followers, Facebook likes, etc. Stats for total followers, following, graphs, etc. Add another layer of game mechanics, interesting achievements, and badges! Follow your friends and beat them! Leaderboards.
What are similar people doing, reading, or working on? Eg. What do HN readers, standing-desk users, Cubans, GEB readers, rock climbers, terminally-ill, etc. like to do? What are their age groups, ethnicities, hobbies, other characteristics? (eg. Reads Reddit? Have a blog? Contributing to StackOverflow? Doing a startup? Working at tech company? Have read book X? Age histogram? Played piano?) You fill out detailed profile and answer questions. Recommends what you could do. Recursively see what people doing each recommended item like to do. Find similar people near me or friends going to same events, and suggest / facilitate going together.
Problem: I need help finishing what I started
For beginning designers:
- I really wish there existed some platform where I could upload a screenshot (or other media: video, text, etc.) of a design I was working on, and just have people suggest ideas to make it look better or annotate and nitpick whitespace, colouring, fonts, etc. I would definitely enjoy critiquing others and may even pay to use it to get high quality feedback, or be hooked in with StackOverflow-like game mechanics.
- Votes: TopCoder for designers, or upload two images and people will select which they prefer
Programming Wiki: There really needs to be more consistent, higher-quality, and collaboratively-edited documentation for every language. Should not only document definitions, but also provide:
- a variety of good examples (hosted on GitHub/gists?)
- usage advice
- links to other references
- links to relevant StackOverflow questions
- the idiomatic/conventional way of doing something
- recipes for common tasks (such as file I/O)
- common beginner mistakes with upvotes (eg. Python’s default args evalled once; JS closure-in-for-loop gotcha)
Also in need is a consistent and high-quality set of tutorials for CS students to prepare them for industry and teach things that might not be taught in school: introduce Linux; Vim; Git and GitHub; JS; Firebug or Chrome dev tools; Python or Ruby; C or C++; a functional language; Makefile and a shell; unit testing; Apache or Nginx; DNS; setting up a server; a web framework; where / how to ask for help; how to debug; blogging and communicating; resume + interviewing; etc.
On any given term, half of all Waterloo co-op students are working while half are studying on campus. With over 100 per term working in California, we need a bike sharing program for Silicon Valley interns!
Conversation: Suggest topics from what you have talked about previously, your interests, news. Mobile app to find similarities between two users and suggests topics based on those, or find people near you that you maybe interested in talking to.
Malware / virus to force-upgrade IE users or install Chrome frame (I won’t be doing this but maybe somebody will :)
Pro-tips mobile app: eg. “Avoid that side trail”, find interesting stories near you, number of deaths near you, closest death near you. Around your neighbourhood / city: Where to eat, best place for haircut, where to get X, where Y hangs out, etc.
Mobile app for ml-class or ai-class or other Stanford online courses
Remote-activated middle finger lights on side or back of car: Would be fun to make and serve a noble purpose — that of expressing my thorough indignation at the offending party regardless of time of day — but alas, I don’t have a car and can’t drive.
Watch TV shows together over internet with live chat, polls, predictions, etc.
Collaborative brainstorming: Real-time with upvotes, game mechanics, and karma. Optionally, will show images and word associations to inspire you.
Git rebase GUI: drag-and-drop where to move branch; visually select a range of commits to squash into one commit
Study aids: collaboratively-created formula sheets. Allow forking, customizing notation, upvotes, customizations for classes. Cheat-sheet creator for a class. User-contributed mnemonics (“Every Good Boy Deserves Fudge”) for every subject (eg. anatomy) and every formula.
Word this sentence in a certain style: eg. Victorian England, gangsta, Southern, lawyerese. How would they say it? User-contributed + game mechanics + upvotes. Would be fun to train a machine learning algorithm (similar to training translation software).
Work bartering: I’ll program something in exchange for you designing or drawing something. More efficient allocation of resources
Decision-making platform: Eg. I’m looking to get a VPS but I’m completely new to this. I’ll explain my scenario, criteria, and budget, and users will comment, suggest and vote on alternatives. May also feature collaboratively-edited pros and cons, gather common scenarios, and StackOverflow-like game mechanics.
Go eat lunch with somebody close by (for those who don’t want to eat alone)
Discussion aggregator: Blog posts that have hit Hacker News or Reddit are commonly edited with a link to those comments. Why not pull in comments from those and other sources, and display them, separated by source, under the post? Could possibly also be a browser extension.
Vim show-off: plugin to save and see time-lapse of how code/article was created. Show all insertions / deletions and commands used.
Restaurants: My friends and I always have difficulty deciding where to eat. I want a daily recommendation that favours places I haven’t visited yet and eventually learns my taste from my ratings. Also, sometimes I just want to find where to get the best pineapple fried rice, and I don’t care which restaurant as long as they have delicious pineapple fried rice. Let me review, rate, and query by menu items.

Noteworthy experiment observations

It seemed to me that I had more thoughts — thus likely more ideas — after waking up to a good night’s sleep.
The inspiration for many ideas came from recent happenings. Many times my thought process would start with recounting recent events and thinking if there were any associated problems.
The previous point suggests that it may make more sense to actually record problems instead of or in addition to ideas — I’ve been taught that there are many different ways of solving a problem, but focusing on a possible solution too early may blind one from other solutions.
One cool thing was that the rate of passive ideas definitely increased. After a few weeks, I noticed that ideas would just pop into my head during my daily 40-minute bike rides to and from my internship — I was not actively generating ideas, but they would come while I was relaxed and having other thoughts. I estimate that, on average, 1-2 ideas per day came in this manner.

Now it’s your turn

I’m not claiming any of these rambling thoughts; I wrote most of these because I truly wish they existed so that I could use them, or because I would derive enjoyment from building it. Further, I’m sure many of these have already been attempted or exist (which, of course, does not preclude another competitor).

This technique of daily morning brainstorming can be applied to almost anything that would benefit from a list, and not just startup/project ideas. Here’s some ideas:

research topics
design / UI ideas
product ideas
blog topics
things to do
conversation topics with friends
gift ideas
self improvements
from a recent HN post: things to learn
this list itself: what to apply this technique to (then generate each of those lists!)

If you’re reading this blog post and others on a daily basis, then you have time to do this. Although I did this in the morning and found that a good time, you don’t have to; my friend Mack tried this when he was showering, biking, or bored and came up with a respectable list after a week. Do it during any dead time, such as in transportation, and use the method of loci to hold on to those thoughts for recording later. I’m sure you can come up with better ideas that are more relevant to your life.

The golden rule is to not filter yourself: switch off your analytic faculties and let your thoughts flow unhindered. It’s amazing what your brain will conjure if you just let it loose with unrestrained thoughts. The platinum rule is to make something of your giant list; without action, all the ideas in the world are for nought.

Give it a try. Pretty soon, you may find yourself with an endless torrent of project ideas to occupy all your free time.

On my Internship at Khan Academy

2012-01-05T00:00:00-05:00

In true Khan Academy fashion I made a video to sum up my internship, at the suggestion of our lead developer Bengineer Kamens. I was not very good at emulating Sal Khan, so the video just evolved in my own style. As I became more and more sleep-deprived, I became less and less rational, and re-recorded some parts I thought were boring. Consequently, some speech is difficult to make out, so you can just read the transcript below or enable captions.

Anyway, the video is not meant to be taken too seriously; I’m not actually that crazy in real life (but I may be if sleep-deprived :P).

(EDIT: Yes, that is actually my voice, although I’ve been told it doesn’t sound like me in real life at all.)

Once upon a time on the internet, there was rage. Sir Rage was learning geometry on Khan Academy, when he forgot to multiply pi by 2 and got a problem wrong. (He should have used tau.) Now, to be considered proficient and move on, one had to get 10 right answers in a row. Having now to restart from the very beginning, with his prior streak of 9 completely erased, caused Sir Rage to rage.

This was not the only problem at Khan Academy. Our data showed that those who took 30 problems to get a streak performed much worse after a break than those who got a streak in 10.

Those are two classification errors — requiring some users to do too many problems, while letting through some users who may need more practice.

What did we do about this?

Well, we developed a machine learning algorithm that takes into account two weighted moving averages, current streak, number of correct answers, number of wrong answers, and percent correct. These inputs are fed into a function which spits out how likely you are to get the next problem correct. Above 94% and we say you’re proficient.

Now, we make you do only 5 problems on what you know well, but give you the chance to do more practice where it’s actually needed.

For two weeks we tested on 10% of users, and found that the new model had 21% more proficiencies, 16% more exercises attempted, and a reduction from about 17 problems done per proficiency to 12.

But did we lower the bar for proficiency?

Well, it turns out that for both models accuracy is 95% after proficiency. After a break, users proficient under the new model tend to perform slightly better.

So, does this tale end happily ever after? Not yet. This is the first step in our journey to improve assessment accuracy and make users happier.

Some other things my internship spawned

To endlessly fascinate, we now have a dashboard of exercise usage statistics mixed in with random trivia!

In our iterative attempts to improve the streak, we started with a constant acceleration, momentum model, but decided to just launch a UI change to abstract away the streak and appear more forgiving. For the model changes, we wanted to use the statistical findings on our data and address the type I and II errors. That seemed like a promising application of machine learning, so I trained a Naive Bayes classifier which yielded interesting results. Comrade Jace then joined forces to produce the logistic regression model that we launched.

In the process I built a tool to allow us to better understand and evaluate different models, with pretty graphs and interactive simulations.

After the first proficiency model launch, we wanted to study long-term impact. So, I hacked together a script to download and analyze problem logs with a map reduce.

I built a UI to go through review problems, which are proficient exercises that you haven’t done in a while. From here we can add game mechanics and an optimal scheduling algorithm.

Comrade Jace and I also collaborated to produce a 55% increase in hint usage.

Now, I really hope the other interns make videos of what they did too, such as building the framework that powers all our exercises with comrade John.

Awesome things about working at KA

But you know what, I’d rather talk about why you should intern at Khan Academy, so I’ll have go at that.

The ship and her crew

Ship. Ship. Ship. Ship. Ship. Ship. Ship. Ship. Mushroom. Mushroom.

Our naval culture is one of fast iterations and continuous deployment. To see your last hour’s work on the live site in 3 minutes, impacting millions of users, has literally had me jumping up and down. I could do this any time of any day. As long as I got stuff done, how long I worked or when I came in was immaterial.

The helpful crew has been helpful. For example, Google App Engine had no A/B testing framework, so comrade Bengineer Kamens built one in a weekend. We code review all non-trivial commits to maintain a clean and consistent code base. I’ve learned a lot here, such as ML, UI design, good coding practices, blog post writing, and all the technologies we use. Also, comrade John Resig is famous!

Interns are given massive amounts of autonomy, and with that, responsibility. You’re an integral part of the core team of a non-profit startup — make decisions, flesh out projects, and shape the future of Khan Academy. You’re not just responsible for awesome code. You’re not just responsible for awesome features. Through doing these things we want to make the world a better place by redefining education.

Perks

It’s a lot of responsibility, but we have fun doing it. We’re just a few blocks away from Laser Tag. We have intense board game nights every week. Being in downtown Mountain View, we’re surrounded by a mountain of food. We pay very well for interns, competitive with Facebook. We have weekly company picnics, and you may get to have 1-on-1 walks with comrade Sal to discuss mathematics, your views on education, ask him questions, anything.

Since we’re completely open-sourced, you can build up a portfolio of cool code and projects to show your mom. I got to write a blog post as part of my job, and it hitched a free ride on the Khan Academy karma train. I’ve received many useful and insightful comments, among which are papers on which we’re planning to base our algorithm for optimal review scheduling. (Thanks commenter Johnfranklin1299!)

We also get to visit schools that have adopted Khan Academy, and it feels great to see your work directly benefiting users.

Impact

Users, our number one priority. Not shareholders, not search engines, not profits, not TechCrunch. We have over 100 million video views and close to 2 million problems done per day.

In redefining education we face some of the hardest challenges.

Why must I learn at the same rate as those my age, when I’m still struggling with basic algebra, or if I’m a second grader about to conquer calculus on Khan Academy (true story)?

Why can’t we figure out exactly how to advance each and every student past a particular concept?

Why shouldn’t learning be as engaging as World of Warcraft?

Why is learning something kids do, and not a lifelong pursuit in and of itself?

Why should quality education be denied to the underprivileged?

How many other Srīnivāsa Rāmānujans are out there, held back by their poverty, just waiting to be discovered?

I want to teach ALL the humans! Teach them ALL the things!

Education is a pillar that underlies our modern quality of life. I’m not a genius. I’m not smart enough to solve world hunger, societal prejudices, cultural clashes, war, the energy crisis, global warming, diseases, cancer. But if I can enable geniuses that do solve these problems, then I will have lived a satisfied life.

Conclusion

To make this dream a reality, we need motivated, talented developers who get things done.

What do you look for in an internship? A talented team? Experience? Learning? Money? Food? Events? Make friends? Have fun? Challenging technical problems? Impact?

Thank you Khan Academy for an awesome internship, and I hope you, dear viewer, may have that experience too.

One more thing

Oh wait a minute, there is one more thing. The newest member of our team is a flying shark! It actually swims through the air! Come and see for yourself; the shark is not a lie.

And, to make up for my inability to draw or enunciate, I highly recommend watching Sal Khan’s TED talk or Vi Hart’s videos on mathematical doodling.

Behind the scenes

Here’s all my drawings:

Here’s my setup at the office, next to the legendary Vi Hart’s desk:

At home getting some more footage with my Android phone:

How Khan Academy is using Machine Learning to Assess Student Mastery

2011-11-02T00:00:00-04:00

See discussion on Hacker News and Reddit.

The Khan Academy is well known for its extensive library of over 2600 video lessons. It should also be known for its rapidly-growing set of now 225 exercises — outnumbering stitches on a baseball — with close to 2 million problems done each day.

To determine when a student has finished a certain exercise, we award proficiency to a user who has answered at least 10 problems in a row correctly — known as a streak. Proficiency manifests itself as a gold star, a green patch on teachers’ dashboards, a requirement for some badges (eg. gain 3 proficiencies), and a bounty of “energy” points. Basically, it means we think you’ve mastered the concept and can move on in your quest to know everything.

It turns out that the streak model has serious flaws.

First, if we define proficiency as your chance of getting the next problem correct being above a certain threshold, then the streak becomes a poor binary classifier. Experiments conducted on our data showed a significant difference between students who take, say, 30 problems to get a streak vs. 10 problems right off the bat — the former group was much more likely to miss the next problem after a break than the latter.

False positives is not our only problem, but also false negatives. One of our largest source of complaints is from frustrated students who lost their streak. You get 9 correct, make a silly typo, and lose all your hard-earned progress. In other words, the streak thinks that users who have gotten 9 right and 1 wrong are at the same level as those who haven’t started.

In Search of a Better Model

These findings, presented by one of our full-time volunteers Jace, led us to investigate whether we could construct a better proficiency model. We prototyped a constant acceleration “rocketship” model (with heavy gnomes that slow you down on wrong answers), but ultimately decided that a prudent first step would be to just abstract away the streak model with the notion of “fill up the bar”.

We went from displaying the user’s current streak (bug not intended; could not find another screenshot):

to this:

and when full:

This gave us greater freedom to experiment with different underlying models without disrupting the interface.

Conversations with the team led me to conceive of applying machine learning to predict the likelihood of getting the next problem correct, and use that as the basis for a new proficiency model. Basically, if we think you’re more than % likely to get the next problem correct, for some threshold , we’ll say you’re proficient.

I started off by hacking together a naive Bayes binary classifier modified to give a probability estimate. I trained this on a few days’ worth of problem logs, and initial results were promising — the most striking being that fewer problem were needed to attain the same level of accuracy.

What do I mean by accuracy? We define it as

which is just notation desperately trying to say ”Given that we just gained proficiency, what’s the probability of getting the next problem correct?”

However, naive Bayes is typically used for classification — the task of determining which discrete category a data point belongs to — rather than for regression — returning a continuous value (in our case, a probability estimate in ).

So, our full-time volunteer Jace, who is much more versed in statistics and machine learning, used R to quickly prototype and evaluate different machine learning algorithms and feature sets. R is the de-facto programming language for statistical computing and comes pre-packaged with data analysis and machine learning tools.

To evaluate the different algorithms, input features, and thresholds, we came up with some metrics to gauge for desirable characteristics:

Mean problems done to reach proficiency — ideally we like to minimize this so that students can spend less time rote grinding on problems they know well, and move on to other concepts.
— Unfortunately, this is hard to correctly measure in our offline data set due to the streak-of-10 bias: students may loosen up after they gain proficiency and spend less time on subsequent problems.
Proficiency Rate — The percent of proficiencies attained per user-exercise pair. Again, this is hard to measure because of the streak bias.
Confusion matrix for predicted next problem correct — This is for comparing binary classifiers on their accuracy in predicting the outcome of any answer in a user’s response history. We build up the confusion matrix, and from that extract two valuable measures of the performance of a binary classifier.

We tested various models, including naive Bayes, support vector machines, a simple 10-out-of-last-11-correct model, and logistic regression. Based on the metrics above, we settled on…

Using Logistic Regression as a Proficiency Model

(Feel free to skip this section if you’re not technically inclined.)

Logistic regression is usually used as a classifier that gives a reasonable probability estimate of each category — exactly our requirement. It’s so simple, let’s derive it.

Let’s say we have the values of input features (eg. percent correct), and we stuff them into a vector . Let’s say we also happen to know how much each feature makes it more likely that the user is proficient, and stuff those weights into a vector . We can then take the weighted sum of the input features, plus a pre-determined constant to correct for any constant bias, and call that :

Now if we set , we can write that compactly as a linear algebra dot product:

Already, you can see that the higher is, the more likely the user is to be proficient. To obtain our probability estimate, all we have to do is “shrink” into the interval . We want negative values of to map into and positive values to fall in . We can do this by plugging into a sigmoid function — in particular, we’ll use the logistic function:

And that’s it! is the probability estimate that logistic regression spits out.

The tricky bit is in determining the values of the weight vector — that is, training logistic regression so that , aka. the hypothesis function in machine learning terminology, gives us a good probability estimate. For brevity I’ll spare you the details, but suffice to know that there are plenty of existing libraries to do that.

So that raises the question, which features did we use?

ewma_3 and ewma_10 — Exponentially-weighted moving average. This is just math-talk for an average where we give greater weight to more recent values. It’s handy because it can be implemented recursively as , where is the weighting factor, is the most recent value, and is the previous exponential moving average. We set to 0.333 and 0.1 for ewma_3 and ewma_10 respectively.
current_streak — This turned out to be a rather weak signal and we’ll be discarding it in favour of other features in the future.
log_num_done — . We don’t try to predict until at least one problem has been done.
log_num_missed —
percent_correct —

As for the proficiency threshold, we chose 94% based on our metrics.

Now for some Python code. To compute the exponentially-weighted moving average:

1 def exp_moving_avg(self, weight):
2     ewma = EWMA_SEED
3 
4     for i in reversed(xrange(self.total_done)):
5         ewma = weight * self.get_answer_at(i) + (1 - weight) * ewma
6 
7     return ewma

and for the actual logistic regression hypothesis function:

 1 class AccuracyModel(object):
 2 
 3     # ... snip ...
 4 
 5     def predict(self):
 6         """
 7         Returns: the probabilty of the next problem correct using logistic
 8             regression.
 9         """
10 
11         # We don't try to predict the first problem (no user-exercise history)
12         if self.total_done == 0:
13             return PROBABILITY_FIRST_PROBLEM_CORRECT
14 
15         # Get values for the feature vector X
16         ewma_3 = self.exp_moving_avg(0.333)
17         ewma_10 = self.exp_moving_avg(0.1)
18         current_streak = self.streak()
19         log_num_done = math.log(self.total_done)
20         log_num_missed = math.log(self.total_done - self.total_correct() + 1)
21         percent_correct = float(self.total_correct()) / self.total_done
22 
23         weighted_features = [
24             (ewma_3, params.EWMA_3),
25             (ewma_10, params.EWMA_10),
26             (current_streak, params.CURRENT_STREAK),
27             (log_num_done, params.LOG_NUM_DONE),
28             (log_num_missed, params.LOG_NUM_MISSED),
29             (percent_correct, params.PERCENT_CORRECT),
30         ]
31 
32         X, weight_vector = zip(*weighted_features)  # unzip the list of pairs
33 
34         return AccuracyModel.logistic_regression_predict(
35             params.INTERCEPT, weight_vector, X)
36 
37     # See http://en.wikipedia.org/wiki/Logistic_regression
38     @staticmethod
39     def logistic_regression_predict(intercept, weight_vector, X):
40         # TODO(david): Use numpy's dot product fn when we support numpy
41         dot_product = sum(itertools.imap(operator.mul, weight_vector, X))
42         z = dot_product + intercept
43 
44         return 1.0 / (1.0 + math.exp(-z))

There’s another interesting problem here — how do you display that probability value on the progress bar? We try to linearize the display and distribute it evenly across the bar. Since it’s 4 am right now, I’ll just give you the code for it (it’s well-commented) and won’t make any helpful explanatory graphs (unless people request it ;)).

 1 class InvFnExponentialNormalizer(object):
 2     """
 3     This is basically a function that takes an accuracy prediction (probability
 4     of next problem correct) and attempts to "evenly" distribute it in [0, 1]
 5     such that progress bar appears to fill up linearly.
 6 
 7     The current algorithm is as follows:
 8     Let
 9         f(n) = probabilty of next problem correct after doing n problems,
10         all of which are correct.
11     Let
12         g(x) = f^(-1)(x)
13     that is, the inverse function of f. Since f is discrete but we want g to be
14     continuous, unknown values in the domain of g will be approximated by using
15     an exponential curve to fit the known values of g. Intuitively, g(x) is a
16     function that takes your accuracy and returns how many problems correct in
17     a row it would've taken to get to that, as a real number. Thus, our
18     progress display function is just
19         h(x) = g(x) / g(consts.PROFICIENCY_ACCURACY_THRESHOLD)
20     clamped between [0, 1].
21 
22     The rationale behind this is that if you don't get any problems wrong, your
23     progress bar will increment by about the same amount each time and be full
24     right when you're proficient (i.e. reach the required accuracy threshold).
25     """
26 
27     def __init__(self, accuracy_model, proficiency_threshold):
28         X, Y = [], []
29 
30         for i in itertools.count(1):
31             accuracy_model.update(correct=True)
32             probability = accuracy_model.predict()
33 
34             X.append(probability)
35             Y.append(i)
36 
37             if probability >= proficiency_threshold:
38                 break
39 
40         self.A, self.B = exponential_fit(X, Y)
41         # normalize the function output so that it outputs 1.0 at the
42         # proficency threshold
43         self.A /= self.exponential_estimate(proficiency_threshold)
44 
45     def exponential_estimate(self, x):
46         return self.A * math.exp(self.B * x)
47 
48     def normalize(self, p_val):
49         # TODO(david): Use numpy clip
50         def clamp(value, minval, maxval):
51             return sorted((minval, value, maxval))[1]
52 
53         return clamp(self.exponential_estimate(p_val), 0.0, 1.0)

Now, until Google App Engine supports NumPy, the implementation for exponential_fit is just the derivative of the least-squares cost.

The full, uncut, unaltered code is available at our Kiln repo.

Showdown: Logistic Regression The Streak

The metrics may tell us that logistic regression wins, but being the illogical, squishy human beings that we are, we yearned for an intuitive understanding of the unique behaviour of the different models. I developed an internal tool to simulate exercise responses and visualise the prediction of different models. Here’s a highlight reel of the salient characteristics.

As expected, order matters. Both models will weigh problems done more recently higher than earlier problems. What may be surprising is the relative importance:

Logistic regression seems to care much less than streak.

Both models monotonically increase confidence the more responses of the same type they receive:

Logistic regression also recognises consistently spotty performance:

Logistic regression takes into account prior performance. So, getting lots correct is always a good thing, and you’ll be able to recover faster from a wrong answer if you were previously doing well. Contrast with the streak model, which loses all memory after a single incorrect answer.

This could also work against you. If you’ve gotten lots of wrong answers, you’ll need to do more work to convince logistic regression that you’re actually competent. This mitigates one of the issues we had with the streak, where we found that there was a significant difference in actual proficiency for those getting a streak immediately vs. after 30 problems.

Could this be overly harsh for struggling students? That’s a question we are actively investigating, and as a stopgap measure we only keep the last 20 problems as history. This compromise has an insignificant effect on logistic regression’s predictive accuracy, but it lets us sleep knowing that a student will not be damned for life if they were doing some unusual exploration and got 10 problems in a row wrong.

Due to 4 am, I don’t have an interactive demo on this page, but it won’t be hard to add it. If you would like to play with this please say so.

Results

This was a fairly large change that we, understandably, only wanted to deploy to a small subset of users. This was facilitated by Bengineer Kamen’s GAE/Bingo split-testing framework for App Engine. Crucially, it allowed us to measure conversions as a way of gathering more accurate statistics on actual usage data.

The experiment has been running for 6 days thus far with 10% of users using the new logistic regression proficiency model. Before I reveal anything else, see a screenshot of GAE/Bingo in action (from a few hours ago):

The graph above shows the results over time, so you can see when trends have stabilised.

Now what you’ve been waiting for, our current statistics (5 am PST, Nov. 2) show that, for the new model, we have:

20.8% more proficiencies earned:

15.7% more new exercises attempted:

4.4 less problems done (26% less) per proficiency:

Essentially the same accuracy at proficiency:

Higher accuracy attained among a set of 3 pre-chosen easy problems. Jace came up with this statistic to gauge any actual differences in learning. The basic idea is: If accuracy as determined by logistic regression is a good approximation of competence, then higher attained accuracies would be indicative of greater competence. Note the precipitous drop at 94% for the accuracy model — this is due to the proficiency threshold set at 94% for logistic regression, so once a user reaches that level, we tell them to move on. (A streak of 10 with no wrong answers nets an accuracy of 96.7%.)

Please enable JavaScript.

Easy Problems - Distribution of Accuracy Attained [% of Users, Accuracy Attained]

Streak: [[0.85, 0.1064], [0.9, 0.1052], [0.92, 0.1029], [0.94, 0.0973], [0.96, 0.0844]]

Accuracy: [[0.85, 0.1207], [0.9, 0.1200], [0.92, 0.1195], [0.94, 0.1181], [0.96, 0.0272]]

Slightly higher accuracy attained for a set of 10 pre-chosen hard problems. Going above and beyond the call of duty seems much less popular here, among accuracy model participants.

Please enable JavaScript.

Easy Problems - Distribution of Accuracy Attained [% of Users, Accuracy Attained]

Streak: [[0.85, 0.0464], [0.9, 0.0433], [0.92, 0.0395], [0.94, 0.0326], [0.96, 0.0193]]

Accuracy: [[0.85, 0.0473], [0.9, 0.0441], [0.92, 0.0407], [0.94, 0.0392], [0.96, 0.0025]]

P(do another problem | just answered incorrectly) not affected
11.7% more proficiencies earned for the hard problems
14.8% more proficiencies earned for the easy problems

In high level terms, we increased overall interest — more new exercises attempted, fewer problems done per proficiency — without lowering the bar for proficiency — P(next problem correct | just gained proficiency) was roughly the same for both groups. Further, it seemed that overall learning, as measured by the distribution of accuracies obtained, went up slightly under the new model.

Optimistically, we hypothesise that our gains are from moving students quicker off exercises they’re good at, while making them spend more time on concepts in which they need more practice. To confirm or deny this…

In the Pipeline

…we will look into truly where the new proficiencies are coming from. We are also interested in seeing if there is any variation in knowledge retention — in particular, we want to know if P(next problem correct | took a few days break) is affected.

This is just the end of the beginning for us. We wish to investigate and possibly implement:

Stochastic gradient descent for online learning of logistic regression
…which would allow adaptive models per user and per exercise. Should we bump up the proficiency threshold for users who find the exercises too easy?
On a similar note, could we define a fitness function that takes into account both accuracy and student frustration, and find the optimal time to tell the student to move on? Could this allow us to maximize student learning by maximizing accuracy across many exercises?
Model improvements. Here are some things we still need to try:
- Incorporate more features, such as time spent per problem, time since last problem done, and user performance on similar exercises.
- Experiment with non-linear feature transformations and combinations. Eg.
- Along with the above, apply regularization to prevent overfitting (thanks Andrew Ng and ml-class!)
- Train and use separate models for the first 5 problems vs. those after that.

This work to create an accurate predictor has many other applications than just to power the proficiency meter:

Determine if the user is struggling, and if so suggest a video to watch, using some hints, or taking a break.
Determine the optimal date to schedule a review for spaced repetition learning.
Tailor a custom mixed-question review session that addresses weak areas.

Stay tuned for a continuation blog post if we find more interesting results!

Obligatory Recruiting Plug

Think you can do better? Well, I agree! I’m sure you know lots of ways to improve what we did. Good news: we’re open-source and hiring!

We welcome contributors to our exercises and exercise framework on GitHub. Some of our best exercises were created by volunteers: check out this awesome derivative intuition exercise created by Bengineer Eater.

Another reason I love working at the Khan Academy is the passionate and talented team. Our lead developer, Bengineer Kamens, is committed to our productivity and well-being. He Bengineers internal refactorings, tools, and spends much of his time getting new developers up to speed. Without his Bengineering, it would not have been possible to collect all this interesting data. Also, if you ever have a question about jQuery, you could just ask John Resig here.

Do you want to make 0.1% improvements in ad click-thru rates for the rest of your life, or come with us and change the world of education?

Also, if you were wondering, we are not based in the UK, Canada, or Australia… my Canadian heritage compels me to spell “-our” and “-ise” when it’s not code. :P

Update (Nov 3, 2 am PST)

Thank you all for your suggestions and feedback!

There’s some interesting discussion on Hacker News and Reddit.

Update (Nov 12)

Having ran the experiment for more than two weeks now, we analysed 10 days’ data to see if longer-term knowledge retention was affected. It turns out that students are slightly more likely to answer correctly after taking a break under the new model:

P(next correct | proficient and took break) = 0.861 for streak, 0.875 for accuracy

P(next wrong | not proficient and took break) = 0.273 for streak, 0.283 for accuracy

These results are encouraging. It shows that the new model attempts to address one of the core problems with the streak — the variability of student success rates after taking a break — while at the same time increasing proficiency rates. Thus, we have reason to conclude that the accuracy model is just a better model of student proficiency.

This information gave us the confidence to roll out from 10% to 100% of users. We have now officially launched the logistic regression proficiency model site-wide!

Test Post Please Ignore

2011-08-27T00:00:00-04:00

Hello world!

This is my first blog post.

I had originally planned to write my own blog engine in Haskell, so that I could have a project through which to deeply learn a functional language and to get some practice with web development. However, I’ve noticed that the larger the task, the more I tend to put it off. So, I decided to just go with an existing framework and then transition to my own blog engine, should I find it worthwhile. Recently, I found out about Jekyll, and it fit my original design requirements very well:

Simple to set up
Store blog posts as static text files
Write and edit blog posts with my editor of choice ([gm]vim)
Markdown formatting
Publish and store blog posts on GitHub

Next I need to integrate Disqus for comments, tweak the CSS a little bit, and test on IE.

In other news, I’ll be starting my internship at the Khan Academy in two days. This is really a dream internship for me—working on something I’m deeply passionate about with a small, world-class team that will hopefully revolutionize the education industry. Can’t wait to start!

The post title is a reference to the top-scoring post on reddit.