The association of startups and growth is an incredibly useful one for the investor class.  They buy call options on a bunch of talented people, and convince them to scale their businesses as quickly as possible.  Most crack under the pressure.  At the best of times, a small company is a series of poorly arranged Jenga blocks.  If you add on new blocks as quickly as possible, most will collapse.

Only a lucky few survive long enough to reach scale and return 100x+ to their original investors.  These are the startups you've heard of - like Uber, Instacart, and Facebook.  They've made the careers of their early investors, and they've made their founders incredibly wealthy.

But they've done this by creating a system of regressive taxation.  Yes, Uber is an incredibly useful service for helping people get around.  But it's also a way to skim 40% of the wages of their drivers - some of the lowest paid people in their countries - and redirect them to the pocketbooks of the rentier class.  And sure, Facebook is a way to keep in touch with your family.  But it's also a way for the wealthy to monetize the attention of everyone else.  I guarantee you that Facebook's early investors weren't spending 41 minutes a day on the site.  

There are people making 7 and 8 figures a year at Uber and Facebook thinking about how to increase these tax rates.  For Uber, this is lowering how much drivers make.  For Facebook, this is "engaging" (read: sucking up more and more of your attention) you more.  This is what happens when your primary obligation is to your shareholders.  You just find more and more ways to squeeze.

It's through pursuing growth at all costs that we get a world where 26 people own as much wealth as 4 billion.  On that list are Elon Musk, Mark Zuckerberg, Jeff Bezos, and Larry Page.  Each of their companies has underpaid workers, knowingly flouted labor laws, and dodged taxes.  It's hard not to see the companies these men created as a form of regressive taxation.

I hope I've painted a picture of how the mantra of startup=growth can lead to horrible externalities.  So what should startups be about?

To figure this out, I think we have to step back to the very earliest days of a startup.  When it's just a couple of founders, and an idea.  What gets a startup from this point to growth in the first place?  It's ambition.  Ambition to solve a pain point for a customer.  In Uber's case, it was helping wealthy people get rides more efficiently.  For Facebook, it was connecting kids on college campuses.

To grow, a small company needs to serve its core customer incredibly well.  So ambition comes before growth.  Would Facebook have been less ambitious if they'd just focused on connecting college students to each other?  I'd argue no, because there is a lot of depth there - everything from better events to dating to textbooks.  Serving a small audience with depth isn't inherently less ambitious than serving a larger audience more shallowly.  It requires just as much product innovation, and is more impactful to the audience.  Think of a teachers.  It's hard to argue that teachers aren't making an impact - they make an enormous impact on the lives of students.  It's not an impact that scales.  But that doesn't make it any less meaningful.

Framing startups in terms of growth means that companies eventually subjugate their impulse to serve their customer better to their impulse to serve their shareholders better.  Facebook could have built a monetizable service around the needs of college students.  But the company chose to chase growth over serving its customers better.  And this growth has ultimately been negative for the world.

It's time for a new definition of startups.  One framed in terms of ambition.  Startups focus on being the best at solving a particular customer problem.  And if growing faster means sacrificing that ambition, then a startup chooses ambition over growth.

When I started Dataquest, I didn't start with a top-down plan of how I was going to build a business.  Instead, I started with a problem - in this case, that data science education wasn't focusing on teaching real-world skills.  Most technical people that I talk to start a business this way.

As Dataquest started to scale and get thousands of users, I knew that I was solving at least some of this problem.  But solving a problem isn't a binary endeavor.  It's not either solved or unsolved.  In my case, I wanted to solve the problem for more people, in more topic areas.  This meant adding more breadth and depth to the product I had.

To do this, I had to scale the company.  This is distinct from scaling the product.  The company is the infrastructure that builds the product.  This point is where things started to get messy.

Reading online about how to scale a small software company inevitably leads advice from VCs and accelerators.  I personally read a ton of advice from YCombinator.  The advice says to raise a seed round, then keep raising more money until you IPO (my summary is reductionist, but generally accurate).

There's nothing wrong with this way of building a business.  The important thing I'd add, though, is that it's a way not the way.  Unfortunately, it's often presented as the only way to scale.

Following this advice meant that I turned my focus from how to solve a problem to how to convince investors I could solve that problem.  These are unfortunately two very distinct skills.  You can be great at solving a problem without being great at convincing investors, and vice versa.  

I see many other technical founders fall into this trap.  I think it's because we're seeking determinism in the act of building a company.  When you're coding and building, there are relatively straightforward ways to know pretty quickly if you're doing the right thing.  While the same thing doesn't exist for building a business, advice from investors and accelerators can make the problem seem deterministic - "if you follow this checklist, you'll build a successful company".

Inevitably, that particular checklist means that the goal becomes raising money in a highly structured way.  At some point, the way you're building the business and what you want to build the business for no longer align.

To combat this, my main advice is to first figure out what type of company you want to build, then work backwards to figuring out the way you want to get there.  

This is another principle that seems quite simple, but is actually difficult to nail down.  Figuring this out involves figuring out what you really value.  Do you want to build a really effective product for a small group of people?  Do you want to build the biggest business possible and reach as many people as possible?  Do you want to be a manager or a builder?

It's only through answering these questions that you can figure out what type of business you want to build.  And you're the only one who can figure that out.

In my next post, I'm going to dive into more depth around how to figure out what type of business you want to build.

It's hard enough to run a bootstrapped startup or small business in normal times, but COVID-19 has made things 10x harder. As the CEO of Dataquest, a 30-person startup, it's been challenging for me to figure out how to support our team and customers.

In the last couple of weeks, I've spent a lot of time researching and planning, and I'm sharing what I've learned in the hope that it will help you.  In this post, I'll share how to navigate COVID-19 from 3 perspectives:

• Impact on your team.  Many people are having to suddenly change the ways in which they work while dealing with anxiety about the future.  This includes working remotely, health impacts, and working around disruptions to the day to day.
• Business impact.  Revenue in a lot of industries is dropping quickly, and being able to forecast and plan will help many businesses weather the next year or two.  This includes forecasting revenue, extending runway, and adjusting strategy.
• Impact on your customers.  Millions of people are impacted around the world, and your business may be a major part of their lives.  Engaging with your customers will help you build trust.  This includes clear communication, giving benefits to customers, and connecting with the community.

If you want to skip to a specific section, you can use these links to navigate:

• Background
• Team impact
  • Remote work
  • Health impacts
  • Disruptions to daily life
• Financial impact
  • Forecasting
  • Extending runway
• Customer impact

Background

Before I dive into advice, it's important to align on the parameters of the situation.  Businesses dealing with COVID-19 really have two different events to navigate; the pandemic, and the recession caused by the measures to contain the spread of the pandemic.

Shelter In Place

To deal with the pandemic first, it seems likely that it will take 2-3 months of lockdown measures to halt the progress of COVID-19 in most countries.  Looking at this chart, we can see that it took about 2 months in China:

The quarantine in Wuhan started on January 23rd, and will end on April 8th.  This is about 2.5 months of quarantine.

This may be a bit optimistic in other countries, like the US, where cases are growing much more quickly.  

As this chart shows, it's possible that some level of lockdown may stay in place through the summer.  It's early, but it also seems likely that there will be some kind of resurgence in the fall, once the lockdown measures end.

However we look at it, it's best to plan for some level of travel restrictions and shelter in place orders through the end of the year.

Economic Impact

The other point to align on is the economic impact of COVID-19.

China's GDP likely contracted 10% in the first quarter of this year, taking into account the shelter in place orders in February and March.  It's unclear how quickly the recovery will happen, but businesses are still not back up to 100% now.

Goldman Sachs forecasts a large contraction in the US economy in Q2, with a gradual recovery:

After the 2008 financial crisis, the economy didn't fully recover until 2011.  It's likely that recovery this time will be gradual as well.  A fast recovery will be prevented by 3 factors:

• Ongoing outbreaks in other countries reducing trade
• Fear of going back to work in-person
• Ongoing shelter in place restrictions

For a small business, it's important to plan for this being a 1-2 year event.  This is painful to think about, but it's better to plan for the worst case than to be too optimistic.

Team Impact

Now that we're hopefully aligned on the background, let's dive into the impact this will have on your team.

There are a few categories to think about:

• Transitioning to remote working
• Health impacts
• Disruptions to normal life

Remote Work

Millions of people are suddenly transitioning to remote work, and tools like Zoom and Slack are seeing massive surges in usage.

But working effectively remotely is about more than tools - it's also about adapting the processes you to use to coordinate your work.

Sharing context

When you're an in-person company, context can get shared implicitly, through casual conversations and brainstorming session.

This isn't easily possible online - you have to be explicit about context.  We were a remote company before COVID-19, and we use a few strategies to overcome with this:

• We document everything across the company using Notion.  Each team can access documentation from other teams.
• We send out a regular internal newsletter that contains important updates, like revenue, key metrics, and customer stories.
• We default to written communication first for sharing context.  This means that we don't hold meetings just to share context - we use the meetings to reinforce context that was already communicated in writing.
• We share notes from all of our meetings with the rest of the company.

Async-first

In person, it's easy to use asynchronous communication channels, like Slack, as a secondary communication channel.  In a remote company, it's important to switch to async-first, where meetings are only used for critical decisions or discussion, and async is used for everything else.

It's also important to set boundaries - Slack can easily turn into a mess of notifications, which interrupt focus time for everyone.

• We setup hours where people shouldn't be responsive on Slack (this varies by timezone).  This is important to ensure that people aren't always-on.
• We have a "standard meeting calendar" where certain types of meetings can only happen at certain times.  This prevents people's calendars from getting clogged with meetings.

• We have agendas for all of our meetings, so it's clear why we're meeting, and what types of decisions (if any) we're making.

Soliciting feedback

In-person, it's easy to tell how people are feeling, and adjust how you manage.  It's much harder to do this remotely.  We use a few tools to ensure that we're able to effectively manage and get feedback remotely:

• We use 15Five to structure our manager 1:1s, and do a weekly "pulse check" to see what challenges people across the company are facing.
• We do quarterly CultureAmp surveys to surface latent concerns and address them.
• We do a regular leadership team Q&A so people can share their concerns. (some of these are anonymous)
• We do check-ins at the beginning and the end of meetings to ask how people are feeling

Building connections

It can be a lot harder to build connections with others in a remote environment.  There are a few things we do to help with this:

• We match people across the company up weekly for social calls
• We have Slack channels for interests like books, scifi, pets, and more

• We do virtual cards and small gifts for birthdays, anniversaries, etc.

Health Impacts

The second area to think about is health impacts on your team.  This is a time when many people are sick, or are anxious about themselves or their friends and family getting sick.  There are steps you can take to give your team the time and space they need to take care of themselves and their loved ones:

• PTO/time off policies
• Cancelling travel
• Expanding benefits

Time off policies

This is a time to think about expanding your PTO policy, and/or making it more flexible.  People may have more unexpected days off, or need more days to recharge during this time.  Consider adopting a flexible time work schedule or time off policy during the next few months.

Regulations here vary by country, but the US just passed the FFCRA, which may apply to your company.  For companies under 500 employees, with certain exceptions, the Department of Labor website states that businesses need to give:

• Two weeks of paid sick leave at the employee’s regular rate of pay where the employee is unable to work because the employee is quarantined and/or experiencing COVID-19 symptoms
• Two weeks of paid sick leave at two-thirds the employee’s regular rate of pay because the employee is unable to work because of a bona fide need to care for an individual subject to quarantine, or to care for a child  whose school or child care provider is closed or unavailable for reasons related to COVID-19
• Up to an additional 10 weeks of paid expanded family and medical leave at two-thirds the employee’s regular rate of pay where an employee, is unable to work due to a bona fide need for leave to care for a child whose school or child care provider is closed or unavailable for reasons related to COVID-19

Employers will be compensated for this time off, and will be able to deduct it from their payroll taxes.  The mechanism is unclear, but the IRS is working on it now.

If your business has less than 50 people, it is possible to apply for an exemption, although it's currently unclear how to do that.  You can read more about the FFCRA here.

Cancelling travel

Hopefully, I convinced you in the background section that there will be disruptions to daily life from COVID-19 through the end of the year.  At Dataquest, we've cancelled travel through at least the end of the summer, and are reevaluating travel for the remainder of the year.

One option is to convert in-person meetings, such as offsites, into remote meetings.

Expanding benefits

Something else to consider is expanding benefits during the next few months.  There are mental health and telemedicine services that might be useful to your team, such as:

• Doctor on Demand
• Modern Health
• Talkspace

In a time of anxiety, benefits like this can make a big difference.  In cities around the world, people are going outside less, which can have mental health consequences:

Mental health benefits can help with the general stress of remote working and remote living.

Disruptions to normal life

Many people are having to shelter in place, and care for loved ones during the day.  You can support folks in a few ways:

• Normalizing schedule changes
• Creating "safe spaces"

Normalizing schedule changes

Many parents are having to take care of their children during the day, because schools and daycares are closed.  Many others are having to take care of elderly relatives.

To help these people feel supported, consider:

• Enabling people to shift their hours.  Schedules may need to shift on a weekly or daily basis, especially when coordinated childcare.
• Enabling people to skip meetings.  If you're documenting agendas and notes, it becomes more "normal" to skip meetings, and there is less fear of missing important context.
• Proactively checking in on team members to see how they're holding up.  This can make a big difference, and can surface latent concerns.
• Normalizing disruptions during meetings.  At Dataquest, many people now have kids and/or significant others popping in on calls.

Creating "safe spaces"

It can be hard to go through an anxiety-inducing time alone.  Many people have concerns other than work on their minds.  Consider:

• Creating a dedicated covid-19 channel in Slack for people to share how they're feeling, and tips for managing anxiety and schedule disruptions.
• Proactively sharing your own concerns can create space for others to share theirs.

Financial Impact

Many businesses around the world have been impacted, or will be impacted, by COVID-19.  This covers both this "lockdown" period, and the longer-term recession that will come about as a result.

Small business revenue has declined significantly in the past few weeks in the US:

It's hard for any business to continue when it loses a portion of revenue.  It's critical to work as quickly as possible to understand and plan for the next 12 months.

In this section, I'll discuss:

• Forecasting out the revenue impacts
• Extending your runway
• Adjusting your strategy

Forecasting revenue impacts

It's critical to have good forecasting, so you can make good decisions.  You're unlikely to make perfect forecasts, but try to plan out 3-4 scenarions that span a range of possibilities from worst case to best case.

To do this, you'll need to:

• Have accurate data
• Find reasonable comparisons
• Do scenario planning

Get accurate data

Ideally, you have an accountant, and use an online tool like Xero or Quickbooks to keep track of expenses.

Look at your profit and loss statements for the last 12 months.  Then do the following to forecast out the next 12 months:

• Add in any future spending you've committed to, like hiring and new tool usage.
• Take into account annual raises.
• Make sure to properly attribute one-time expenses, like annual subscriptions (these will create "spiky" expenses in future months that you may not have accounted for).

This link has a basic financial planning template if you need more information.  Startup Runway and LivePlan are also tools that can help you model and forecast expenses.

If your business has a SaaS model, I'd also recommend setting up a tool like Baremetrics or ChartMogul.  These tools will help you understand and forecast your revenue.

Find reasonable comparisons

You need to find comparisons for both what will happen during the shelter in place, and what will happen afterwards due to the recession.

For the shelter in place part, I'd recommend researching how your sector is doing in terms of revenue in areas that have been locked down, or are sheltering in place.  For example, if you're in an area where the lockdowns have just started, you can look to see what happened in areas that have been under lockdown for some time, like California, Wuhan, Spain, and Italy.

For the recession part, I'd recommend looking at how public companies in your sector did during the last recession.  In our case, we're in online education, and were able to look at companies like Pluralsight, Lynda, Capella, and Rosetta Stone.

These comparisons will help you make more realistic estimates about what will happen to your revenue in the coming months.

Doing scenario planning

Once you have accurate expense and revenue data, and have a rough idea of how you'll be impacted in the next year or so, you can start to do some scenario planning.

• Based on your data, what's the "realistic" worst case?  Obviously, worst case is losing all of your revenue for the next year, but that might be very improbable.  What's the 5th percentile case for your revenue in the next year? (ie, there is a 95% chance your revenue will be at this level or higher in the next year)
• What's the average case for your revenue?
• For the purposes of this exercise, it's not completely necessary to plan for the best case, but you might want to do it, just in case.

Extending your runway

Once you understand your "realistic" worst case, you can plan for how to get enough runway to avoid that case.  First, determine how much additional money you'll need to make it through the next 12 months in the worst case.

Then, you'll need to:

• Cut expenses
• Get loans and/or grants
• Adjust your strategy

Cut expenses

Cutting expenses in a small company is a very hard decision.  It's likely that you don't have a lot of room to cut expenses without impacting your business.

For example, we've prioritized keeping the team together, and we're cutting other expenses, like hosting costs and other tools.

You'll need to audit all of your expenses, and categorize them by how important they are to your business.

Since you already know how much you'll need to cut, you'll be able to work through this list and figure out what expenses to cut.  Don't forget to cut future expenses if necessary, like hires your have planned on your roadmap.

If you have to conduct layoffs, many countries are offering enhanced unemployment benefits.  In the US, several states are offering enhanced unemployment benefits.  The federal government is also giving an extra $600 a week to people who are unemployed.

Get loans or grants

There are many loan options that are becoming available.  In the US, the SBA is offering economic injury and disaster loans (EIDL) and enhanced 7a loans, called the paycheck protection program (PPP).  You can read more about them here.

Here's the breakdown:

• Paycheck protection program loans are 100% federally guaranteed loans that require no collateral or personal guarantee.  You can borrow up to 2.5x your average monthly payroll from February 15th, 2019, to June 30th, 2019.  Some portion of this loan can be forgiven if you use it for payroll.  Otherwise, it turns into 10 year term loan with 4% interest.
• Economic injury and disaster loans can be up to $2 million, and are to cover costs, like payroll, that now cannot be met due to COVID-19.  If you apply, you can get an automatic $10k grant within 3 days.

You can apply for both programs, although you'll need to roll the grant into the PPP if you get an EIDL, then a PPP loan.  You can apply for PPP loans through any SBA 7a lender, such as Wells Fargo and Chase Bank.  You can apply for EIDL on the SBA website here.

Some states and cities in the US are also offering grants, which you can read about here.

Finally, some private entities are offering grants, including:

• Facebook, which is offering ad credits.
• Google, which is also offering ad credits.

Adjust your strategy

Per Good Strategy/Bad Strategy, which I highly recommend reading, strategy is about:

• A diagnosis of a challenge or opportunity
• A guiding plan to work through the challenge
• A set of actions to work towards the plans

COVID-19 has changed many of our background assumptions about the world, and thus makes most diagnoses obselete.  Strategy helps us apply the limited resources of a startup towards the places where they'll create the most leverage.

Without a correct diagnosis, you won't have an effective strategy.  You may have set your annual strategy just a couple of weeks ago, but it's probably obsolete already.  Think about how your customers might shift in the next year.  For example, if your company does something related to delivery, will this spike in deliveries last?  How might shelter in place interact with the recession to change demand?  

On the other hand, if your business is related to restaurants, how long might the lockdown last?  Post-lockdown, how might spending patterns change?

Customer Impact

COVID-19 is affecting most people around the world - a group that probably includes your customers.  There are many ways you can help support your customers through this time, including:

• Clear communication how your business will be impacted
• Giving benefits to people affected by COVID-19
• Helping your community connect

Clear communication

If your business will be impacted, it's very important to message your customers about this.  People may rely on your services day to day, and being proactive in messaging will help to build trust.

One example is this message from UPS:

Giving benefits

The world has changed a lot in the past few weeks:

• Many people have been laid off
• Many are in lockdown, and can't go outside
• The ways we all work have changed

Companies that offer tools for remote working have a lot of expertise to share in times like this.  One example is this email from BambooHR:

If you have expertise that could be valuable to your customers at a time like this, sharing it will help you add value to people in what is a pretty dark time for many.

At Dataquest, we're working on extending existing scholarships for our students, and offering new scholarships to students affected by COVID-19.

Community connection

Finally, if you have a customer community, don't neglect to help your customers connect to each other and share their advice and experiences.  Your customers will be able to support each other in ways that you can't.

You can get through this

This is a dark time for small businesses worldwide.  We have to help our teams and customers navigate this pandemic, while potentially dealing with falling revenue.  To add to the stress, many governments are prioritizing helping big businesses over small businesses.

In writing this guide, I hope that sharing what's worked for me will help you through this time.  Small businesses are critical to the world, and I hope that you, and your team, are able to get through this time.

The success criteria for your role suddenly change, and in many ways, you're responsible for defining them.  Even if you were good at what you did previously, you basically have to transition into a completely new role.

It's common when you hit a state change like this to seek out resources like books and advisors.  The problem is that these resources often increase the feeling of being overwhelmed.  The Effective Executive has dozens of precepts that you have to follow to succeed as a leader.  High Output Management has multiple new frameworks that you'll need to implement to succeed.  Radical Candor asks you to reshape how you approach giving feedback.  And on and on as you read more management books.

Even advisors can't always help you cut through the feeling of having to complete dozens of items on a checklist to succeed.  Every advisor has a slightly different management philosophy honed over years.  Thus, advisors often forget how hard it is to be a new manager, and add dozens more items to your list.

Not to say that these resources are bad.  On the contrary, they're essential to succeeding as a manager.  The key is to figure out how to prioritize the skills you need to develop to succeed as a manager and tackle them in small chunks.

As a solo technical founder, I've faced this problem multiple times as Dataquest has grown from just me to 30+ people.  And I haven't always succeeded.  

Through my failures (and my successes), I've identified a framework for how to rapidly and effectively build management skills.  In this post, I'll share how you can stop feeling overwhelmed and start improving.

Step 1: Put yourself in a situation where you need to grow your management skills

I firmly believe that it's impossible to learn management skills in a vacuum.  You need a context in which you're able to continuously practice what you're learning and get feedback.

This means that if you're on the management track, but aren't yet a manager, you should find a way to start taking some management responsibility.  Can you take over 1:1s with some of your managers reports?  Is there a way for you to lead some of the strategy and planning of your team?

Part of this is also tailoring the resources you seek out to your level as a manager.  For example, if you're a completely new manager, The Making of a Manager is a good book to read, but The Effective Executive probably isn't.  Similarly, finding advisors who have led or are leading small teams will be more productive than reaching out to Fortune 500 CEOs.  Look for resources that are "just ahead" of where you are.

Step 2: Gather data

As a new manager, or an existing manager stretching your skills, you're going to make mistakes.  A lot of mistakes.  And that's okay.  We often think of management as a "you have it or you don't" skill, but the truth is that management skills are built over years.  Being a successful programmer, designer, or marketer doesn't mean that you're going to be a successful manager right off the bat.

As you make these mistakes, it's important to gather data on what you're doing correctly, and what you're not.  This is the only way to identify areas to improve.  Remember that being a successful manager is about the long-term and short-term output of your team.  The only way to judge how effective you are at this is to ask your reports, and the person you report to.

There are three main ways to gather feedback:

• Structured surveys, like 360s
• 1:1s
• "Just-in-time" feedback

Structured surveys

I highly recommend doing periodic 360s to check in on your performance.  I recommend doing both anonymous and non-anonymous surveys.  They will both give you different signals, and the anonymous surveys might uncover trust issues on your team.

At Dataquest, we use 15Five to do 4 annual 360 surveys.  The specific tool is less important than making sure that you do them on a regular cadence, and genuinely reflect and internalize the feedback with an open mind.

We also do 1-2 annual anonymous 360 surveys.  We do this though a coaching platform that we use, Torch, but there are many other ways to do these.  Torch breaks down the anonymous feedback along skill axes, which helps you prioritize what to improve:

This anonymous feedback can help identify blind spots, and helps people feel more safe to give feedback.  This is especially useful when you haven't yet built trust with your team.

1:1s

1:1s are a great place to solicit and get more context on feedback.  I recommend asking a consistent set of questions to people on your team weekly, and then using them to create a 1:1 agenda.  We also use 15Five for this at Dataquest.

In the weekly set of questions, I specifically ask for critical feedback:

Asking more freeform questions about your performance and building understanding during 1:1s is also very valuable.

Just-in-time feedback

This is very hard to do if you don't have bidirectional trust with your team, so I don't recommend doing it if you're new to management.  But asking for feedback during a meeting, or just after a meeting, can be a great way to get actionable feedback.  The context is still fresh in everyone's mind, and you can quickly get to the heart of concerns.

Step 3: Prioritize

If you're like me, at this point, you'll have a long list of things to work on.  It may seem like all of these concerns are urgent, and that you need to work on them immediately.  Resist that impulse.  Similarly to a team that is overloaded with too many priorities, if you try to improve everything at the same time, you won't make any progress.

At this stage, I recommend narrowing your focus down to 1-2 key areas.  For example, my goals for the last 3 months were to improve my facilitation, and how well I listened and internalized feedback.

It can be extremely hard to prioritize, but luckily, you have the data that will help you do so.  What themes come up over and over in your feedback?  What would make the biggest impact on the team?

If you have advisors in place, they can be extremely helpful with this question.  Using your team as a sounding board can also be very productive (if you have the right level of trust, of course).

Step 4: Find the right resources

I categorize resources into three groups: advisors, books, and coaches.  They're all useful in different ways, and I'll walk through how you can maximize the effectiveness of each.

Advisors

Advisors are people who have managed before.  Getting their perspective can be extremely important in identifying your strengths and weaknesses as a leader, and narrowing down your focus areas.

Advisors can also be very helpful for walking through tactical problems that you're having in management, and brainstorming how you can improve.

Advisors are most useful for perfecting the craft of management - how you should structure your time, how you should relate to your team, and so on.  The downside of advisors is that they often lack specific context on your team.

One commonly overlooked source of advisors is others within your company.  It's not uncommon for your peers or direct reports to have more expertise in a specific area than you.  Seeking them out and asking for advice can be a fast way to improve, since they already have built-in context.

Books

Management books can be extremely useful, if you fit them to your goals, and not the other way around.  For example, if you've identified giving feedback as a focus area, then reading Radical Candor might be helpful for you.  

As I mentioned earlier in this post, reading management books can be overwhelming, and it's the process of fitting the books to your goals that makes them more manageable.

Coaches

Leadership coaches can be extremely valuable to growing your management skills.  This is because management is about how you relate to others.  In order to be able to relate to others, you also have to understand how to relate to yourself.

Coaching and therapy are closely related.  I've found that a therapist helps you explore the landscape of your mind and your emotions, and increase your awareness.  A coach helps you operationalize this awareness, by helping you relate to the outside world (your reports and your manager) more effectively.  Coaches also help you dig underneath the surface, and discover what is causing your strengths and weaknesses in management.  It may seem very fuzzy, but I've found it extremely effective.

For me, most of the barriers to being an effective manager weren't external factors - they were all about my mindset and awareness.  Additionally, you'll often need to coach your team members and help them improve their skills.  Being able to learn the frameworks of coaching can be invaluable in this.

If you have the time and money, I highly recommend getting a coach and a therapist.

Step 5: Test

At this point, you understand your focus areas, and you've gathered a bunch of ideas on how to address them from your resources.

Once you've done that, it's a matter of experimenting, and seeing what works for you.  It's very similar to a product development process - not every feature is going to work, but it's through experimentation and measurement that you find the right course.

I recommend trying to keep the feedback loop here as tight as possible.  If you try a new way to run a meeting, ask for feedback as soon as possible.  If you try a different way of structuring your 1:1s, ask how it went at the end.

If an experiment is working, try consulting your resources to see how you can double down on it.  If it isn't consider dropping it and moving on to the next idea.

Progress won't come instantly, but if you try these experiments for a few months, you'll be amazed at how far you'll go.

Step 6: Keep improving

As you try your experiments, you should be constantly gathering data.  Eventually, you'll hit a point where you feel like you've improved your focus areas enough.  Once you hit this point, it's time to move on to new focus areas.  Go back to the gather data step, pick 2 new focus areas, and continue from there.

The fun and the pain of management is that you're never done improving.  There will always be new challenges - new team members, new ways of working, and new responsibilities.  It's only by continuously improving that you can stay ahead of these and keep succeeding.

Becoming a better manager

I hope that this framework helps you become a better manager.  I personally have found them extremely valuable.  If you have any questions or comments, please reach out to me on Twitter @vikparuchuri.

To ramp up the difficulty even more, the criteria for whether or not you're succeeding suddenly change once you become a manager.  In your previous role, you could be successful even if your team wasn't.  As a manager, you're suddenly managed on the quality of work of your entire team.

If you were lucky, you had some time to apprentice under a successful manager, and picked up some lessons.  If you weren't, you're trying to cobble together lessons from books, blog posts, and other managers to make it work.

Either way, it quickly becomes obvious that there's a steep learning curve ahead.  Once you get to this point, I think there are two paths you can take:

• You can try to get everything right off the bat, and be "perfect"
• You can accept that you aren't going to be the best manager immediately, and focus on learning as fast as you can

I made the mistake of taking the first route when I first became a manager.  I had been an an engineer before I became a manager.  As engineers, we're conditioned to understand that becoming an excellent engineer takes time - it's hard to architect amazing systems when you're just a couple of years in.

But it wasn't clear to be at the time that the same thing is true of management.  Most books you read on management talk about what it means to be a good manager, but very few talk about how long of a road it will be to get there.

Thus, it's easy to fall into the trap of feeling like you have to follow all of the advice you read and be "perfect" to succeed.  If you've read more than a couple of management books, you know that there are a seemingly endless list of things you need to do to be a "good" manager.  You have to do everything from set the vision to plan roadmaps to coach employees.

Thinking that you have to start doing all of these things at once can make the job of manager overwhelming.  When we're overwhelmed, it's easy to get into reactive mode, and jump from crisis to crisis.

I fell into this trap for a while.  I was doing the things that a manager should be doing (1:1s, performance reviews, roadmaps, etc), but I wasn't getting the right results.  The output of the team was low, and we weren't aligning on goals.  I was stuck, and didn't know how to get better as a manager.  I thought I was doing all of the things I was supposed to.

Over time, I realized that this was because I was doing too much.  I realized that trying to be a perfect manager had led me to follow a todo list without actually considering the core of the role, and the impact I wanted to have.

I realized that by doing less, I could be intentional about what skills I wanted to build over time.  Instead of being perfect in every area, I could accept that I wasn't going to be great in some areas for a while, and progressively build my abilities in them over time.

For example, 6 months ago, I wasn't great at setting OKRs.  I set a 3-month goal to get better at setting OKRs, which led to me getting much better in that area.  In the past few months, I got feedback that I wasn't doing well at facilitation, and have been able to focus on improving my skills there.

The only way to do this was to accept that there are things that I'm doing poorly as a manager, and seek out feedback on what those areas were.  It's possible to improve as an individual contributor without feedback (although it's very hard).  As a manager, your job is to get people working together effectively towards a goal.  Without feedback, it's impossible to know your impact on those people.  

Feedback is the best tool you have to improve as a manager.  It's only through accepting that you're not doing everything perfectly that you can set the stage to receive this feedback and grow.

I think the most important thing any manager can do is recognize that getting good at management will be a long road.  Continuously getting feedback, learning to identify areas of growth, and improving a couple of areas every few months will work much better than trying to get everything correct right off the bat.

As a result of my own disinterest, the system seemed to write me off.  When applying to summer internships, one teacher wrote an anti-recommendation for me, warning them not to select me.  My parents had high expectations for colleges for me – Harvard, MIT or bust.  To say this was unrealistic would be an underestimate.  My guidance counselor told me that getting into college, period, would be a stretch.

My high school being torn down (via Laurie DeWitt⁄The Gazette)

I applied to around 15 colleges, and got into 2 – the University of Maryland (state school for me), and UC Irvine.  I ended up going to Maryland as part of the class of 2008.  I’m from an Indian-American family that puts a lot of stock in scholastic achievement, so this was a constant source of shame to me for years.  I would tell people “no I got waitlisted to school X”, or “I actually chose Maryland because of low tuition”.  In retrospect, it’s a silly attitude, but I was young, and put a lot of stock in appearances.

At Maryland, I wasn’t sure what I wanted to major in.  My parents wanted me to become a doctor.  I wasn’t very interested in it, but I took some pre-med classes anyways.  I chose general studies as a major (the major for people who don’t know what they want to major in), and started fulfilling my basic requirements.  College is really unstructured, and I lapsed into a mode where I didn’t go to class at all and didn’t do homework.  Most classes were boring, and I can’t learn at all in a lecture setting.  Whenever I did go to class, I just ended up counting the minutes until I could become free again.  I figured that learning was just not something I was good at.  I enjoyed a few classes, like microbiology, where we grew bacteria cultures in petri dishes, history, and math classes.  I only discovered I liked multivariable calculus after I stopped going to lecture – I went to class every day for the first third, and got a 68% on the test.  I switched to staying home and reading the book before tests, and started getting As.

UMD has a pretty nice campus

I never really found my passion in college, and college didn’t provide the tools to help me do it (or maybe it did, and I never looked for them).  This meant that I drifted for a year in general studies, and then switched to american history when it became obvious that my grades would never let me get into a med school.  It’s really easy to get depressed and think that there’s something wrong with you when you’re conditioned to think that grades are everything, and you have bad grades.

A couple of years into college, I decided that I needed to pursue my own destiny in life, and part of that was paying for college and my own living expenses.  There aren’t many jobs out there for college students with low GPAs, but manual labor is always hiring.  I went to work at UPS as a loader.  My starting pay was around 8 dollars an hour.  I always tell people “I threw boxes into trucks”.  Boxes came down a conveyor belt really quickly, and we took them off the belt, scanned them, and stacked them up inside semi trailers that would go out to other UPS “hubs”.

This is a “wall”, and how you load trucks at UPS.  This is a nice one – most look pretty bad.

I eventually was promoted to “part-time supervisor”, which meant that I was responsible for running an area, which consisted of a few loaders.  I usually worked about 30 hours a week, and was really exhausted the rest of the time, so I went to class less that usual.  On the plus side, UPS paid half my tuition with their reimbursement program, and I had enough money to pay my living expenses and even save a bit.  I figured that since my grades were really bad, I’d just keep working at UPS after college.

The next two years went by in the same vein.  I enjoyed a few of my history classes, notably my capstone where I got to study the reconstruction of Japan post-WW2.  When I graduated in 2008, I had a 2.1 GPA.  I was almost on academic probation a couple of times, but managed to skate out of it by putting in a bit more effort in a class.

My options were pretty bleak post-graduation.  The higher-ups at UPS wanted me to stay, but they thought I had an attitude problem that I needed to fix before they were willing to promote me (most policies at UPS are really backward and inane, and resulted in people I was supervising getting shafted, so I got upset a lot).

The tragic thing about manual labor is that your peak earning potential is when you’re in your 20s.  People who’d been working at UPS for more than 10 years had back and knee issues.  I got injured a couple of times (banged my head on a pole and got dizzy for a while, later cut my knee really badly, supervisors pressured me not to report it), and my knees weren’t in great shape, so staying at UPS probably would have been a bad bet.  I learned a lot there, about hard work, motivation, and managing other people, but it wasn’t a good long-term fit.

I ended up getting a job at Pepsi as an operations manager, thanks to a friend (one reason college is valuable!).  Around when I started, I also applied to the US Foreign Service on a whim.  There aren’t many jobs that will accept people with low college GPAs and no “hard” skills, but getting into the foreign service consists of passing tests, so it seemed like a good fit (plus I was interested in it from studying the Japanese reconstruction).

The inside of a pepsi warehouse looks something like this.

I ended up getting into the Foreign Service, although I eventually found that it also wasn’t a good long-term fit.  I started learning coding towards the tail end of my foreign service life.  You can read more about that experience here.  I left in late 2011.

I was posted to Georgetown, Guyana in the foreign service.

Coding is my passion, and I only really discovered it by going through years of switching jobs and wondering if I would ever find something interesting.  But in retrospect, there were signs.  I spent every day riding the bus to school in middle school learning to program my ti-83 calculator, and then making a simple game where a monster chased you around the screen.  I made scripts to automate certain tasks when I was doing summer internships.  I played an online game at one point, and hacked together some simple scripts to help me manage things.  But I didn’t realize that scripting could ever amount to anything more than a way to make a “real job” slightly quicker and simpler.  My dad also learned programming, but he worked at big consulting firms, and did work that neither he or I found especially interesting.  I thought that is what “real programming” was, and that all the fun scripts that I was making were sideshows.

Even when I left the Foreign Service, came back to the US, and kept learning to code, I didn’t have a clear end path.  I just knew that I was doing something that deeply interested me, so I kept doing it.  I wanted to predict the stock market, but didn’t really know how.  I spent time reading up on C# and Ruby, and interfacing with stock price engines.  I ended up making a terrible “algorithm” that would have hard-coded rules (if the stock went up for the past week, sell it today).  I remember my excitement at discovering that I could write network code to have some machines be “workers” and one machine as a control.

The engines never amounted to anything, but my interest in stock price prediction led me to machine learning, which led me to Kaggle.  Kaggle is a competition platform for machine learning.  You get to make algorithms, which compete against algorithms made by other people around the world.  It’s a great method for alternative credentialing – you can say “I don’t have a degree from MIT, but I did beat a couple of phds in a Kaggle competition”.

Kaggle was a perfect place for me at the time.  I spent my days learning, coding algorithms, and competing.  I ended up getting fifth place out of a hundred or so in a stock trading competition, and then I won a few competitions on bond trading and automated essay scoring.  I still remember the feeling when I did well in my first competition, and think back on it when I’m down – the elation that I was actually good at something.  The great thing about Kaggle is I suddenly had some validation in the field.  I was able to leverage the automated essay scoring algorithms I made to eventually work as a machine learning engineer at edX, a Harvard/MIT/Berkley collaboration in online learning.

The edX office that I worked at.

Unsurprisingly, my experience with learning has made me deeply interested in improving the educational system, so I’m now working on Dataquest, a way to help people learn data science in their browser.  I’m getting to do things I never imagined were possible, all because I pursued my passion instead of letting my preconceived notions of what a career “should” be define me.

In writing all of this out and recollecting, I’m startled by how easily something could have gone very differently.  If the higher-ups at UPS had promoted me right after I graduated, I might have stuck there for years.  If my friend hadn’t helped me get a job at Pepsi, I would have been stuck at UPS.  If the arbitrary test process at the Foreign Service hadn’t accepted me, I may have ended up never coding.  I wouldn’t change anything, as all of these experiences have shaped me, but it’s amazing how a life can hinge on seemingly simple moments.

It’s also interesting how much what happened at school still shapes and defines me.  I’ve become better at this lately, but when I was first getting into coding, I felt that I constantly had to “prove myself”, and show that I belonged.  I still struggle with feeling like an impostor because I don’t have the right credentials.  Although the climate is getting better, some companies still share this viewpoint – a few finance companies were interested in interviewing me after I won the Kaggle competitions, but stopped responding after I sent my resume.  “Falling through the cracks” of our system can have a negative impact, both in the way you see yourself, and the way the world sees you.

That said, it’s finding your passion and getting good at it that matter, not school.  It’s never too late to discover or pursue what interests you.  You can change your own path whenever you want.  I think that if my story illustrates anything, it’s that anything is possible.

So what do I wish had been different?  For starters, I wish I had been exposed to more viewpoints on life.  Everything drilled into me from a young age was that you go to school, get good grades, and eventually become a doctor.  Not getting good grades was never considered an option.  Doing what you’re interested in instead of what seems like a “good career” was never considered an option.  Ironically, only through discarding that notion did I actually end up at a good career.

I wish there had been a teacher or other authority figure who took more of an interest in me and helped encourage me to learn what I wanted to learn.  My parents bought me books and tried, but coming from India, their mindset was very system-oriented (“learn what school makes you learn”).  I had some great teachers that tried, but they never had the time to devote to individual students.

I would love to see some fixes for these issues.  I think a great starting point is credentialing.  How do we help people prove that they know something?  Newer services like github, dribbble, and kaggle are great for this, but how do we extend it to teachers?  Lawyers?

Ultimately, I would like to see college either become optional, or more of a free-form experience where you get to actually have more experiences instead of in-classroom time.  Getting to intern at a few interesting companies in different fields would have been great.  I’m already starting to see some really smart high schoolers skip college to get into the workforce because they know what they want, and good for them.

If you’re where I was, and you need someone to talk to, feel free to drop me a line.  I know how hard it can be, and I know how valuable talking to someone with a different perspective can be.

Knowing this, she went back to the same women, and told them about the available hospitals, cost, and their quality.  The information asymmetry is startling – some women were taking buses and travelling 20+ kilometers for appointments when better hospitals were much closer.  Another woman was paying bribes to get basic care in a government hospital.  All told, 25% of women changed hospitals after hearing about the new alternatives, 43% talked to their providers about getting better care, and 100% talked with their families about the information.

Here’s Syed (a data collector on the Tulalens team) surveying a young woman

She wants to scale up her process, and is going back to India in a week to do a larger survey.  The only problem?  She and her team were using android tablets to collect data.  Essentially, you go to a house, talk to a woman, and record her responses into an app while having a conversation.  As there is no internet access in the slums, this needs to work offline.  The available apps to do this kind of work are very limited, and most are pretty bad.  She was using an app that cost ~100 dollars a month for a few users, but rapidly climbed in price to a few thousand dollars a month.  It also didn’t have all of the features she wanted (multiple language support, ability to edit responses after they were given, ability to identify who collected which response, ability to skip some questions, etc).

Priya’s worked hard to find people who can help her make an app, and two awesome people even helped make a prototype at a hackathon last weekend.  It’s really hard as a non-technical person to network and find people who can help, but Priya’s also found people who can help with making analysis scripts, and do other work.  The challenge is pulling all the threads together and building a production-ready survey system that can be used in a week.

I love the work that Tulalens is doing, and I owe Priya for putting up with me all these years, so I decided to help her make the app as a Valentine’s day gift.  We’re not big on Valentine’s day, but it seems like as good an occasion as any for gifts.  I’m terrible at getting gifts (I remember getting a dress a couple of sizes too big once, but as a plus, I did get some awesome earrings), and Priya doesn’t ever ask for anything, so this seemed like a good fit.  The only challenge?  I’m working on Dataquest, so I only had about a day to work on the app (this day was Monday (2/16), so a tiny bit late for Valentine’s day, but hopefully you’ll cut me some slack).

Getting started (8am - 9am)

I spent a little bit of time looking for existing apps that were good for offline surveying.  I found a lot of projects that appeared dead, but couldn’t find anything good.  Lacking time, I moved on to building.

It had been a couple of years since I touched Java and Android development, and I remembered it being painful.  I was right, but not immediately.  The installation actually went pretty well.  I first installed Android Studio, then had to grab the JDK.

I then imported the existing prototype app and took a look at it.

Login screen from the prototype

I spent some time playing around with the prototype.  It was very well done for having been done in an overnight hackathon that was ~16 hours (7:30pm - 12pm the next day).  I’m not sure why this particular format was picked, but it may have had something to do with most of the participants being college students, and having scheduling conflicts.

The good with the prototype:

• It implemented an authoring platform in-app for surveys.
• It had individual user accounts.
• It used parse, which is always a great first choice for an app backend.
• It was an app people made because they cared about Tulalens (this is awesome!)

The problems with the prototype:

• It crashed after some questions were answered.
• It didn’t have a way to update/edit surveys.
• Survey answering wasn’t fully implemented.

Ultimately, rather than spend time trying to re-architect the prototype, I took some design decisions from it and started fresh with a new app.  This was because the time to re-architect the prototype was unknown, but I knew I could make an app from scratch in a day.  I can’t emphasize enough how awesome it was that Leo and Jeffrey took the time to work on this, and the design decisions (use parse, have individual accounts, etc), helped a lot with making the app.

The fun part! (10am - 10:30am)

Making surveys on-device is a very helpful feature, but it wasn’t strictly necessary, so I decided to skip it and make survey authoring happen on a computer.  The surveys could then be synced over to Parse, which is just a database in the cloud, and accessed from the app.

I googled a bit looking for a good standard survey format, but couldn’t find one.  I would have searched some more, but time constraints meant that I needed to make one.

I made a yaml format for surveys that looks like this:

----------

name: Test Survey
description: Figure out stuff about people
survey_number: 1
language: english
author: Test Guy

----------

type: choice_single
question: |
  What kind of person are you?
options:
  - Choice 1
  - Choice 2
  - Choice 3
  - Choice 4
next_screen:
  - 2
  - 2
  - 3
  - 3
  
-----------

type: text
question: |
  Where are you going next?
  
----------

Sections are delimited by 4 or more dashes (—–). The top section is metadata about the survey, and each subsequent section is a screen in the survey.  next_screen refers to the number of the screen that will be shown, depending on the response.  With some questions, like Do you own a phone, you want to be able to skip some subsequent questions if someone answers No.

The other types of questions were choice_multiple (choose multiple options), text (freeform text response), and integer (integer response).

I whipped up a quick database schema for the surveys and screens, and a python  command to sync all the surveys that were put in a folder.  Parsepy was a huge help with the syncing.

Surveys are uniquely identified by survey_number.  The database schema stores a version number, so you can update a survey, and all the versions will be stored.  Each screen is stored as a row, with a foreign key to the screen.  This way, each version of the survey has its own screens, so you never get confusion over whether someone answered the old version of a question, or the new version.

This was the fun part because I’m already really comfortable with python, and that’s why doing all of this was so quick.  The final code for this step is online here.

First steps with making an app (10am-11am)

I started fresh with a new app in Android studio.  It’s on Github here.  I used the navigation drawer template because it seemed easier than writing my own navigation.  After Android Studio created an app skeleton, I stared around for a while trying to re-familiarize myself with Java and the structure of Android programs.  I’m not ashamed to say that there was some frenzied googling and looking at stackoverflow.

I needed to first integrate the Parse SDK into the app.  I wasted an embarrassingly large amount of time wondering why I was getting errors about symbol parse not found, when I had just forgotten to import it in the main activity of the app, but eventually got over this hump.

After Parse was integrated, I could add in a login screen with just a couple of lines of code.

The login screen.  Yes, it’s super light, and should be fixed.

Adding a listview and sync (11am - 1pm)

I now needed a way to list out surveys on the device, and to sync them.  I decided to use a ListView to do this, with the ParseQueryAdapter.  This is a parse builtin class that will do a query to the parse database, and present the results in a list.  I used the parse concept of pinning to only show surveys in the local database to the listview (the listview doesn’t access the internet).  Pinning is used to add results to the device storage, so you don’t need internet access to use them.

Basically, the data flow is Parse Cloud -> Storage on device -> Listview.

I needed to build a sync service to get data from the cloud to the device.  Once the data was on the device, the sync could be used to update it as needed.  I used a query, along with pinning, to store all the current surveys (latest versions) to the local datastore.  The list view could then pick these up and show them.

The survey list screen.  The sync button is at the top right.

Adding a settings view with login/logout (1pm - 1:30pm)

I now needed to add a way for people to log out if they wanted.  Priya has three tablets, and different people will be using each at different times, so they all need to be able to sign in under their own account.  The logout isn’t fully implemented (you need to force-close the app after you log out to get back to the login screen), but it works for now.

The settings view with the logout button.

The hard part!  The survey engine (1:30pm - 3pm)

I now had to work on the hardest part, rendering the surveys dynamically from the database schema.  I made a new fragment called EngineFragment to be the survey rendering engine.  A fragment is just a partial view in android, so it’s just a screen.  When you click on a survey in the surveys list, this fragment is initialized.  It loads the screens for the survey from device storage, figures out what type of question needs to be shown first, and loads and shows the right fields for that question type.  Different types of questions have different fields that are shown.

A survey with the choice_single type.  You can pick one option.  The next button goes to the next screen.  The bar at the bottom was added later, and will be explained later.

A survey with the text type.  You can enter any text you want.

The hardest part here was dynamically rendering any amount of radio buttons and checkboxes for the choice_single and choice_multiple types.  I solved this by creating the buttons dynamically and adding them to the screen.

Storing the responses (3pm - 4pm)

Once someone hits “Next”, the response to the question should be stored to the cloud, so it can be accessed and analyzed later.  In order to do this, I pulled out the values from the displayed fields, depending on what the question type was, and created a new ScreenAttempt object.  This object was associated with a screen (which is associated with a survey object that has a version), so you can keep track of who answered what.  A participant id is also generated every time a new survey is started, so you can see which responses were given by which woman.  This object was stored to a database row.

I used the parse saveEventually function, which synced the objects to the cloud once a network connection was available.

Setting up everything on windows (4pm - 5:30pm)

This was honestly the most frustrating part, and involved setting up Python on Priya’s computer so she could run the scripts and update the surveys while she’s in India.  She’s a Windows user, sadly.  We got python setup, thanks to Anaconda, but Android Studio just didn’t want to work when it came time to import the app project.

While I worked on helping her, Priya was setting up her tablets and installing all the needed software (Pycharm, Java, Android Studio, Anaconda, Git, etc).

Pushing the app to Google Play (5:30pm - 5:45pm)

Because we couldn’t get Android Studio setup on her computer, we needed a way to distribute app updates to the tablets.  I pushed the app to Google Play alpha testing (needed to add some screenshots, and write a description) so it could be downloaded remotely to the tablets.

User Testing and Bug Fixes (ongoing)

Priya started testing out a lot of the functionality once everything was setup, and started finding some bugs, and needed features.  Some of the things we’ve addressed:

• Ability to remove surveys from the survey list.
• Ability to go back and edit responses a woman gave if the conversation reveals new information. (this is the seek bar in the screenshots above)
• Crash when using certain values in survey options

Here we are super spaced out after climbing Mt. Cadillac in Maine.  I don’t have a good picture from after coding this app up, but this approximates our mental states.

Conclusions

I’m really excited to see this app used, and it’s really rewarding to have had the chance to help out with this.  I hope it can help a lot of women in India, and around the world, and I’d love to see how it could be extended and built on.  Right now, the major needs are fixing bugs in the app, and adding in analysis scripts that can pull the data from Parse and display summaries and charts.

If you want to contribute, drop Priya a line – she’s priya at tulalens.org, and TulaLens is on twitter @tulalens.  The app is on Github here, and the survey authoring part is here if you want to take a look.

In June, I was sent to Georgetown, Guyana, for a two year assignment (a “tour” at a “post” in Foreign Service lingo).  During the first year of my tour, I was to do consular work, and then switch to economic work for the second.  Consular work covers three major areas: nonimmigrant visas (people who want to visit the US), immigrant visas (people who want to become permanent residents of the US), and american citizen services (visiting american citizens in jail, etc).  The main commonality between all of them is paperwork.  Lots of it.

The famous Stabroek market in Georgetown

Pre-processing immigrant visas

All new consular officers at my post were assigned to do pre-processing of immigrant visas.  These visas were for people who wanted to migrate to the US and get permanent resident status, so we needed to do a lot of checks to ensure that their applications were legitimate.  The task that fell to me was to go through the passport of each person who was applying for an immigrant visa.  I had to get the passport number and their name.  I then had to do a few things (I am obfuscating this process a bit because the State Department teaches you to be very paranoid):

• Search an online database (let’s call it “DB1”) for previous visa applications
• Search by passport number, and see if any hits came up.
• Search a database for their name, and see if any hits came up.
• Guyana is a country where assumed names are common, and some people don’t have a last name, so search for variations of the name on the passport.
• For each hit that was returned, click on the record, wait for it to open, and print it out.
• Search a different database with a separate program (“DB2”) for criminal and other history
• Use the passport number and name to search.
• Only one record would come up, so print it out.
• All of the printouts were then put in the applicants’ file so the officer who interviewed them for their visa could have all the information in one place.

I quickly came to loathe this process.  Because we were using very old computers, we only had Internet Explorer 6, and each page could take 10-20 seconds to load.  Going through the whole process took two hours (I usually had 20-30 of these to do per day), and it was two hours of unending monotony and boredom.  Open a passport, read a name, type in a name, click once to search, click 1-10 times on each record that comes up (DB1 could have multiple records for a single name), and print each one out.  And that was just for DB1.  I still sometimes look back on the whole process and get unreasonably angry – it was a convoluted mess that absolutely should have been automated, and I feel incredibly sorry for anyone who has had to do it.

Remember IE6?  I wish I didn’t

Using Excel

I went through the whole process for about a week before I realized that I needed to fix it to retain my sanity.  I starting optimizing parts of the process.  The first thing I did was go through all of the passports at the same time, and type everything into Excel.  I could then do all of the checks without going back and forth between the computer and the passports.  I deleted the file at the end of the process, for security (none of the information was classified or secure, but we had guidelines for handling sensitive information).  Net savings from this step was about 10 minutes.  I felt kind of good about taking a first step, but I wasn’t satisfied – the work was still pretty horrible.

If you never used Excel, its a spreadsheet tool

Automating a single search

I had taken a Java class in high school, and I knew the very basics of how to program.  A crazy idea came to me – one any experienced programmer would have probably shot down immediately or found a better way to accomplish.  I decided to write a program that clicked on things.  The basic idea was that I was entering a name, then clicking on a few things in a predictable sequence.  What if a program could move the mouse and click for me?  I would save a ton of time.  The major wrinkle was handling the varying number of records in the DB1 – sometimes there were none, sometimes 1, sometimes 10, and so on.

I stayed up at night reading about Java, and started doing some simple coding at home.  I read some programming books online, the names of which escape me now.  I installed Eclipse at home, and started writing programs.  I remember my joy when I first made a program move the cursor – it’s very cool to step back from a computer and see a program you made clicking on things.

The nice thing about Java was that I didn’t need to install anything at work – Java was already on the computer, and I could just download and run eclipse portable edition without installing it.  I decided to make a simple program that displayed one button in the window – the button was marked “Run”.  You had to navigate to the starting search screen of DB1 and enter a name or passport number into the search form.  Then you brought up the program and hit “Run”, and some cool stuff happened:

• It moved the cursor over the “search” button in the form, and clicked it to run the search.
• It waited 30 seconds for the search to finish.
• The results were presented in a list, and each result was a link (and therefore was blue).  The results started at certain screen coordinates (let’s say (600,500) – screen coordinates are in pixels, so if you have a 1920x1080 pixel resolution monitor, (600,500) is close to the center, but in the top left corner), and then each result would be 20 pixels below the previous one.  So, I first tested the color at (600,500) on the screen.  If it was blue, I clicked the link.  Then:
• Wait 30 seconds for the result page to load.
• Click “File” then “print” (I forget the exact menu structure in IE6)
• Hit “next” once on the print dialog and then click “print”
• At this point, the print dialog went away, and the page was printed out.
• Click the “back” button to go back to the results page.
• Add 20 to the y axis of the screen coordinate to test.
• Test the color.  If blue, then click on the link and go back to the top of these instructions.  If not blue, end the program.

This was way better than what I had before!  I could type in a name, hit the “Run” button in my program dialog, and come back to a screen full of purple links, and a printer full of printouts.  It prevented me from having to click on each report, and improved my sanity a lot.  But I still had to enter each name manually, and go back and forth a lot to enter the next name.  I also couldn’t really use my computer while the program was running, so I started reading books at my desk.

It didn’t help at all with DB2, which still had to be searched manually.  I then had to put all of the printouts from DB1 together with the matching record from DB2 and put them into the applicant’s file.

Net savings from this step was about 30 minutes a day.

Automating many searches

I realized that the bottleneck was me – if I didn’t have to type in each name manually and click “Run”, I could save a ton of time.  I wondered if there was a way to get data from Excel into Java, so that Java could type in the names for me and run the searches.  I read up on ODBC, and how to read excel worksheets from Java.  I also read up on how to simulate keyboard entry with Java, and how to show file picker dialogs.

The “Run” button morphed, so that now when you clicked it, it popped up a file chooser.  You picked a worksheet (in .xls format, I didn’t know how much easier csv is!), and then it was loaded in.  The first column of the worksheet was the person’s name, the second column was the passport number, and each row was a different person.

A Java file chooser

The program went through each row of the worksheet and:

• Went to the DB1 search page
• Clicked on the “first name” box, and typed in the person’s first name.
• Clicked on the “last name” box, and typed in the person’s last name.
• Ran a search, and then followed the steps to automate a single search above.
• Clicked “back” when that was done, to go back to the search form.
• Erased the data in the “first name” and “last name” boxes.
• Clicked on the “passport number” field and entered it.
• Re-ran the search, and repeated the steps from automating a single search.

By the end, I had a nice stack of printouts!  All I had to do was enter all of the passport information into the spreadsheet in the right format.  This was a major leap forward, and made my life much much better.  I still had to do all of the DB2 searches manually though, and go through the stack of DB1 printouts to match the records from DB1 and DB2, and that was a pain.

Net savings here were about 40 minutes.  I had automated the process down from 2 hours to 40 minutes a day – a savings of 6 hours and 40 minutes a week.  Things were looking up, but there was still more work to do!

A hacky way to use alt-tab

If you use Windows a lot, you know about the alt+tab key combination – it lets you switch between programs.  You hold them both together, then release the tab key while holding the alt key.  This will show a mini window with the active programs in a set order.  Then, every time you press tab, it highlights the next program in the window.  Whenever you release alt, it will switch to the highlighted program.  Using alt-tab helped me switch to DB2 and print out the DB1 and DB2 records together, so I didn’t need to collate them manually.

The alt-tab mini-window

The process looked something like this:

• Print out hits from DB1 for a given name/passport number combo
• Alt-tab to DB2
• Run a search on DB2 and print out records
• Alt-tab back to DB1 and go to the next person

The alt-tab code worked using keypress events.  It then used pixel color recognition to figure out which programs were open.  I made this simpler by only leaving Internet Explorer, Java, and DB2 open (DB2 was a separate program).

The alt-tab code worked like this:

• Set tabCounter to 1.
• Hold alt and tab, then release tab, and hit tab tabCounter number of times.
• Test the pixel at some top left coordinate in the current program (I found a coordinate that was a different color in the three programs)
• If it is the program you want, stay here.  Otherwise, increment tab counter by one, and try alt-tabbing again.
• If you went through 3 alt-tab cycles without getting the program you want, something is wrong, so stop the program.

DB2 was much easier because you didn’t have to search for name and passport number (so you only searched once), and only one record came up, so you only had to print once.  The search process was very similar to DB1 – enter the information needed, and click “Search”, then print the page out and go back to the search form.

With this method, all I really needed to do was enter all the passport information into Excel, and then I could just let it run.  I set it up on an unused computer in the office, and just let it run while I did other work.  I would come back to a nicely collated stack of printouts that I could then put into the proper applicant files.  I would sometimes just stare at the program working – it’s oddly mesmerizing to see the cursor moving and clicking on things automatically.

Net savings was 20 minutes a day.  I was now saving 8 hours and 20 minutes a week with my program!  I was really happy, and much more sane than I was before, although I still had to do nonimmigrant visa interviews (another story, but very draining).

Conclusion

I didn’t know it at the time, but that program was one of the best things I’ve made, if not the best.  It solved a real problem, and saved me time.  It made my life in Guyana much better (loved my tour, but that’s also another story).  Best of all, it sparked my own curiosity and love of programming.  I was always a mediocre student at best, and making that program motivated me in a way that school never could.  It showed me what was possible.

Almost exactly 1 year after I made that program, I cut my tour short, sold most of my stuff, and came back to the US.  I learned programming (starting with machine learning, fun!), and never looked back.  I’ve managed to make a lot of things that have helped people since then, but few of them can equal what I made then.

Life’s too short to be filled with tedium.  If you ever have a chance to automate your job, take it.  You’ll never know where it might take you.  It might even completely change the path you were on.

Immediately after the release, we started seeing news stories about how some doctors were making millions of dollars. This information is easily found, and easily sensationalized, but I started to wonder what less obvious things might be in the data.

Getting the data

You can grab the data here.

I decided to use R to analyze it, because of the ease of interactive exploration and making visualizations.  You can get the code used throughout this post here.

Actually working with the data can be a bit tricky, depending on how much RAM you have. I have a good amount, and for convenience just read the whole csv file in with read.csv. You could always use something like the ff package or read the data straight into a database if you have memory limitations.

pm = read.delim(physician_Medicare, stringsAsFactors=FALSE)

Great, now that we have our data, let’s explore it.

Surface level explorations

It’s good to explore this kind of dataset rather than starting with specific questions to answer.

Here is a truncated view of the data:

         npi hcpcs_code line_srvc_cnt average_Medicare_payment_amt
2 1003000126 99222 115 108.11565
3 1003000126 99223 93 158.87000
4 1003000126 99231 111 30.72072
5 1003000126 99232 544 56.65566
6 1003000126 99233 75 81.39000
7 1003000126 99238 95 55.76884

The important columns are npi, which is a unique ID for the physician, and hcpcs_code, which is a unique ID for the service the doctor performed. The other two fields will be important down the line. line_srvc_count is how many of the given service the doctor performed, and average_Medicare_payment_amt is how much Medicare paid each time it was performed. You should look at the data documentation for what the other fields that aren’t shown are.

Right now, the data is unique when npi and hcpcs_code are taken together. So we have a summary of each service that each doctor performed.

We need to turn this into something that is unique on npi – something that is a summary of what each doctor did. This will make comparison and finding useful things much simpler.

Converting the data

To convert the data, we could use something like ddply or the base function by. The problem with these is that they will be very slow for 9 million rows. Even solutions like ff or bigmemory won’t help much. We could read the data into a database and then do a group by query to get data in batches, but we already picked the lazy route of reading into memory.

Fortunately, the data table package for R is awesome, and will make what we are doing easy.

pm = data.table(pm)
phys_summ = pm[
  ,
  list(
    service_total=sum(line_srvc_cnt),
    ben_total=sum(bene_unique_cnt),
    payment=sum(average_Medicare_payment_amt * line_srvc_cnt),
    charged=sum(average_submitted_chrg_amt * line_srvc_cnt),
    allowed=sum(average_Medicare_allowed_amt * line_srvc_cnt),
    unique_services_per_patient=sum(bene_day_srvc_cnt)/sum(bene_unique_cnt),
    duplicates_per_service=sum(line_srvc_cnt)/sum(bene_day_srvc_cnt),
    services_per_patient=sum(line_srvc_cnt)/sum(bene_unique_cnt)
  ),
  by="npi"
  ]

The above code will transform our data so that npi is unique. It will calculate some descriptor variables (feel free to add your own) while it does it. In this case, we will see how much each doctor charged Medicare, how much they were paid, how many beneficiaries they served, and more.

Finally, some graphs!

Media reports focused on how much the top doctors made, and inequality in general is an interesting approach to this data. Let’s look at income inequality among doctors by finding what percentage of the income is made by what percentage of doctors (ie top 5% makes 50% of the income).

Inequality across all physicians

To do this, we need to first just extract the data for doctors, and then calculate the cumulative sum of doctor payments.

I was actually simplifying things before when I said that the data showed how much each doctor charged Medicare. The data actually has information from organizations (labs, hospitals, etc), as well as doctors. We can filter each one like this:

docs = phys_summ[nppes_entity_code=="I"]
orgs = phys_summ[nppes_entity_code=="O"]

nppes_entity_code indicates whether an individual or organization made the charges. We can now calculate our cumulative payments:

phys_ord = docs[order(docs$payment),c("npi", "payment"), with=FALSE]
phys_ord$pay_cumulative = cumsum(phys_ord$payment)
split_dist = floor(nrow(phys_ord)/20)
groups = as.numeric(lapply(1:20, function(x){
  phys_ord$pay_cumulative[split_dist * x]
}))
groups = (groups/groups[20]) * 100

The above code will give us how much the first 5% of doctors made, how much the first 10% made, and so on.

The above plot shows how stark the inequality is. The bottom 75% of doctors get 25% of the payments. The top 5% get 47% of all payments. Doctors in this case also includes nurses and other practitioners who get Medicare reimbursement, along with doctors who don’t bill Medicare much, so these numbers are likely too extreme.

Gender based numeric inequality

We can also look at the data by gender. Overall, there are 523085 male physicians, and 302023 female.

Let’s break down most common occupations by gender.

common_occupations = names(tail(sort(table(docs$provider_type)),15))
occupations = do.call(rbind,lapply(common_occupations, function(x){
  data.frame(occupation=x, female_count=nrow(females[females$provider_type==x,]), male_count=nrow(males[males$provider_type==x,]))
}))

This will give us a list of the 10 most common occupations, and how many males and females are doing them. Let’s make a chart:

We see that women outnumber men in jobs like Nurse Practitioner and Physician Assistant, whereas there are more men in jobs like Internal Medicine and Emergency Medicine.

Gender based payment inequality

This is interesting, but doesn’t tell us much about payments. Let’s look at how much practitioners of each speciality are paid, broken down by gender:

Men are, on average, reimbursed more from Medicare for every single speciality in the top 10 most common. This is kind of insane, and I don't know how to explain it. Anyone with insight here would be welcome.  

So where are all these doctors, anyways?

Let's move from high-level analysis into location based analysis. One interesting way to do this is to see where the "million dollar doctors" -- the ones who bill the most to Medicare -- are. Let's make a map.    

The lesson seems to be, if you are a doctor, get to Florida ASAP.  

State life expectancy

Let's look at life expectancy by state and see how that correlates with Medicare spending. Two theories would be that spending is higher in states with a lower life expectancy (because it is more needed there). The opposite could also be true (spending is higher in states with higher life expectancy, which is what leads to the higher expectancy).  In order to do this, we can get life expectancy data from here. It is pretty easy to copy/paste the data into a csv file, or use an automated scraper.  We can then create a per state charges data frame:  

state_data = tapply(docs$payment, docs$nppes_provider_state, mean)per_state_charges = data.frame(state=names(state_data), charge=state_data)

And we can read in the life expectancy data and combine them (assuming we read the data into life_e):

life_comp = merge(life_e, per_state_charges, by.x="Code", by.y="state", all.y=FALSE, all.x=TRUE)

We can then look at the correlation between life expectancy and average Medicare payments by state.

groups = c("African.American", "Asian", "Latino", "Native.American", "White", "Total")correlations = as.numeric(lapply(groups, function(x){ dat = life_comp[!is.na(life_comp[,x]),] cor(dat[,x], dat$charge)}))cor_frame=data.frame(group=gsub("\\.", " ", groups), cor=correlations)

This will actually find the correlation for each racial group (the life expectancy data has it, so why not use it?).      

This is a really interesting result! The total is negatively correlated with life expectancy very strongly, which indicates that Medicare spending is higher where it is needed (ie states with lower life expectancies get more Medicare spending).  

As for the individual ethnic groups, I am not 100% certain what it means, but I will try to interpret (let me know what you think!). The interesting part is that life expectancy for whites correlates more strongly with spending, which indicates that Medicare is more strongly optimized towards the needs of the white population than the population as a whole. Other groups are less strongly negatively correlated, and some are positively correlated, which indicates that Medicare isn't as responsive to those groups. Of course, the data is only 100% complete for whites and overall, so missing data may be causing noise. But it is very interesting that Medicare spends much less in areas where the native american life expectancy is lowest, for instance.  

The End / Future analysis

I really enjoyed this analysis, and want to do more, but I am running out of weekend time in which to do it! Next time, I think I will take a look at fraud, and see if it is possible to make models to detect fraud. I also want to see how combining this dataset with some of the other interesting Medicare datasets will look.

I looked for open source libraries, and was pleasantly surprised to find Sphinx, a CMU project.  It has python bindings, and even lets you train your own language models (awesome!).

Unfortunately, it was hard to find a working example that takes in audio from a microphone, and does speech recognition on it.  I decided to spend a couple of hours on getting all the configuration and steps down.

It’s pretty cool – you start the python script, say some things, and it converts that audio to text.  Check it out here.

Last year, I wrote this post on automated essay scoring.  This was essentially distilling my experience with automated essay scoring and trying to introduce it to people unfamiliar with it.  I got a lot of great reaction, and based on this post, and some of my other work in the field, I get a lot of emails about the topic.

Most people who email me want to know how to get started with automated essay scoring.  I always struggle with where to point them, because none of the tools that exist right now (including ones written by me!) are extremely easy to use.

A couple of weeks ago, I reached a tipping point, and told myself that I wouldn’t give another person a non-answer, and started working on an easy-to-use, and open source, essay scoring server.  You can find it here.

I will go into more detail about how I made it and how the algorithm works later on, but the keys for now are:

Nontechnical:

• Web interface that makes it easy to create problems and enter essays
• Visual feedback on how the model is scoring
• Easy to install and start using

Technical:

• Good unit test coverage for the algorithm (>90%)
• Easy to develop on with vagrant

The primary caveat is that I made this in 1.5 days.  It works, and people have tested it, but some parts are unpolished, and there are likely bugs that I don’t know about.

Please feel free to reach out if you want to contribute, or just want to chat about this.  I’m excited to see how people use it!

Happiness is this strange concept that nobody quite understands, nobody quite wants to admit to not having, and isn’t always consistent (do we care about moment-to-moment happiness, or long-term happiness?). I think Wikipedia says it best with “happiness is a fuzzy concept and can mean different things to different people.” Great. Thanks, Wikipedia. The more I read about happiness research, the more I realized that in order to truly understand what makes me happy, I would have to start capturing information about it. This has led me to develop an Android application called Happsee that can be used to track and visualize happiness. The results have been fantastic so far, and I will be sharing them in a later post. This post, though, is about looking at happiness on a higher level.

In the course of creating Happsee, I met/have been meeting with a lot of people in the Boston area who are doing interesting things in the field. One of these people is Daniel Hadley, the head of SomerStat, a very cool department of the city of Somerville that tries to quantify various aspects of life in the city and make them better. One of these aspects is happiness, and SomerStat has compiled data on happiness in Somerville going back to 2006. I tried not to let my eyes get too wide when Daniel told me about this data (normal people like normal things – I like data). He was gracious enough to share the anonymized data with me, and I will be looking at it in this post in order to see if it can help us better understand happiness.

More about the data, and our first chart

The data from Somerville consists of a survey that asks people how happy they are, how satisfied with life they are, and some other questions about their happiness on a 1-10 scale. It also contains other information, like location, race, and gender. Here is a small sample:

As you can see, the data is pretty messy in places. My first step was to cleanup and explore the data. After doing so, I got this plot:

This plot comes from taking survey questions about how satisfied people are in general, and how satisfied they are about their neighborhoods, and turning them into two dimensional data. The technique I used (read more about SVD if you are interested, it’s very cool) thought that people’s happiness and their satisfaction with their neighborhood were both very important, and clustered people according to them.

What does this tell us? Well, off the bat, most people are happy and/or really like where they live. I don’t know how this data was collected, so there could be sample bias, but it seems like Somervillians (is this even a word?) are very well off. It’s also interesting to note that even though how much you like your area correlates strongly with happiness, it’s still possible for a lot of people to be happy if they don’t like the area or vice versa. I would have thought this wouldn’t happen for a lot of people, but I guess the Wikipedia quote had it right.

Mapping happiness

People seem to have very strong feelings about their neighborhoods. Let’s try mapping out happiness by area to see if there is any correlation:

The above shows the coordinates of each survey response to the question how happy are you right now in 2013. All coordinates are anonymized by adding random noise to both latitude and longitude (so some points are in weird places, like the middle of the highway).

It looks like there are some clusters of low happiness towards the edges of the city. These may be correlated with certain features of the neighborhood. I would love to hear insight on this if anyone has any.

So it looks like while how much you like your neighborhood is pretty strongly correlated with happiness, there is no particular neighborhood that makes people extremely unhappy (or vice versa). Let’s look at what does correlate with happiness.

What correlates with happiness?

We can use a statistical significance test to discover the things that are closely related to happiness. We will be using an ANOVA test. This test will give us a p-value. We will take any p-value below .05 to mean that the variable is related to happiness.

The survey questions whose answers correlated closely with happiness in order of most to least:

The above items are correlated with happiness, which means that generally, as happiness changes, so do they (can be either positive or negative). However, this doesn’t tell us anything actionable. For example, happier people could be more likely to think that the city of Somerville is headed in the right direction. What we really want to know is if thinking Somerville is headed in the right direction causes people to become happier.

What causes happiness?

Correlation versus causation is an extremely difficult problem to solve, especially when you can’t gather additional data. Kaggle, a website that posts machine learning challenges, recently had a competition on the subject. Luckily for us, I participated in the competition, and have a program that can tell if something is a cause or an effect with reasonable accuracy (but not perfect). Let’s run the program on the factors that are correlated with happiness to discover if any of them were likely to cause happiness.

The analysis indicates that how you answer the following question is caused by how happy you are:

Taking everything into account, how satisfied are you with Somerville as a place to live?    

This is a very interesting result, as it shows that how satisfied you feel about life can be caused by how happy you feel in a particular moment. I will let the psychologists sort this result out, but for me personally, it indicates that trying to maximize moment-to-moment happiness could be a viable strategy.

The analysis indicates that how you feel about these questions causes how happy you are. So, if you answer more positively to these, you will generally be happier:

Taking everything into account, how satisfied are you with Somerville as a place to live?

How satisfied are you with your neighborhood as a place to live?

In general, how similar are you to other people you know?

How do you feel about the Public Schools?

How do you feel about the Public Works Department (e.g. requests for yard waste stickers or white good pick-up)?

How do you feel about the Recreation Programs in Somerville?    

There are some surprises here (public works?), but they generally seem consistent with prior results. How people feel about their neighborhoods strongly affects how happy they are. Improving things like public school quality should boost resident happiness. Personally, this indicates that moving to another place would be a great way to boost happiness (luckily, I love Cambridge).

A closer look at some interesting correlations

Let’s take a closer look at some of the interesting variables that could potentially cause happiness:

It seems that happiness goes up as you like your neighborhood more, but there is a strange set of high happiness levels associated with low neighborhood satisfaction. I wonder if anyone in Somerville has insight into this.

This is very interesting to me, as surrounding yourself with diverse viewpoints is often helpful.

A lot of public services look like this one (although to lesser degrees).

What does it all mean?

As with everything about happiness, its hard to tell. Below are some provisional conclusions that are by no means ironclad.

On a personal level

• Where you live seems to be very important.
• Who you associate with seems to be important.
• How you interact with your surroundings (school, recreation, etc) seems to matter.

On a city level

• The quality of city services can matter a lot.
• More “customer-facing” services like schools and recreation seem to be the most important.
• Citizen happiness can be affected at a city level.

What’s next?

I will be looking into this data more and seeing if anything else remains to be discovered. There are a huge amount of interesting things that can be done, and thanks again to Daniel and SomerStat for letting me look at this.

As I continue making Happsee, I am going to try to figure out these kinds of insights and more in real-time, which I am hugely excited about. Here is a map tracking two weeks of my own happiness in and around Boston:

Imagine being able to anonymously map happiness in real-time throughout your city, and couple the mapping with realtime insight. This could be incredibly powerful. The more steps I take down the path to understanding myself and my own happiness, the more I am struck by how much it can tell me and help me.

If you are interesting in talking more about this post, what I am working on, or data in general, feel free to contact me. If anyone wants to do similar analysis in their own city, I am happy to help. I normally put the code for my blog posts on Github, but I will need to hold off for now to clean the code up and check on whether or not the data can be released.

Traditional LMS tools are great for what they do – enable course content to be translated from in-classroom to online. They often serve as content-delivery mechanisms rather than skills measurement mechanisms. This is extremely useful, and I have learned a lot from content hosted on LMS sites. “Social” LMS tools are also fantastic for what they do – reimage the LMS experience for a more connected, Facebook-driven era. These often have content delivered in social-media like “streams.” I know both of these characterizations are oversimplifications, but I have limited space.

Neither tool can cover all learners and learning styles, and as I reflected more about how I learn, talked with people about how they learn, and read about research in the field, I realized that perhaps there was space for a tool that approached learning in a different way.

Where did Movide come from?

I have thought a lot about learning tools in the past year, in the course of which I have been privileged enough to work at edX. This led me to think about how I learn. Over the past two years, I have learned programming, machine learning, data science, web development, and a whole host of other associated skills.

Increasingly, the problem isn’t finding the right content – content is everywhere. I’ve learned from Khan academy, I’ve learned from Andrew Ng’s machine learning course, I’ve learned from StackOverflow, and I’ve learned from this great book, among dozens of other sources.

I operate best when I find a problem that I want to solve, and then learn how to solve it. The hardest part for me often is finding that defined problem, and finding an engaged community of peers to share knowledge with. For data science and machine learning specifically, I found both on Kaggle, but this doesn’t work for every field. What if I wanted to learn history, or how to add, or even something as abstract as how to think critically?

This reflection on how I learn led me to think back on my own in-classroom experiences, and what worked best there and what didn’t. Most of the time, I was a terrible student. I almost never turned in homework, and I generally had a bad attitude. I only occasionally connected with the material, and often did not feel engaged. As I look back, I realize that I didn’t approach it the right way. The experiences that I can unequivocally remember as good centered around people and problems – the amazing teacher in high school who brought history alive, the group report I did in elementary school on the Chesapeake Bay, the time my elementary school teacher asked me to research Japanese-American internment during WWII for a school challenge, and many others. I also remember being extremely curious and eager to share my knowledge – I would often stay up at night and read novels by flashlight.

I often wondered if I was the only person whose desire to learn was not sated through school. As I have talked to people over the years, I realize increasingly that I am not alone. Looking at research like Sugata Mitra’s hole in the wall experiment, and recent news on it has also been very interesting to me. I’m not as radical as some of the debate surrounding the research, but I think the ultimate goal is to harness student curiosity and let it drive the learning process instead of the other way around.

What is Movide?

Movide is a tool oriented around the idea of a class. A class can be a teacher and students, a class can be a group of adult learners, a class can be a group of students sharing notes, or a class can be anything in between. Movide lets anyone create classes.

Here is how the class dashboard looks. It lets you create classes and view your existing classes:

Inside a class, Movide lets anyone, student or teacher, add content and problems. Content can be videos, links, personal blog entries, and so on. Problems right now are restricted to assignments, where students respond to a question with text as well as images, video, and files if they choose, and multiple choice. But the selection will expand over time. Once this content exists, teachers can organize it, and students and teachers can discuss it. Teachers can define concrete learning objectives and skills, and track student progress towards those skills.

Here is how the learning resources (which contain content and problems) look inside a class:

And here is how the learning objectives and skills look inside a class:

We can quickly see how some of the ideas I expressed above have made it into Movide. Movide enables students to find interesting and useful content and then share and discuss it with each other, all while still working towards learning objectives. This combines the power of self-paced learning with the necessities of the curriculum, while encouraging students to interact with each other. It also enables teachers to direct the conversation and add course content.

Here are some things that you can do with Movide:

• Give students a way to create and share interactive notes.
• Maintain in-class student blogs.
• Teach a skills-based course, with online course material and the ability to track student progress.
• Create a continuous in-class discussion.
• Ask students to create assignments for each other.
• Offer students homework help outside of class.

Movide is also designed to be easy to use. I remember talking to someone a year ago about the importance of design, and I, in my arrogance, thought that software was all about coding. As I have worked more in the field, I have learned how wrong I was.

Obviously, Movide isn’t perfect, and it is under continuous development. But I think that it is a step in the right direction.

How can I help or use Movide?

So far, what I’ve really discussed is my own take on learning. I recognize the immense value in the expertise of learning professionals. I would love any and all feedback on Movide, and on the ideas I have expressed. Please feel free to contact me at vik dot paruchuri at gmail, or by any of the means listed on this page.

If you want to use movide, you can go here. If you want to run Movide on your own server, you can checkout the github repo, which has a full deployment script that can get you started very quickly.

If you want to contribute Movide, check it out on Github. Feel free to message me about what might be most useful to work on.

Technical bits

Movide is really extensible, and is built in Python using the Django web framework. It uses django-rest-framework to provide a full API, which could be very useful down the road. The API allows for authenticated retrieval and creation of anything used by the frontend.

The frontend is written in javascript, using Backbone, and most of the design is taken care of by Twitter Bootstrap. What these technologies mean is that it is very easy to customize Movide to your own needs, whether that need is a retheming or building an iPad app. A mobile application could easily use the API endpoints and construct a user interface around them. Retheming is as easy as overriding the default bootstrap stylings.

Movide is currently deployed to Amazon EC2, and uses Cloudformation to spin up the resource stack, and Ansible to deploy to it. The Github repo for Movide has more information about how to do this yourself.

What is the future for Movide?

I’ve been talking to people and pondering a lot of future directions for Movide. Phase one is to create a useful system that enables teachers and students to share and discuss content.

Phase two could involve automatically using machine learning to determine when a student is struggling, or has mastered a skill, and alert the teacher. I discussed some of these ideas in my automated essay scoring blog entry. It could also involve gamifying learning by giving students levels of progress to work towards. It could also involve creating better collaboration tools within Movide for students and teachers to create documents, or make a class whiteboard online.

Ultimately, the future direction will be dependent on you and your feedback. I would love to hear any and all thoughts.

I was then able to turn this data into two dimensions and plot it to show how the voting patterns of the senators differed. This led to the plot below:

From the above plot, it appears that the Massachusetts and New Jersey senators are very extreme in their voting patterns. Both Juan Carlos Borrás and Fr. pointed out in the comments that this was due to the coding of the votes. Massachusetts and New Jersey had senators who were changed mid-term, causing their votes to be coded as a 3. Since nobody else had votes coded as a 3, this caused them to appear like they had very different voting patterns, when in fact, they just were not there.

I had known of the 3-coding initially, but opted to keep the data “as-is.” The more I think about it, the more I realize that this could be used to spin a false narrative. I easily could say “democrats tend to be very extreme in their voting, just look at John Kerry!”, or “Massachusetts is the most radical state in the country!” based on the above chart. Of course, neither of these statements are strictly true, but the chart above, which is based on accurate data, could be used to tell such a story.

If we reconstruct the chart without any senators who were switched mid-term, we can tell a very different story.

Our new story

The above plot is an updated version, and tells our new story. As we can see, the new story shares a lot of similarities with the old story – we still see that Collins and Murkowski are close to the center ideologically, and we still see that there is a clear ideological dividing line between the parties.

A key difference is that both dimensions are meaningful now. The x dimension is telling us how far the parties are from each other, and the y dimension is telling us how much variation there is within each party. We still see solid voting clusters around party leadership, but we also see senators such as Paul and Heitkamp who vote significantly differently from their own parties (but not in a way that moves them in the ideological direction of the other party).

Looking at the “most extreme” senators again gives us all Republicans, which makes sense, as they are the party out of power, and are thus the farthest from the “average” view.

What lessons can we learn?

Any data visualization can be manipulated, sometimes unconsciously, to weave a certain tale. Often, this tale can be backed up with data. This brings us to the major trap of visualizations. They can simplify data to the point where it is comprehensible. Coupling the power of a machine to rapidly scan large amounts of data and the power of a human to recognize patterns can lead to powerful insights. But unless it is clearly spelled out how the visualization came to be, and we think clearly about it, we risk something being oversimplified, often to the point where the meaning that is conveyed is different from the actual meaning of the data. By simplifying, we remove understanding.

This problem can be addressed in two broad ways, I think. One is to publish the source data and methodology for creating the visualization. In my case, this coupled with some smart readers led to a good catch. The second is to directly allow the viewer to manipulate the visualization in a data-driven way. This would allow a viewer to draw their own conclusions, without the preformatted data being given to them. D3.js and shiny allow this, and I would like to play around with them to see if it is feasible to make visualizations in this way.

If you have any other ideas for balancing simplification and understanding, please let me know.

A piece of software coldly judging the quality of our carefully constructed phrases and metaphors based on unknown criteria is more than most writers can bear. But is this what automated essay scoring (AES) is? If not, what is it? In this article, I aim to explore what AES is, the state of field, some of the lessons I have learned along the way, and where I think it is going.

Who am I?

I’m going to ramble a bit here, so please indulge me. I have been involved in the AES field for 2 years now, but the building blocks were laid well before that.

I didn’t know it at the time, but my interest in AES began with my own struggles with higher education. I never knew what I wanted out of my life when I was in school, and wouldn’t have seen college as the way to get there even if I had. Afterwards, this gave me a keen interest in trying to find ways to personalize learning.

I later joined the US foreign service, a career that required me to do a lot of writing (see: Wikileaks cables). I ended up leaving the foreign service, a decision that led to me learning programming and machine learning, the art of how to teach computers to predict things, through online materials. But I kept my love for writing alive.

Imagine my surprise when I found a three month long competition sponsored by the Hewlett Foundation, and hosted by Kaggle, that aimed to develop algorithms to automatically score essays. I won’t try to dance around the issue; I initially competed because I noticed that some extremely smart people were in it, and I wanted to see how I stacked up. But as time went on, I became more and more invested in the subject, and began to recall my own experiences with higher education and writing.

I was fortunate enough to be able to work with Justin Fister, and we ended up coming in 3rd place out of 156 teams in the competition. Most of the major assessment companies (think Pearson, McGraw-Hill, etc) also participated, but more on this later. In a second competition for short answers, we teamed up with Measurement, Inc. (MI), and came in first place on the leaderboard, although we were ineligible for prizes due to our company affiliation. You can find the excellent papers from the winners, as well as their code, here.

As strange as it sounds, even though I was sitting at my computer, coding for hours on end, participating in those competitions was a lot of fun. I was able to spend every day learning, striving, and implementing (I may have had a less rosy recollection had I not done so well). But the luster quickly faded post-contest. We had made some interesting advances, and now possessed a lot of knowledge, but so what? The knowledge was not being applied to anything, and there is a huge gap between theoretical and real-world results. Justin found a job at MI, and I went to work for edX, a massively open online class (MOOC) provider based in the Boston area, where I started to apply what I learned.

You may have heard of the edX automated essay scoring algorithm, and the backlash such as this and this to it and AES. I created this algorithm, and as much as the criticism can sting or get really strange (the guy who wrote the first backlash article I linked to decided to call it first year graduate student level, which I guess he thinks is an insult), I think that there are valid points on both sides of the issue, and I will try to go through them in my discussion.

What is AES?

To me, AES is the art of giving students automatic, iterative, and correct, scores and feedback on their essays and constructed responses.  You may notice that I inserted several concepts that go beyond the narrow scope contained in the words “automated essay scoring”. I will go through each one in order:

• Feedback - in many applications of AES, such as in a “second reader” role on the GMAT(more on what this is later), feedback isn’t as important. But in all cases where a student will be learning based on how their response is scored, feedback is absolutely critical.
• Iterative - One of the main advantages of AES is that students can submit their response as many times as they want, getting very quick feedback and scoring each time.
• Constructed responses - Automated scoring isn’t just about essays. As the second Hewlett Foundation competition showed, computers can grade constructed responses as well.
• Correct - Automated essay scoring is somewhat useful if its scores are only marginally inaccurate, but its utility goes away if it can’t score properly.

Now, maybe you don’t agree with my definition, but at least we have a baseline idea of what it is for when we proceed through this article. If you don’t agree with this, I would love to hear what your definition is, and how you think this definition can be made better.

A brief history of AES

Remarkably, the idea of AES first came about in 1966, and was advanced by Ellis Page. Amazingly, it only took him two years to come up with working software. I can’t even imagine how much ingenuity it must have taken to do what he did with the hardware/software limitations of the time. His software, called Project Essay Grade (PEG), was later purchased by Measurement, Inc., which continues to develop it.

Through the 1990s and 2000s, several other companies, such as Educational Testing Service, Pearson, and CTB/McGraw-Hill, started developing their own tools. Some open tools, such at BETSY, also appeared. ETS in particular has published a lot of interesting papers, which you should check out if you are interested.

One major use case of these tools was as an automated “second reader” for high stakes tests. A human first scored the test, after which a machine scored it. If the two scored differed by a certain amount, then a third human re-scored the paper to resolve the dispute. Another major use case was as an in-classroom learning tool. In fact, these are still the primary use cases for AES.

How does AES work?

Here is a rough diagram of automated essay scoring:

So, students first write some essays. Teachers then grade these essays using whatever criteria they want and a machine learning model is created.

A machine learning model differs from a machine learning algorithm. A machine learning algorithm is a blank slate that can be trained to do a certain task. To make a bit of a stretch analogy, think of it as a computer brain – it is capable of learning something, but it doesn’t know how to do it yet. We then train this algorithm, this computer brain, to score essays. After it has been trained, it gives us a machine learning model, which can be used to score more essays.

In order for a machine learning model to be created, features first need to be extracted from the text, as a computer cannot directly understand English. We need to use the numbers as proxies for meaning.

Features are just numbers that describe certain things. For example, in my current apartment, one feature is that it has 1.5 bathrooms, and another feature is that it has 2 bedrooms. If I was going to build a machine learning model to predict apartment rents, I might pass in these features. I would then map the features to a certain amount of rent. So, for example, if one apartment has 1.5 bathrooms and 2 bedrooms and costs 1,000 dollars a month in rent, whereas another apartment has 1 bathroom and 1 bedroom and costs 500 dollars a month in rent, a machine could learn that a certain number of bedrooms and a certain number of bathrooms equal a certain amount of rent. So, if we ask it to predict the rent for an apartment with 1 bathroom and 2 bedrooms, it might say 900 dollars.

Let’s look at this is the context of essays, using some examples from a presentation I did last month:

Say that I wanted to give a survey today and ask you why do you want to learn about machine learning?

The responses might look like this:

I like solving interesting problems.
What is machine learning?
I'm not sure.
Machine learning predicts everything!

Let’s say that the survey also asks people to rate their interest on a scale of 0 to 2.

We would now have the responses and associated interest scores:

numberresponsescore1I like solving interesting problems.22What is machine learning?03I'm not sure.04Machine learning predicts everything!2$('.table').dataTable({ "bPaginate": false, "bLengthChange": false, "bSort": false, "bStateSave": true, "sScrollY": 200, "sScrollX": 500, "bFilter": true, "bInfo": false, "aLengthMenu": [[50, 100, -1], [50, 100, "All"]], "iDisplayLength": 4, });

So, let’s say that we get a half-filled-out survey that forgot to include the interest score. All we got was the sentence I really like solving problems. Machine learning is very useful. Now, if we look at this in the context of the other responses, we can infer that the interest of the person is likely a 2/2. But how would a computer do the same thing? Through features.

Some of the features we might extract:

• Presence/absence of the phrase solving problems. (0 if absent, 1 if present)
• Number of sentences.
• Presence/absence of machine learning.
• Average word length.
• Presence/absence of machine.

This is a very simple example, but it gives you a good idea of what features are. Features allow us to represent text, which a machine does not understand, as numbers, which it does understand.

We can then tell a machine learning algorithm, such as a random forest, or a linear regression, that a certain sequence of features means that the teacher gave the student a 2, another sequence of features means that the teacher gave the student a 0, and so on. This trains the algorithm, and gives us a model.

Once the model is created, then it can predict the scores for new essays. We would take a new essay, turn it into a sequence of features, and then ask our model to score it for us.

As you can see, what the model is trying to do is mimic the human scorer. The model is figuring out how an expert human scorer grades an essay, and then trying to apply that same criteria to other essays. So, it isn’t actually a machine judging essays on arbitrary criteria; its a machine trying to figure out the criteria a human uses to score essays, and then apply those criteria to grade other essays.

More on this is beyond the scope here, but I recently did a talk about AES, and also have a tutorial on my blog, both of which I highly encourage you to check out if you are interested.

Applying AES

Now that I have given you the theory, let’s talk about application. Here is a diagram of how we grade essays and constructed responses at edX:

So, when a student answers a question, it goes to any or all of self, peer, and AES to be scored. Written feedback (from peer assessment), and rubric feedback (from all three assessments) are displayed to the student.

It is completely up to the instructor how each problem is scored, and how the rubric looks. Here is an example rubric (I’m not good at making rubrics, so it’s not excellent):

Topicality
0 points - Student is off topic
1 point - Student stays on topic

Photosynthesis
0 points - Incorrectly defines photosynthesis
1 points - Partially correct definition
2 points - Fully correct definition

The AES would tell you how you did on each of the rubric dimensions (which are customizable by the instructor).

Here is specifically how the AES works:

The main difference between this and the generic workflow I showed you before is that edX allows teachers to regrade essays that AES has scored poorly. When a machine learning model scores an essay, it doesn’t just give you a score; it also gives you a confidence value from 0% - 100% associated with that score. A low confidence indicates that the machine learning model does not know how to score a given essay well. We show student papers that AES has already graded to the teacher, in order of lowest confidence to highest. When a teacher re-scores a paper, it gives the student the correct score, and makes the machine learning model better (it won’t make the same mistake twice). This is called active learning.

The AES will give the student feedback on how many points they scored for each category of the rubric. I show you this example less to discuss the strengths and weaknesses of the edX system (it has both), but more to lead into a discussion of how, when, and why AES should be deployed.

Lessons of application

I personally have learned a lot of lessons in both developing and applying AES algorithms. Below are some, in no particular order. I talk about the edX system a lot, because I have a lot of recent experience with it.

Don’t forget the goal

The goal here isn’t to impress people with fancy technology or tell teachers how they should teach. The goal is to maximize student learning and limited teacher resources (time) in a way that is flexible, and under the control of the subject expert (teacher).

Scale

In a MOOC setting, AES makes sense. It is hard/impossible for a teacher to score thousands of students each week, and writing is a critical component of many courses. But scale can also play a big part in the classroom. Can a teacher grade 10 drafts per student per week? Maybe it makes sense to allow students to score their “intermediate revisions” with AES, improve their writing, and give their key drafts and finished products to a teacher for more detailed feedback.

AES is (mostly) best used in combination with other ideas/technologies/concepts

In the same vein as the point above, AES is useful in some domains, and can given students accurate scores and rubric feedback. However, AES cannot give detailed feedback like an instructor or peer can. You should evaluate your options and see how you can best use AES. Maybe it works for certain questions. Maybe you can grade tests with AES. Maybe it is good for grading first drafts. Maybe you should combine it with small group discussions or peer scoring. If the tools are built properly, it will be possible to evaluate all these options, and figure out which one, if any, has the most value for students.

Put the power in the hands of teachers

AES is useless when the power is in the hands of researchers and programmers (although it does make us feel important). The real people who need to shape and implement these technologies are teachers and students, and they need the power to define how the AES looks and works. Maybe a teacher doesn’t need to define what features the AES uses, but being able to turn off the AES for certain students might be useful.

Give people the information that they need

AES is a semi-shadow world to a lot of people, and that may be partially by design. The less we tell people about how things are done, the more valuable and important we become. I am always leery of researchers who take the “non-cooperative expert” stance. All of the software I developed/helped develop at edX is open source, and I have been open sourcing all of my recent personal work. But just open source is not enough. We need to discuss what the code is doing, build up documentation around it, and most critically, allow people to contribute to it, to make it truly useful. The Hewlett Foundation, and particularly Vic Vuchic, have done some great work here, and I hope it is continued.

Have the algorithm tell people how it is working

Algorithms can estimate their own error rates (how many papers they grade correctly vs incorrectly). At edX, these error rates are displayed to teachers, so that teachers can make the machine learning models better if they want to. Giving teachers and students as much information as possible within an AES system is key. If we don’t know how something is working, how can we tell if it is doing what we want?

It’s not all about the algorithm

Most of the time I spent creating the edX open response assessment tool was spent on things that don’t have to do with the algorithm. Algorithms are fun and exciting, but learning tools are only useful if they help students, well, learn. The most important thing in this is usability. Can a student quickly digest and use their feedback? Can a teacher quickly create a new problem and deliver it to students? It is actually pretty easy to implement an algorithm. It is hard to put the things in place around it to allow students to succeed. I would even venture to say that once you get a certain level of accuracy in your algorithm, improving usability should become the primary goal.

Make everything usable

Is the product designed for teachers or for “expert” researchers? Does a user have to manually read a ton of essays into a command line or GUI program (think Microsoft office)? How do students get papers into the system? At edX, everything is a web-based tool, and students can write papers and receive feedback entirely through a web interface. Teachers can create problems that use AES in a few clicks, and can grade student papers through a web interface. This isn’t the end all be all of ways to approach this, but more user friendly is better.

Grading isn’t all about essays

Can we grade uploaded videos? How about pictures or songs? This can be done with peer and teacher grading, but AES needs to be extended to work with alternative media as technology advances.

Keep coming up with new ideas

I alluded earlier to several large assessment companies participating in the Kaggle essay scoring competition. In fact, these companies were the American Institutes for Research, Carnegie Mellon University (Lightside), CTB/McGraw-Hill, Educational Testing Service (ETS), MetaMetrics, Measurement, Inc., Pearson, Pacific Metrics, and Vantage Learning. The Carnegie Mellon (CMU) tool is and was open source, but crucially, it does not appear to be open information or open contribution (edit: Elijah Mayfield points out in the comments that the Carnegie Mellon tool is on bitbucket, and is open contribution). The vendors and the teams didn’t compete on exactly the same leaderboard, and you can read more about the results here, but the algorithms of the vendors and the competition participants were evaluated on the same data sets (edit: Shayne Miel, referenced below, has told me that the vendors were evaluated on a slightly different data set. However, there were advantages on both sides, as vendors got to talk to the Hewlett Foundation about the data several times).

Competitors and vendors were ranked by quadratic weighted kappa (QWK), which measures how closely the predicted scores from the models matched up with human scores (higher kappas are better). We can summarize the performance with this excellent charts from Christopher Hefele:

We can see that the top six competition participants did better in terms of accuracy than all of the vendors. I have discussed before what I think of accuracy as the sole metric for AES success, so take this with a bit of salt. The main reason I show this is to illustrate that open competition, with a fair target, can lead to very unexpected results and breakthroughs.

Even the open source solution from CMU that was included in the competition scored a QWK of .7538, good for only 19th place on the final leaderboard, which indicates that it is less about open source than about open information, access, and competition (observant readers may notice that the guy who made the CMU tool is the same guy who called my code “first year graduate student level” and disparaged the edX tool. I note this only to acknowledge potential bias – I do not think I am biased in thinking that open information is key, but I may be wrong, and let me know if you think I am).

I think that the best results come about when fresh ideas can be combined with existing knowledge and expertise. There are some extremely bright people in the AES industry that I have been fortunate enough to meet and/or work with, such as Chris Brew, Shayne Miel, David Vaughn, and many more. When Justin and I teamed up with Shayne and David, we ended up doing very well in the second Hewlett Foundation competition.

This second chart, also from Christopher, is also interesting:

Each line is how one of the top competitors performed on the public leaderboard (essentially us testing our algorithms before the final evaluation). Looking at the “VikP & jman” line, which is the line for my team, brings back some memories of frantic coding and thinking up crazy solutions to increase accuracy.

We can see how performance changes over time, as algorithms got more and more accurate. But only up to a certain point. The data that we worked with in the competition to train our algorithms was limited – we could not create more. What you are seeing is everyone converging on a maximum theoretical accuracy. After this point, there is not much more that can be achieved (which is another reason why I think things other than accuracy should be more emphasized).

I went through all of this to get to a relatively straightforward point – we need to make tools that are open and usable, and we need to make information readily available. I have a strong feeling that doing this will lead to breakthroughs and new directions that nobody has thought of yet.

Everything else I didn’t think of

There are many more lessons to be learned. Let me know if you think of any.

The bright future

The Hewlett Foundation has been doing some excellent work in the AES space, and I know that they are planning in-classroom trials of various AES products. Unfortunately, I believe that the group of products has already been picked, and is mostly commercial products or “established” products.

Some scenarios that I hope emerge out of this trial and others are below. Some of these are already being done to varying degrees:

• AES is used as a way to give students iterative feedback before submitting a final draft to a teacher for a full evaluation. Does this help learning?
• AES systems experiment with giving teachers information, and allowing them to regrade and customize grading.
• Automated systems pre-score essays, and automatically identify students who might need teacher intervention.
• A teacher scores an essay, then is given the score the same essay received via AES, and can update their score if they believe that the AES brings up useful points.
• Small group discussions and peer grading are tried in combination with AES.
• Automated scoring of alternative types of media, like videos, begins to emerge.

Hopefully I have given you a good idea of what AES is, and what it can do, and how it might look in the future. Please let me know if you have any questions or want to share something. You can find me in a ridiculous amount of places: the comments section here, twitter (@VikParuchuri), linkedin, and email (vik dot paruchuri at gmail). Pick one.