Grockit games can now be customized by skill tags and difficulty level.  This helps you optimize your study time.  Custom games are best explained with an example.

They work something like this;  Sue is studying for the GMAT and her Grockit analytics indicate she needs the most improvement on Verbal questions.  More specifically, Sentence Correction and Idioms give her the most trouble.  Knowing this, Sue creates a custom game and selects the Sentence Correction and the Idiom skill tags to include.  This creates a game which only contains these two types of questions. You can see the summary that Sue sees when she is creating the game in this picture.

In addition, Sue can adjust the difficulty level of the questions to her level of ability.  In short, custom games make it easier than ever for Sue to optimize her study time and concentrate on the areas she needs the most practice.

I’m blogging today from a plane, on my way home from the 14th International Conference on Artificial Intelligence in Education. The conference — a biennial meeting of researchers who build and study software systems that help students learn — began with a number of sessions organized around specific topics, including a great workshop on Intelligent Educational Games. At this workshop, I introduced Grockit to an international group of scholars researching various uses of games in learning.

I spoke about how we leverage various game mechanics (think: XP, GP, badges, quests, leader-boards, etc) in order to help motivate and engage students in learning-conducive activities (such as helping each other!) One particularly interesting finding that I reported was on the effect of varying if and when students had access to Quests (a tailored, student-specific sequence of questions informed by our Item Response Theory model.) We found that students who could only access a Quest by “unlocking” it — by answering a specified number of questions during collaborative practice games — rated measurably higher with regards to several of the outcomes that we strive to maximize. (Perhaps the presence of an attainable goal to work towards served as a motivator for students to continue to engage?) More details are included in the short paper included in the workshop proceedings.

(While this is the first experimental outcome that we’ve published, it is only one of many randomized controlled experiments that we have run in our system. For the past several months, we’ve been relying on a custom-built split testing infrastructure to (near-effortlessly) run and evaluate small-scale experiments in any part of our system. The topic merits its own discussion, so I’ll save the details for a future blog entry…)

One of my favorite parts of a conference like this one is the opportunity to talk — face-to-face — with other people who are thinking deeply about similar issues in very different ways. In addition to the conversations in and around the Games workshop, I found myself caught up in discussions on a number of other key issues that we think about at Grockit: designing learning environments for out-of-school use, the promises and challenges inherent to peer collaboration, the scalability and adoption issues specific to educational software, striking a balance (or mix) between AI-assisted learning and peer-assisted learning, and how to use the learning data collected about students today to improve our systems for students tomorrow (both figuratively and literally.)

It’s been a full week and I’ll be happy to get off the plane. But it’s satisfying to know that on Monday morning, I have a notebook to open that is now filled with new insights and ideas about how we can better foster learning among our Grockiteers.

Diagnostic Linking Pattern (not to scale)

At Grockit we believe in collaborative learning and continual improvement. This is an ideal we encourage both in our users and in ourselves. Towards this end, we continually research and develop more exact methods for estimating question difficulty and student ability to enhance the Grockit experience.

Grockit diagnostics indicate current capability level for our users by capitalizing on the mathematical tools of item response theory. These are the same methods used by major testing companies to create computer adaptive tests (CAT) such as the GMAT.

Our capacity to estimate current capability for the individual depends upon the belief that if item A is more difficult than item B, item A should be more difficult than item B consistently across people.

Getting the item difficulty estimates was one of the early keys to providing better feedback for our members. Early on, this was a fairly arduous process because data from practice questions left us with a sparse data matrix that required many levels of data filtering to ensure reliable estimates. As an example of the filtering, people were only included in these calculations if they had both correct and incorrect responses to practice questions. We could not include data from those people because the information did not help us determine which questions are more or less difficult.

In addition to our desire to get away from the intensive filtering, we also wanted a way to get difficulty estimates for the new questions that we publish on a continuous basis.

Our new approach entails making our diagnostics dual purpose. Not only do the diagnostics serve to estimate members’ current capabilities, they also increase the accuracy of our estimation process. We have implemented a shifting diagnostic schedule in which certain questions serve as links from group to group, enabling us to place all of our data on the same metric. Newly published questions are also included so that every question on Grockit gets a difficulty estimate as soon as possible.

More importantly for our members, we already have solid difficulty estimates on enough diagnostic questions to generate the ability estimates necessary for targeted quests. With the new shifting diagnostic plan, we are now able to rapidly turn out new content with scientifically-based difficulties and experience points. Take a diagnostic on Grockit and see where you rank!

The SAT has three main sections and our analytics track your performance across dozens of different skills in each of the sections. Here are some examples of the skill tags we track on the different sections of the test.

Critical Reading Section - sentence correction, vocab in context, structural agreement

Math Section - functions, geometry, rational equations

Writing Section - punctuation, parallel structure, verb forms and tense

You can also use your analytics to create a custom review of that specific skill area. This can help you focus your review on the skills that need the most improvement.

We hope you enjoy this new, detailed look into your SAT practice!

We recently posted about moving our test suite into the cloud and how it resulted in a 90% decrease in our build time. We’ve now moved our entire site into the cloud as well.

Nathan Sobo, Grockit Director of Engineering, explains,”Other providers may offer value-added services and support, but our engineers have all the expertise we need to administer our production environment.  We simply don’t want to house and maintain the physical hardware.  EC2 is a great choice for us because of its narrow focus.  It gives us flexible access to the raw commodity of computation,  storage at minimal cost, and allows us deal with the rest on our own.”

We hope you enjoy our latest effort to make Grockit faster and more enjoyable for you to use!

If you are curious like me to know how much you remember from your high school days, create an account and play one of our SAT Diagnostics to find out.  SAT Diags are a new feature we released this week and are a sampling of questions taken to determine a baseline of a student’s abilities before they begin playing games in Grockit.  The diagnostic helps Grockit adapt to a student’s strengths and weaknesses and gives you a rank amongst other students.

If you don’t already have a Grockit account, you may sign in with your Facebook credentials or create a free account to play.  Let us know how you do.  Good luck!

Last week we began rolling out analytics and custom reviews to our groups. You’ll see the features mature over the next few weeks as we incorporate feedback from our users.

The GMAT Group is the best example of fine grained analytics. If you’ve been studying for the GMAT and you’ve attempted a significant number of questions you will be able to see your accuracy across 50+ quantitative skills and 30+ verbal skills. With this information you can identify what your weaknesses are (presuming you have any) and track your progress in those areas. You can filter by time range and difficulty; for example, you can filter to see your performance on difficult questions in the last 7 days.

SAT members currently see a smaller set of specific skills and will start to see more detail over the coming weeks.

So, I bet your thinking “Great, but what can I do with this information?”

Research* supports that getting feedback on your performance is one of the keys to learning. So play some practice SAT or GMAT games then checkout your profile to see your performance across skills.

Select the skills that you are most interested in and create a custom review of just those types of questions. Our expertly written explanations will help you gain insight into those skills.

* Research dating back to theorizing by William James‘ in his seminal book “Talks to Teachers on Psychology and to Students on Some of Life’s Ideals”

(Angie Tombari, Grockit’s summer intern specializing in Educational Psychology collaborated on this post)

Check out this snap-shot of live SAT games

We work hard to maintain a disciplined, agile, and test driven development practice at Grockit. To this end, our test suite must be green (have no failing tests) before we can push our new code and features to our demo, and ultimately, production sites.

Until recently we ran our test suite locally on a Mac Pro machine running multiple VM ware instances. Our build was split into fast and slow builds so that we could shorten the feedback on tests that fail. This process got us to a build time of about 45 minutes from beginning to end assuming the build didn’t stop short due to a test failing.

We have since moved our test suite into Amazon EC2 so that we could reduce the overall build time by parallelizing our tests from two tests running simultaneously to over 10 tests. We are really excited by the results. We’ve managed to reduce our build time by almost 90%. The build now takes approx five min from start to finish.

Our EC2 distributed build depends in a large part on TeamCity, a continuous integration product from JetBrains. Our build is split into 15 fragments which are each defined as a TeamCity build configuration. When code is checked in, the TeamCity hub, running in EC2, can distribute the running of build configurations to any of nine build agents, which are also running in the cloud. In minutes, all of the build configurations are executed and results are reported back to the central server, informing our team members via a web interface displayed on a large screen in our office. We also made a simple GreaseMonkey script to turn the background of the monitoring page red when any of our build configurations fail, which helps increase visibility.

With our build cut down from 45+ minutes to less than five minutes, there is a palpable sense of increased momentum in our development process. It’s also interesting how removing this bottleneck made other process problems less of an issue. For instance, we were considering adding a feature that allowed our product manager to toggle feature sets as visible or hidden in production dynamically. This was needed because once a feature set was ready, we would need yet another round trip through the build cycle for developers to ‘unhide’ the functionality in the production environment. When our test suite took 45 minutes, this was a big deal. But now that its fast, the pain of this manual step is far less acute. The moral of the story: find the biggest bottleneck and eliminate it before worrying about others.

Most importantly, for our users, this means more frequent deployments of new features, bug fixes and performance improvements.

So, Grockit now keeps track of the questions you’ve seen and only shows you new ones. There are a couple of exceptions:

If you haven’t gotten it correct yet it’s fair game for obvious reasons.

If you have answered it correctly but a teammate hasn’t you may see the question again. This has two benefits: Your teammate gains from your experience with the question and teaching others is a great way to further your mastery.

The no repeats feature along with our ever increasing question pool will set you up for many hours of productive studying for the GMAT or the SAT.