Ari Bader-Natal

Welcome!

Sun, 22 Apr 2007 14:27:20 -0400

I'll be posting an entry here every time I've added something significant to the website.

Paper - BEEweb: A Multi-Domain Platform for Reciprocal Peer-Driven Tutoring Systems

ari@cs.brandeis.edu — Sat, 22 Apr 2006 14:29:59 -0400

Bader-Natal, A. and Pollack, J.

in Proceedings of the 8th International Conference on Intelligent Tutoring Systems, Springer, 2006.

[draft pdf] [published pdf] [demonstra]

Tutoring systems that engage each student as both a tutee and a tutor can be powerfully enhanced by motivating each tutor to try to appropriately challenge their tutee. The BEEweb platform is presented as a foundation upon which to build such systems, based upon the Reciprocal Tutoring protocol and the Teachers Dilemma. Three systems that have recently been built on the BEEweb platform are introduced.

Paper - Towards Metrics and Visualizations Sensitive to Coevolutionary Failures

Fri, 22 Apr 2005 14:30:31 -0400

Bader-Natal, A. and Pollack, J.

in AAAI Technical Report FS-05-03. Coevolutionary and Coadaptive Systems, AAAI Fall Symposium 2005, Washington D.C. 2005.

[pdf] [demonstra]

The task of monitoring success and failure in coevolution is inherently difficult, as domains need not have any external metric to measure performance. Past metrics and visualizations for coevolution have been limited to identification and measurement of success but not failure. We suggest circumventing this limitation by switching from "best-of-generation"-based techniques to "all-of-generation"-based techniques. Using "all-of-generation" data, we demonstrate one such techique -- a population-differential technique -- that allows us to profile and distinguish an assortment of coevolutionary successes and failures, including arms-race dynamics, disengagement, cycling, forgetting, and relativism.

Paper - Motivating Appropriate Challenges in a Reciprocal Tutoring System

Fri, 22 Apr 2005 14:31:03 -0400

Bader-Natal, A. and Pollack, J.B.

in Proceedings of the 12th International Conference on Artificial Intelligence in Education, IOS Press, 2005.

[pdf] [demonstra]

Formalizing a student model for an educational system requires an engineering effort that is highly domain-specific. This model-specificity limits the ability to scale a tutoring system across content domains. In this work we offer an alternative, in which the task of student modeling is not performed by the system designers. We achieve this by using a reciprocal tutoring system in which peer-tutors are implicitly tasked with student modeling. Students are motivated, using the Teacher's Dilemma, to use these models to provide appropriately-difficult challenges. We implement this as a basic literacy game in a spelling-bee format, in which players choose words for each other to spell across the internet. We find that students are responsive to the game's motivational structure, and we examine the affect on participants' spelling accuracy, challenge difficulty, and tutoring skill.

Paper - A Population-Differential Method of Monitoring Success and Failure in Coevolution.

Thu, 22 Apr 2004 14:31:29 -0400

Bader-Natal, A. and Pollack, J.B.

in Proceedings of the 2004 Genetic and Evolutionary Computation Conference, Springer-Verlag, 2004.

[draft pdf] [published pdf] [demonstra]

Coevolutionary algorithms require no domain-specific measure of objective fitness, enabling these algorithms to be applied to domains for which no objective metric is known or for which known metrics are too expensive. But this flexibility comes at the expense of accountability. Past work on monitoring has focused on measuring success, but has ignored failure. This limitation is due to a common reliance on "best-of-generation" (BOG) based analysis, and we propose a population-differential analysis based on an alternate "all-of-generation" (AOG) framework that is not similarly limited.

Website redesigned

Sun, 22 Apr 2007 14:31:43 -0400

I've merged my two websites together here, and refreshed things a bit in the process. Check it out and let me know what you think.

Software Update - BEEweb Status Menu now at version 0.75

Sun, 22 Apr 2007 14:33:37 -0400

BEEweb Status Menu is designed for those of you using Mac OS X that want to watch for players in the various BEEweb games.

The source code (for XCode) is released under the GNU General Public License.

Software Release - Coevisualizer.app

Sun, 22 Apr 2007 14:34:49 -0400

Coevisualizer is research software designed to help visualizing coevolutionary simulations. I wrote it several years ago, but realized that I never made it publicly available. So I've posted a version that's been wrapped up nicely for Mac OS X. The program itself is Java, though, so once I publish the raw jar file, you can run it on the platform of your choosing. After I get a chance to clean up the source code (Java), I intend to publish that here, too.

Software Update - Page Axe now at version 1.1

Sun, 22 Apr 2007 14:37:28 -0400

Page Axe enables true data privacy in the context of personal web-based applications. Many web applications provide users control over who can access their data, but the implicit assumption is that the web application provider itself always has access. Page Axe changes that. (I consider this to be a user-driven spin on Translucent Database design.)

Web-based applications are becoming increasingly popular, offering a variety of convenient benefits over desktop-based applications. These applications are effectively platform-independent, can be accessed from any computer on the internet, offer offsite data storage and backups, and often provide integrated tools for sharing with others. The primary tradeoff is privacy, as user data (emails, address books, calendars, to-do lists, etc.) slowly migrate from the privacy of the home computer to various webapp-provider servers across the internet. Some of these web applications allow for data to be classified as public or private, but this form of privacy is limited: The web-application notion of privacy refers only to whether (and with whom) your data is shared beyond your webapp provider, but at no time is your data kept private from the webapp host. This is a step back from the privacy afforded by desktop-based applications, and should be recognized as such.

Rather than entirely giving up on using webapps with personal information, I've developed a simple tool to restore the user's ability to maintain true data privacy (if and when desired.)

The source code (AppleScript) will be released shortly.

Paper - Assessing Learning in a Peer-Driven Tutoring System

Mon, 9 July 2007 10:00:00 -0400

Bader-Natal, A. and Pollack, J.B.

in Proceedings of the 13th International Conference on Artificial Intelligence in Education, IOS Press, 2007.

[pdf] [demonstra]

In many intelligent tutoring systems, a detailed model of the task domain is constructed and used to provide students with assistance and direction. Reciprocal tutoring systems, however, can be constructed without needing to codify a full-blown model for each new domain. This provides various advantages: these systems can be developed rapidly and can be applied to complex domains for which detailed models are not yet known. In systems built on the reciprocal tutoring model, detailed validation is needed to ensure that learning indeed occurs. Here, we provide such validation for SpellBEE, a reciprocal tutoring system for the complex task domain of American-English spelling. Using a granular definition of response accuracy, we present a statistical study designed to assess and characterize student learning from collected data. We find that students using this reciprocal tutoring system exhibit learning at the word, syllable, and grapheme levels of task granularity.

Paper - Evaluating Problem Difficulty Rankings Using Sparse Student Response Data

Mon, 9 July 2007 11:00:00 -0400

Bader-Natal, A. and Pollack, J.B.

in Supplementary Proceedings of the 13th International Conference on Artificial Intelligence in Education, IOS Press, 2007.

[pdf] [demonstra]

Problem difficulty estimates play important roles in a wide variety of educational systems, including determining the sequence of problems presented to students and the interpretation of the resulting responses. The accuracy of these metrics are therefore important, as they can determine the relevance of an educational experience. For systems that record large quantities of raw data, these observations can be used to test the predictive accuracy of an existing difficulty metric. In this paper, we examine how well one rigorously developed -- but potentially outdated -- difficulty scale for American-English spelling fits the data collected from seventeen thousand students using our SpellBEE peer-tutoring system. We then attempt to construct alternate metrics that use collected data to achieve a better fit. The domain-independent techniques presented here are applicable when the matrix of available student-response data is sparsely populated or non-randomly sampled. We find that while the original metric fits the data relatively well, the data-driven metrics provide approximately 10% improvement in predictive accuracy. Using these techniques, a difficulty metric can be periodically or continuously re-calibrated to ensure the relevance of the educational experience for the student.

Software Update - Coevisualizer 0.9 (cross-platform jar)

Tues, 20 Nov 2007 16:00:00 -0400

I built the Coevisualizer framework to help design, experiment with, and visualize coevolutionary simulations. A few of these were published as part of my paper for the AAAI Fall Symposium 2005 in Washington D.C.: AAAI Technical Report FS-05-03 Coevolutionary and Coadaptive Systems (PDF). I've recently picked up development on this project again in order to simulate the effects of driving a coevolutionary process with the Teacher's Dilemma (publications forthcoming.) Today I'm releasing a version of this as a JAR file, so it can be run on any machine with Java 1.4+ installed. There are many changes included, as documented in the changelog. If you try this out, please let me know!

Coevisualizer page

Software Update - Coevisualizer release 2007.11.27 (cross-platform java)

Tues, 28 Nov 2007 16:50:00 -0400

Today I'm releasing an updated version of the JAR file, chock full of compatibility fixes for computers other than my own. There are many changes included, as documented in the changelog. If you try this out, please let me know!

Coevisualizer page

Feed moved!

Sun, 1 Dec 2007 16:00:00 +0000

I've moved this feed to Feedburner. Please update your subscription to http://feeds.feedburner.com/ABN_Research.

Tech Report - Incorporating students' probabilistic expectations into a peer-driven tutoring game

Wed, 16 Apr 2008 17:28:00 -0400

Games provide a promising mechanism for intelligent tutoring systems in that they offer means to influence motivation and structure interactions. We have designed and released several game-based tutoring systems in which students learn to identify the best game strategies to adopt, and, in doing so, create for each other increasingly productive learning environments. Here, we first detail the core game underlying our deployed systems, designed to leverage human intelligence in tutoring systems through the tutor's identification of ``appropriate'' challenges for their tutee. While this game works well for task domains in which problem difficulty is known, it cannot be applied to domains if nothing is known about a problem beyond its correct solution. We introduce a second, more robust, game here capable of addressing this larger set of task domains. By incorporating player-generated probability estimates (in place of a difficulty metric), we show that a game can be designed to simultaneously elicit best-effort responses from tutees, honest statements of probability estimates from tutees, and appropriate challenges from tutors. We derive a set of constraints on the parameterized version of this game necessary for rational players to converge on this ``Teacher's Dilemma'' learning environment. Beyond providing a foundation for future tutoring systems, this work offers a new mechanism with which to simultaneously leverage and enhance the knowledge of peer learners.

Dissertation Accepted!

Wed, 16 Apr 2008 17:35:00 -0400

In classroom-based studies, peer tutoring has proved to be an effective learning strategy, both for the tutees and for their peer tutors. Today, the increasingly widespread availability of computers and internet access in the homes and after-school programs of students offers a new venue for peer learning. In seeking to translate the successes of peer-assisted learning from the classroom to the Internet, one major hurdle to overcome is that of motivation. When teachers are no longer supervising student activity and when participation itself becomes voluntary, peer tutoring protocols may stop being educationally productive. In order to successfully leverage these peer interactions, we must find a way to facilitate and motivate learning among a group of unsupervised peers. In this dissertation, we respond to this challenge by reconceptualizing the interactions among peers within the context of a different medium: that of games. In designing a peer-tutoring experience as a two-player game, we gain a valuable set of tools and techniques for affecting student participation, engagement, goals, and strategies.

Our contributions:

We define a criteria for games -- the Teacher's Dilemma criteria -- that motivates players to challenge one another with problems of appropriate difficulty;
We show three games that satisfy the Teacher's Dilemma criteria when played by rational players under idealized conditions;
We demonstrate, using computer simulations of strategic dynamics, that game-play will converge towards meeting these criteria, through time, under more realistic conditions;
We design a suite of software that incorporates a Teacher's Dilemma game into several web-based activities for different learning domains;
We collect data from thousands of students using these activities, and examine how the games actually affected the game-play strategy and learning among these students.

The game-theoretic analysis establishes the possibility for a game-based mechanism for motivating appropriate challenges, the simulations support the plausibility of this approach given non-optimal players, the implemented software systems demonstrate the scalability of this model, and the data analysis supports the real-world applicability of this game-based approach to motivating appropriate challenges for learning among unsupervised peers.

My dissertation is now online (PDF)

Tues, 6 May 2008 18:26:00 -0400

The final version of my dissertation is now online and available for download. I've also uploaded the datasets from Appendix B.

Website moving

Sat, 20 June 2009 11:00:00 -0400

This website is moving to aribadernatal.com.