Chapter 1. MAVE Survey

Introduction

MAVE is Worth Publishers' new "Mechanical Answer Validity Evaluator". Based on contemporary research in computational linguistics, machine learning, and data mining, MAVE's purpose is to determine if a student made an honest attempt at answering a free-response (e.g. essay) question. It also checks for plagiarism—to make sure that two students don't enter the same (or very similar) answers to a question. We'd like to know what you think about MAVE.

The first important thing to understand is what MAVE does not try to do. According to the experts we've talked to, no current technology exists that will allow a machine learning algorithm to accurately evaluate the correctness of a free-response answer in a scalable way. The best we can do is determine the validity of an answer: whether or not the answer reflects an honest effort on the part of the student to attempt to answer the question.

Still, we think that an effective validity checker would be a huge pedagogical boon for formative assessment activities. Formative activities such as Worth's LearningCurve are great tools for learning, but the types of questions that LearningCurve and similar systems use (e.g. multiple-choice and fill-in-the-blank) can really only challenge students on the lower Bloom's levels (remembering, understanding, and, to some extent, applying). Using MAVE, we can ask students essay-type questions that get at higher Bloom's levels (analyze, evaluate, and create). Again, MAVE can't tell us whether students have mastered concepts at this level, but at least it can guarantee that students have thought about concepts at this level, and we think that getting students to think is a worthy goal in and of itself.

The tabs in this activity feature a series of questions that are designed to introduce you to MAVE. Please complete them in order and then fill out the survey at the end. Thanks!

1.1 The Sky (traditional)

Traditionally, when essay questions are given online, students receive credit as long as they enter something. Sometimes they might be required to enter a minimum number of words, but still, students are given credit for virtually any answer—even if it's clearly not an honest attempt at answering the question.

The question below is an example of such a question. Try it, and see what you can get away with entering while still receiving credit. On the next page, you'll see the difference that MAVE can make.

1.2 The Sky (MAVE)

MAVE can determine whether an honest attempt was made at an answer. It works by analyzing the "features" (i.e. words and phrases) in the student's answer and comparing them to features of other answers that it knows are valid or invalid attempts at an answer.

Before you make an honest attempt at an answer, try a few things you think your students might enter if they didn't want to really answer the question (e.g. "I don't feel like answer this right now", or something like that). These attempts should be rejected by MAVE, but pay attention to the options the activity gives you when it thinks you've entered an invalid attempt.

Once you have submitted a valid answer above, try submitting the same answer again to this copy of the question. MAVE should detect that you are copying from an answer that has already been submitted, even if you change a few words in the answer.

1.3 Healthy Eating

Source: The Big Hurt by CWK Network/Connect with Kids (www.connectwithkids.com)]

The following activity is an example of part of an actual activity where we plan to use MAVE. (It's part of our "Developmental Video Tool Kit" series.)

It would be great to have a signal system that changed our taste preferences. A possible interior monologue would go something like this:

Unfortunately, it doesn’t work that way. Foods that are no longer beneficial are still appealing.

Play the video to listen to some older children and their parents discuss food choices and some problems that overweight children face.

1.4 Survey Questions