MAVE is Worth Publishers' new "Mechanical Answer Validity Evaluator". Based on contemporary research in computational linguistics, machine learning, and data mining, MAVE's purpose is to determine if a student made an honest attempt at answering a free-response (e.g. essay) question. It also checks for plagiarism—to make sure that two students don't enter the same (or very similar) answers to a question. We'd like to know what you think about MAVE.
The first important thing to understand is what MAVE does not try to do. According to the experts we've talked to, no current technology exists that will allow a machine learning algorithm to accurately evaluate the correctness of a free-response answer in a scalable way. The best we can do is determine the validity of an answer: whether or not the answer reflects an honest effort on the part of the student to attempt to answer the question.
Still, we think that an effective validity checker would be a huge pedagogical boon for formative assessment activities. Formative activities such as Worth's LearningCurve are great tools for learning, but the types of questions that LearningCurve and similar systems use (e.g. multiple-choice and fill-in-the-blank) can really only challenge students on the lower Bloom's levels (remembering, understanding, and, to some extent, applying). Using MAVE, we can ask students essay-type questions that get at higher Bloom's levels (analyze, evaluate, and create). Again, MAVE can't tell us whether students have mastered concepts at this level, but at least it can guarantee that students have thought about concepts at this level, and we think that getting students to think is a worthy goal in and of itself.
The tabs in this activity feature a series of questions that are designed to introduce you to MAVE. Please complete them in order and then fill out the survey at the end. Thanks!
Traditionally, when essay questions are given online, students receive credit as long as they enter something. Sometimes they might be required to enter a minimum number of words, but still, students are given credit for virtually any answer—even if it's clearly not an honest attempt at answering the question.
The question below is an example of such a question. Try it, and see what you can get away with entering while still receiving credit. On the next page, you'll see the difference that MAVE can make.
Why is the sky blue?
Correct.
1.2The Sky (MAVE)
MAVE can determine whether an honest attempt was made at an answer. It works by analyzing the "features" (i.e. words and phrases) in the student's answer and comparing them to features of other answers that it knows are valid or invalid attempts at an answer.
Before you make an honest attempt at an answer, try a few things you think your students might enter if they didn't want to really answer the question (e.g. "I don't feel like answer this right now", or something like that). These attempts should be rejected by MAVE, but pay attention to the options the activity gives you when it thinks you've entered an invalid attempt.
Why is the sky blue?
When light from the sun hits gas molecules and dust in the air, some wavelengths are scattered while others are absorbed or pass right through. Blue wavelengths get scattered more often then other wavelengths. When we look up we see that scatted blue sunlight coming from everywhere, so the sky appears blue.
Once you have submitted a valid answer above, try submitting the same answer again to this copy of the question. MAVE should detect that you are copying from an answer that has already been submitted, even if you change a few words in the answer.
Why is the sky blue?
When light from the sun hits gas molecules and dust in the air, some wavelengths are scattered while others are absorbed or pass right through. Blue wavelengths get scattered more often then other wavelengths. When we look up we see that scatted blue sunlight coming from everywhere, so the sky appears blue.
1.3Healthy Eating
Source: The Big Hurt by CWK Network/Connect with Kids (www.connectwithkids.com)]
The following activity is an example of part of an actual activity where we plan to use MAVE. (It's part of our "Developmental Video Tool Kit" series.)
It would be great to have a signal system that changed our taste preferences. A possible interior monologue would go something like this:
“Looks like there is plenty of food and there is going to be abundant food in the foreseeable future. The layer of fat in my body is at an optimal level. The regulatory system and the sensation/perception system regarding taste and smell will now change so that foods high in sweets and fats will be rejected as not very appealing foods.”
Unfortunately, it doesn’t work that way. Foods that are no longer beneficial are still appealing.
Play the video to listen to some older children and their parents discuss food choices and some problems that overweight children face.
Do you often eat foods that you don’t consider healthy? If so, what thoughts direct you to these food choices? How do you compare your relationship to food with the relationship the children discuss in the video?
Children and adults sometimes eat unhealthy foods as a reward after a stressful day or a special event. Foods may be associated with certain activities, such as popcorn or candy with watching a movie. Adults often feel time pressures and may settle for unhealthy food choices because these foods can be purchased and eaten quickly. Because adults have more control over food choices, they can decide to make a commitment to increase healthy foods and decrease fast foods. Unfortunately, a 10-year-old cannot do this without parental support!
1.4Survey Questions
1. About your courses
Do you use online essay questions in your courses?
A.
B.
C.
What courses do you typically teach?
2. Your Experience
Have you ever used or seen something similar to MAVE? If so, what was it? What did you like and/or dislike about it?
3. Using MAVE
Would you be more likely to assign online activities including essay questions if you knew that MAVE could make sure that each student had to make an honest attempt at an answer in order to receive credit?
A.
B.
C.
D.
We can adjust MAVE's criteria for judging answer validity so that it is more or less strict about what it counts as an honest attempt. To some extent, this will be dictated by the context where the question is being asked. But in general, how concerned would you be that MAVE might count invalid answers as valid? This could allow some students to receive credit without making an honest attempt. (Keep in mind, though, that MAVE would only be used in low-stakes, formative assessment activities.)
A.
B.
C.
D.
Now what about the opposite... How concerned would you be that it might think that honest attempts at an answer were invalid? Students could still submit their answer and receive credit, but they would be told that MAVE doesn't think their answer is valid.
A.
B.
C.
D.
4. Other Comments or Feedback
What are your general thoughts about MAVE? Anything else you would like to say?
Would you like to be involved with further development of MAVE?
