Auto-multiple-choice (AMC) is an open-source optical mark recognition software package built with Perl, LaTeX, XML, and sqlite. I use it for all my in-class quizzes and exams. Unique papers are created for each student, fixed-response items are scored automatically, and free-response problems, after manual scoring, have marks recorded in the same process. In the first part of the talk I will discuss AMC’s many features and why I feel it’s ideal for a mathematics course. My contributions to the AMC workflow include some scripts designed to automate the process of returning scored papers
back to students electronically. AMC provides an email gateway, but I have written programs to return graded papers via the DAV protocol to student’s dropboxes on our (Sakai) learning management systems. I will also show how graded papers can be archived, with appropriate metadata tags, into an Evernote notebook.
Decoding the Tweet _ Practical Criticism in the Age of Hashtag.pptx
Streamlining assessment, feedback, and archival with auto-multiple-choice
1. Streamlining assessment,
feedback, and archival with
auto-multiple-choice
WEB SIGMAA meeting
January 8, 2016
Matthew Leingang
Courant Institute of Mathematical Sciences
New York University
2. My issues
• I like weekly quizzes on paper
• I hate data entry
• I commute by bus
•I am a digital packrat
•I have bad handwriting
3. Things I have tried
Home rolled
scripts with PDF
forms
9. Why AMC
• FOSS (perl, sqlite, LaTeX)
• Good LaTeX package for creating docs
• features MCQ, FRQ, ID encoding,
numerical answers
• GUI with many features
• Unix Xwindows; Macports package
13. Supported Variants
• Check all that apply (questionmult)
• Horizontal vs. vertical layout
(choiceshoriz)
• compatible with the multicols
environment
• smart “None of the above” option
16. begin{question}{SEssCalcET2 12-3-016}
Let $E$ be the solid that is bounded on the
outside by the sphere $x^2 + y^2 + z^2 = 16$
and inside by the cylinder $x^2 + y^2 = 4$.
Find the volume of $E$.
AMCOpen{lines=3,dots=false}{%
wrongchoice[0]{0}scoring{0}%
wrongchoice[1]{1}scoring{1}%
wrongchoice[2]{2}scoring{2}%
wrongchoice[3]{3}scoring{3}%
correctchoice[4]{4}scoring{4}}
end{question}
Free Response Qs
45. Cool trick: returning
graded “papers”
• AMC annotation method marks MCQs and
creates a new PDF
• Email feature sends them back
• Sakai “dropbox” (not Dropbox) has a DAV
interface
• Script to batch PUT PDFs into Sakai!
46. What is DAV?
• Distributed Authoring
and Versioning
• Treat website “directory”
like a bona fide
filesystem share.
• Extension of HTTP
47. AMC-sakai-return.pl
• Command line, interfaces with AMC data model and
app config.
• Attach files (e.g., solutions, summary stats)
• Encrypt so they can't alter the file
• Dry run feature
• Upload via curl library
48. $ curl --netrc -T file.pdf
https://newclasses.nyu.edu/dav/group-user/<si
Money Line
50. What is Evernote?
• Instead of folders and files, notebooks and
notes (and tags)
• notes can contain rich text and file
attachments
• a note is only in one notebook but can have
arbitrarily many tags
• Very searchable
51. Archiving Script
• Same command line interface
• Create note with PDF, tagged by course,
term, student, etc.
• Upload via Evernote SDK
52. Advantages
• Paper when you need it, not when you don't
• No more office/book bag clutter
• Quick turnaround
• Reduces cheating by good eyesight
• Eliminates cheating by altering the graded sheet
• Regrades and post-exam conferences are easy
53. Disadvantages
• Command line is not for everyone
• Printing individual exams can take time
• Scanning multipage exams on a ScanSnap
takes time
• Files can be big
• Students who write in the margins!
62. Another cool trick: saving
files to Evernote
Seattle Municipal Archives
Credits
Ethan Kan
Clipart from OpenClipArt.org
Notas del editor
I don’t think of this as SIGMAA WEB so much as SIGMAA Tech or SIGMAA Cloud. I’m demonstrating a workflow that involves desktops, mobile devices, and paper. And a scanner. That’s a big deal.
PDF forms was my previous workflow for quizzes.
big choke point when creating PDF forms. Took me a long time.
Also a drag to identify students.
One less link in the process!
More automated
One less link in the process!
More automated
Image &lt;https://en.wikipedia.org/wiki/Optical_mark_recognition#/media/File:LegacyStyleOMRFormSm.jpg&gt; by Wiki episteme [CC-BY-3.0]
SDAPS is python (good), more focused on SSR and surveying
queXF is PHP. I am done with PHP.
G’n’T Eval is beta and in Ruby. I know no Ruby.
AMC is perl (meh), more focused on math and assessment
Each question has an ID. More about that later.
The dots are to help the processor find the corners.
The bar code identifies which paper this is of how many.
The numbers are a human-readable version of the same.
ID and MCQs on front, SAQ/FRQ on backStack of 30-40 is pretty lightGrade on paper, fill in the circleScan the graded papers, run AMCProcessingScores MCQReads FRQ marksReads IDsAfter loading CSV roster file, matches namesEasy matching of badly encoded IDs with handwritten names (screenshot?)Export spreadsheet report (good to find invalid responses or missing responses)Export CSV file (customizable fields) for uploading to LMS
Pretty darn close to Dream Workflow #1
More automated
ID and MCQs on front, SAQ/FRQ on backStack of 30-40 is pretty lightGrade on paper, fill in the circleScan the graded papers, run AMCProcessingScores MCQReads FRQ marksReads IDsAfter loading CSV roster file, matches namesEasy matching of badly encoded IDs with handwritten names (screenshot?)Export spreadsheet report (good to find invalid responses or missing responses)Export CSV file (customizable fields) for uploading to LMS
ID and MCQs on front, SAQ/FRQ on backStack of 30-40 is pretty lightGrade on paper, fill in the circleScan the graded papers, run AMCProcessingScores MCQReads FRQ marksReads IDsAfter loading CSV roster file, matches namesEasy matching of badly encoded IDs with handwritten namesExport spreadsheet report (good to find invalid responses or missing responses)Export CSV file (customizable fields) for uploading to LMS
This is a rather pricey app ($10), but I have gotten more than that out of it.
Handwritten comments with stylusTyped commentsCircles, boxes, arrows
Students love these.
One way this is useful is that you can “mount” parts of a WebDAV share on your filesystem. So Sakai exposes this protocol to make it easy to upload files by drag-and-drop.
BUT when you have to transfer 40+ files into separate directories, that can be a pain. Plus mounting the DAV share requires a little handshake which is annoying.
SO here’s the other cool feature of DAV: it’s built on HTTP so you can upload files with PUT requests. That makes it scriptable.
--netrc tells curl to read your .netrc file for passwords (potential security issue)
-T issues a PUT request
&lt;site-id&gt; is an identifier of your section (All Sakai instances have this)
dav/group-user/&lt;username&gt; is location of dropbox
name it anything you want. I crunch the CSV file and the AMC config file (XML) to get it.
Evernote is a cloud storage systemRather than files and folders it uses notebooks, notes, and tags.Notes can hold text, images, PDFs, and any other file in the premium versionNotes can be tagged any which wayA note can only be in one notebook, though.Storage limit is per month instead of total!FERPA compliant privacyNotebooks can sync to platform or live in the cloudFast, broad searching (by text, tag, date, etc)Available on just about any platform used (desktop, web, mobile)Developer API for uploading notesImage via &lt;https://blog.evernote.com/blog/2012/04/18/14-new-evernote-wallpapers-designed-for-your-ipads-retina-display/&gt; by Adelle Charles
notebook: big projects, status
tags: student, course, subjects, etc.
can create views by saving searches but also just “google your Evernote”
available on all platforms
upload limit is by month rather than total storage
I’ve written an earlier version of this script (in python) that inputs a flat CSV file and uploads to Evernote. For tighter integration with AMC, I should either pull from the SQLite Database and XML application configuration. Or I should use the AMC data model and write in perl. Kinda torn.
I keep the hard copies on a shelf and shred at the end of the semester.
No more having to track down students to give them weeks-old quizzes
Regrades—have you ever given an MC item and discovered an error in the key?
Post-exam—pull it up on the big monitor, project onto board even?
We have a multifunction printer/copier/scanner. But even then a single exam takes about a minute to print.
Maybe multifunction can do scanning faster, haven&apos;t tried yet. Of course, it can also be parallelized
Scaling—more exams this spring.
Integration—There are plugins to AMC but not for all features (Report, yes; Send, no)
Uploading CSV to Sakai should be possible.
An obvious place to plug in the “return by DAV” script would be at the point of “Send” but there’s no plugin mechanism there.
Item analysis is ready to go! Each item has a unique ID. Student scores and responses by item are accessible. NumPy or R (or perl) should be able to handle that. There is a report plugin.