Meta-assessment

The other day I wrote a quick blog post from school, making noise about assessments and a new format and the like.  As I was finally assembling the tests, I was having second thoughts.  It wasn't a grammar-and-vocabulary test, which is what I think of by force of habit as a test.  This assessment was only communicative skills--listening, speaking, reading, and writing.  I keep telling students that I don't care if they memorize vocabulary lists or whether they can conjugate verbs or not, that what matters to me is their ability to communicate.  In the first year class, the emphasis is strongly on the comprehension skills, with production being fairly limited in length and scope.  This is in line with language acquisition theory, which says that comprehension will develop before production, and will always occur at a higher level.  It's also in accordance with the ACTFL national standards and the corresponding levels of performance for Novice-Mid to High.

But my tests haven't really reflected that--until this year, the tests have always been (1) listening comprehension (2) reading comprehension with a cultural trivia component (3) grammar and vocabulary sections.  Largely, this is because I took as much of the text-book-provided tests as my students could reasonably do in a day, copied them off and stapled them together.  But I've started redesigning my unit plans the way they're supposed to be designed (see here and here)--which means my assessments needed to be re-written to match learning goals.  And if I don't care if they can memorize vocabulary words, I shouldn't test them on memorizing vocabulary words.

I've hesitated to do this, for three reasons.   1.  It marks a dramatic departure from what I think of as a Spanish test, and I had a hard time wrapping my head around it.  2.  I was worried that a sampling of communicative tasks would overestimate students' abilities to use the language comprehensively.  3.  I wasn't really sure that I was good enough at communicative assessments and using rubrics to assess communicative ability to create a reliable assessment.

I've learned to live with (1) in other contexts--the job of professional educator is not at all what I thought it was.  It's a great deal deeper, more exciting, and science-based than I expected it to be.  (If I'd known what my job was actually going to be, I would have taken a lot of laboratory science and social science classes, and not, for example, Astronomy, Ocean Systems, or Health and Well-Being.)  So, I'm just kind of getting used to the idea that almost everything I thought I'd be doing is the wrong thing, on some level, to do.  (Still trying not to throw out the baby with the bath water...)

Having run a couple of tests, I can now address (2) fairly accurately.  When the learners are participating in good faith in the assessment, their communicative performance gives at least as accurate picture of their language skills as the previous test formats.  And, of course, it has the added advantage of, y'know, actually testing what I want them to know.  So we can pretty well set (2) to bed.

Concern (3) remains a concern.  I'm not sure that this first communicative-based exam I gave really gets to the heart of the matter.  But neither did what I was doing before, and change and learning have to start somewhere.  I've changed, and now I'm learning.  A greater sampling of communicative tasks is probably in order, although the format is about right.  My primary concern is with the speaking/listening section.  The first time I tried this, I asked the students a question, and graded them on how well they answered it.  They then had to ask me a question, and write down the answers.  It was the first chapter test of a high school Spanish I class, so an open-ended question probably would have been too much.  But that section as it stood was much more reactive than creative, I think.

The other part of that is the rubrics I'm using.  Given the nature of a limited testing scenario, I don't think I can break the comprehension skills up the way I did.  So, I'll be looking for another way of doing it, so that the comprehension grade really tells the learners what they need to know in order to improve.

Speaking / listening assessment strategies

Today is the first time I'll assess speaking and listening as part of a larger summative assessment.  Already, I can see that the rubric by trimesters that I gave the students will be insufficient to the task of assessing their conversation accurately in this context.  The fundamental conflict is that the assessment task doesn't reflect the grading scale, and the grading scale doesn't accurately reflect the learning objectives. 

I remember that creating the comprehension rubrics were particularly difficult; I may need to revisit them in light of these difficulties.  More clarity when I'm not actually at school.

Spanish I Rubrics

If you're just stopping by:  What follows is really World-Language-Teacher specific.  If you're a WL teacher, I'd like your feedback on the trimester-level breakup of performance.  If you know anything about rubrics, I'd love your thoughts on the format.  Otherwise, feel free to skip this; it's long in the tooth.

My reader will remember that, a few days ago, I tried to identify the gradeable elements of the various communication tasks.  (The other branches of the expectations will have to wait.)  After several days of making stuff up, I think I have a few useable rubrics.  I present them for your consideration and commentary.

SPEAKING

Communication
END OF SEMESTER I:  Expresses ideas, in familiar contexts, by selection from a list
END OF SEMESTER II:  Expresses ideas, in familiar contexts, using memorized words and phrases
END OF SCHOOL YEAR:  Expresses ideas, in familiar contexts, using memorized words and phrases, including some studied in other contexts.


Pronunciation
END OF SEMESTER I:  Forms Spanish sounds appropriately when they're similar to English
END OF SEMESTER II:  Forms Spanish vowel sounds appropriately
END OF SCHOOL YEAR:  Forms most Spanish sounds appropriately

Syntax
END OF SEMESTER I:  Word order always resembles English word order, even when it's different
END OF SEMESTER II:  Uses correct word order in the context of adjectives and questions
END OF SCHOOL YEAR:  Uses correct word order much of the time

Fluency
END OF SEMESTER I:  Short (3-5 word) utterances, long pauses in between
END OF SEMESTER II:  Medium (6-9 word) utterances, short pauses in between
END OF SCHOOL YEAR:  Medium (8-12 word) utterances, almost no pause in between them

WRITING

Communication
END OF SEMESTER I:  Expresses ideas, in familiar contexts, by selection from a list
END OF SEMESTER II:  Expresses ideas, in familiar contexts, using memorized words and phrases
END OF SCHOOL YEAR: Expresses ideas, in familiar contexts, using memorized words and phrases, including some studied in other contexts.

Spelling
END OF SEMESTER I:  Frequently makes spelling errors based on English spelling conventions
END OF SEMESTER II:  Occasionally makes spelling errors based on English spelling conventions.
END OF SCHOOL YEAR: Rarely makes spelling errors.  Some of them are based on Spanish spelling conventions.

Syntax
END OF SEMESTER I:  Word order always resembles English word order, even when it's different
END OF SEMESTER II:  Uses correct word order in the context of adjectives and questions
END OF SCHOOL YEAR: Uses correct word order much of the time


Fluency
END OF SEMESTER I:  Few sentence types, mostly short
END OF SEMESTER II:  Combination of short and medium sentences
END OF SCHOOL YEAR: Combination of short and medium sentences, with some compound ("and" statement) sentences



Listening / reading comprehension
Big Idea Comprehension
END OF SEMESTER I:  Identify the main idea of a simple oral presentation, in a familiar context
END OF SEMESTER II:  Identify the main idea of a medium-length oral presentation, in a familiar context
END OF SCHOOL YEAR:  Identify one or two main ideas of a medium-length oral presentation, in a familiar context

Detail Comprehension
END OF SEMESTER I:  Identify one or two supporting details of the main idea
END OF SEMESTER II:  Identify three or four supporting details of the main idea
END OF SCHOOL YEAR:  Identify five to eight supporting details of the main idea

Word Comprehension
END OF SEMESTER I:  Identify a few familiar words and cognates
END OF SEMESTER II:  Identify several familiar words and cognates, and define a previously-unstudied word
END OF SCHOOL YEAR:  Identify most familiar words and cognates, and several previously-unstudied new words


COMMENTARY:

I'm well familiar with the ACTFL Proficiency Guidelines.  They were written by people way smarter than me.  Our state standards and benchmarks are based on the ACTFL Standards and the Proficiency Guidelines.  It's also based on the assumption that a secondary school student can attain the Novice-High level of proficiency after two years of skilled study.  What I'm trying to do with these rubrics is picture how to break that process down into six trimesters.

I'm shooting to stride the line between high expectations and realistic achievement, but I'm afraid I may be underselling my students a bit. 

Some of my language is a little squishy--how long is a short utterance?  Is it the same length if a student is listening to it or speaking it?

This is really content-specific, but I think the rubric is open to commentary from any educated educator.  I'm slow to the coming of rubrics, and can use help making them.  If you have any thoughts, please let me know.  If you know anybody who might know more about this than me, please send them my way.  If you know of performance rubrics broken down into sememsters or trimesters, I would REALLY like to see them.