SUBJECT: Using peer review for student work

DATE: 2/97

 

 

Labbers,

I used to have students evaluate each others' paper drafts in a

zoology class at Elmhurst College. Unfortunately, the only comments made

by student reviewers were "good job" and "OK". I think it is MUCH wiser

to have the instructor 'review the reviewers' and grade them. That is the

current approach in the senior seminar course that we offer as a capstone

course. Students put some real effort into the review and often have good

advise to offer their colleagues.

 

This content in no way reflects the opinions, standards, or policy

of the United States Air Force Academy or the United States government.

 

Helen Pigage

HQ USAFA/DFB

2355 Faculty Drive, Suite 2P389

USAF Academy, CO 80840

 

 

I also have students evaluate each other's work in our senior seminar,

and the evaluation becomes a part of the evaluating student's grade but

not that of the one evaluated. I get a wide range of results, even

though I emphasize that the job of a reviewer is to help the author or

presenter improve the report. I provide examples of helpful reviews and

non-helpful ones, go over those with the students, and have them review

the first seminar without being graded.

 

Some then are very good at providing reviews that anyone should be able

to use for improvement. Others simply report "Great Job!!" and award

the maximum possible points for each criterion on the evaluation form,

even for work that is mediocre or worse. When I discuss these

evaluations with the students who wrote them, the usual reasons given

are, "I didn't want to hurt him," or, "I was trying to be encouraging."

 

But for the most part, for most students on both sides of the

evaluation, it has been a positive learning experience. Even those who

were very ineffective reviewers at first have learned from it.

 

Dave McNeely, Biology, University of Texas at Brownsville, 80 Fort

Brown, Brownsville, TX 78520; mcneely@utb1.utb.edu

 

 

 

GRADING PROPOSALS VS. FINAL REPORTS:

 

I have not tried having students review proposals for experiments. That

would probably be tougher for students to do that reviewing the final

product of the research (a paper or poster), since to critically evaluate

a proposal you would have to imagine or anticipate its flaws and

potential problems. When reviewing the results of a research project

students can evaluate the design with hindsight and with the benefit of

seeing how the experiment went, what interpretations and conclusions were

made (and how valid they are) and how effectively and convincingly the

results and conclusions were presented. Call it learning for one's

mistakes, and often that's how most real learning happens.

 

GRADING VS. REVIEWING:

 

I have had students grade each other's final reports (in journal article

format) or posters or lesson plans (teachers). I usually have the

assignment grade amount to 50% my grade and 50% the average of the

students' grades. They don't like it! They are really uncomfortable about

grading each other, though interestingly the education majors do this

better (are more critical, and give real A's, B's, and C's) than do my

science majors (all A's and a very few B's). For the most part I have

asked each student to assign a grade on a score sheet, with only a few

comments, and require that they grade (and thus, hopefully, read) each of

the other groups reports. My intention here is that the students are at

least exposed to all of the class's reports, and perhaps will learn to

recognize and learn from the best and the worst of other students' work.

With this kind of "review" assignment I have them review many posters,

but with only a modest depth of analysis on any individual review. I

figure that with enough students grading each poster/report any biased or

ridiculous grades will average themselves out, and I drop the highest and

lowest grades from the average to make the average more representative of

a class consensus.

 

A more thorough review assignment involves each student reviewing one or

a small number of reports, and writing a one-page critical review, which

itself is graded as a separate assignment. I use this to introduce to the

students the peer review process and the importance of critical analysis

and communications skills in science. I model this after the process we

go through to publish our scholarship in a scientific journal. Lab groups

submit their research report in journal article format, which is

forwarded to several peer reviewers. The reviewers must read and

critically review the manuscript, not to grade it but to make a

rcommendation as to whether it is suitable for publication. Criteria

include the quality of the writing, the clarity with which data is

presented, how well the conclusions are supported by the evidence

presented, whether alternative explanations exist, etc. The reviewers

must clearly present their analysis, and make concrete recommendations

for the publication of the manuscript and/or its improvement. My Adobe

Acrobat handout for the peer review process is available online at:

 

http://www.carlow.edu/~rhershberger/library/writing/Review.pdf

 

Each semester we get one instance of a report with unsupported or

incorrect conclusions, which go undetected by the reviewers (both lose

points). Another "teachable moment" happens when the writers have made a

correct conclusion, but because it was not presented in a clear,

understandable way (because of deficiencies in the students' writing),

the reviewers believe that the conclusions are invalid or unsupported.

The reviewers lose a few points if they really believe that a correct

conclusion is wrong (they should be able to arrive at the same correct

conclusion if they understand the experiment), but the writers also lose

points because, even though they are right, they failed to communicate

and convince their peers.

 

I use this assignment in my Genetics course (required of all bio majors),

and the experiment everyone is conducting and reviewing is a set of

inheritance problems generated by the BioQUEST Genetics Construction Kit.

Each experiment is different, but the concepts, experimental tools, and

research strategies are familiar to both the experimenters and the

reviewers, so the reviewers can't plead ignorance of the experimental

system.

 

OK, now get away from the computer and get back to grading ;-)

 

Rick

 

RICHARD P. HERSHBERGER, PH.D.

* Asst. Prof. of Biology * CARLOW COLLEGE * Pittsburgh PA 15213

rhershberger@carlow.edu * http://www.carlow.edu/~rhershberger/

* rhershberger@earthlink.net * http://www.earthlink.net/~rhershberger/

 

 

 

For what it's worth...

 

Helen Pigage alluded to non-critical peer reviews by

students of their classmates' work. I've had the same experience

when I've asked each student in my "writing enriched" Introductory

Ecology course to comment on a draft of another student's

literature-based review paper. (Each author subsequently

revises their work, based on comments from both the peer and

myself, before resubmitting it for a final grade.) Many

students (including even upper-level biology majors) seem

to be extremely reluctant to criticize each other AT ALL...

even when the reviews are anonymous.

 

Why? I have two guesses. First, substantive criticism requires

serious thought and time; it's a lot easier to be uncritically

positive. I suspect this is a large part of the story, especially

when it's true out there in "big science" too: as noted

in an commentary in _Nature_ from 30 January, many professionals

reviewing articles submitted for publication to journals pronounce

judgement just as blithely, with minimal constructive input. Doing

more is a lot of hard work ("intellectual heavy lifting").

 

The second guess has to do with the culture of our

current crop of college-age students. Might it be the case that

they've been poorly schooled in developing the ability to

make critical distinctions about what they read (and hear)...

and to have enough _confidence_ in their conclusions to

risk transmitting their ideas to others? Seems that way to me.

 

Student authors therefore may not benefit much from

comments they get from peer evaluators of their papers.

So, what's the value of having students write peer reviews?

 

Well, it never hurts to _try_ to get students to think critically

about written work. There is, though, another advantage:

when the students know that their work will be read

by their classmates, and not just by their professor, it helps

focus their attention on the communication task at hand. Instead of

taking for granted that the reader ("The Prof") will understand

a mass of undefined jargon and acronyms (those that the student

might hide behind, rather than displaying true (lack of)

comprehension of the material), the writer knows that they have to

explain carefully what they're trying to say...or their peer reviewer

won't be able to understand. Another advantage relates to the

revise-and-resubmit aspect: I suspect that some students

who might not be bothered at all about showing a professor a

crudely prepared first draft (if it's not graded) don't dare risk the

embarrassment of showing the same raw stuff to their classmates.

Wonderful thing, psychology!

 

I'd love to hear about related experiences or impressions of

others, especially those involved in efforts to help students

to learn how to write about biology (both learning-to-write

and writing-to-learn aspects).

 

--

Robert L. Curry

Assistant Professor of Biology

Biology Department

800 Lancaster Avenue

Villanova University

Villanova PA 19085 USA

 

curry@ucis.vill.edu

http://www.bio.vill.edu/HTML/FACULTY/Curry/Rc1.HTM

 

 

 

I just had my microbiology students review each other's first lab report of

the semester. They traded reports, so they were working in pairs. The

review process was not graded -- I told them that they were to help each

other write better reports. Most of the class thought that it was a useful

exercise. Reasons given included "when you write something you can't always

tell what is wrong with it" and "I got good ideas from the paper I read."

Those who didn't find it useful wanted more time to revise after review.

Before they reviewed, I told them what I would be looking for when I graded

the reports, so this gave them one more opportunity to hear the criteria.

I'd like to write out a scoring rubric that they can use, which would

really help them internalize the grading criteria. Lab reports are due on

Monday, so we'll see if the quality was affected.

 

Lynda Harding

 

 

 

Richrad Hershberger wrote a lengthy explanation of the experience he has had

in having students grade each other's work. I will repeat (without the

detail I provided before) that I have students evaluate each other's work,

and that I include the evaluation in my assessment of the STUDENT DOING THE

evaluation. The evaluation enters into the EVALUATOR'S grade, but NOT the

evaluatee's.

 

I agree with all the positives in having students evaluate each other's work.

However, I WILL NOT assign, even as only a part of the the grade, a grade

arrived at by a student, or even use the student's comments in arriving at

the grade I assign. Several concerns cause my opposition to such a practice,

concerns that arise out of long experience with students.

1. Concerns about conflicts of interest on the part of students.

2. Concerns about incorporating student assessments into grades\

assigned when the students have little or no experience in

making such assessments.

3. Concern that I am responsible to the student, my collegues,

the administration and society for being the one to make the

judgement about a student's grade. For high achieving and low

achieving students (those who may get into professional school,

a top graduate school with support and so on, or those who mail

fail to graduate or fail to achieve a credential required for

a job) a great deal is riding on the grades. Admittedly, for

"average" students, the consequences may not be so drastic, but

all in all, students feel, rightly so, that we owe them an

absolutely fair evaluation and an absolutely fairly determined

grade.

4. If you think about the third concern, and don't feel that using

student comments or student assigned grades is ethically

questionable, perhaps the threat of legal action might be a

little stronger deterrent. Even if there is no legal concern

(and I'm certainly no lawyer), there might be the threat anyway.

 

I absolutely owe my students my own best professional judgement. Unless a

student is receiving payment from my university for grading, I will not

use student assigned grades as a part of the grade I assign. Think about

it. The student assigning the grade is also taking the class? That's too

much for me.

 

Dave McNeely, Biology, University of Texas at Brownsville, 80 Fort Brown,

Brownsville, TX 78520; mcneely@utb1.utb.edu

 

 

Hi, labbers:

I don't have much experience with students critiquing each others'

written work, but I do require my senior seminar students to critique

departmental seminars. Lo and behold! I find the same thing --

students are generally able to find something to say, but the vast

majority of them rate *all* speakers very highly (and much more

highly than many deserve).

 

In this case, students aren't critiquing other students -- so I don't

think there's any real concern about hurting someone's feelings.

And I really don't think it's lack of desire to work. Rather, I

think it's because they have so little basis for comparison (i.e., no

one has really taught them, by repeated example and discussion,

what makes really good seminar). So I wonder -- might that be a

contributing factor to the problem with written work?

 

Cheers,

Kerry

 

**********************

Kerry S. Kilburn, Ph.D.

Department of Biological Sciences

Old Dominion University

Norfolk, VA 23529

(757)683-5680 FAX 683-5283

 

http://www.odu.edu/~ksk/HOME.HTM

 

 

 

An approach that I have used that has produced fairly well thought out peer

reviews actually follows a typical essay assignment. It involves

distributing 5 of the student essays to the entire class. Each class member

is asked to rank the papers from "most effective" to least "effective" in

achieving the objectives of the assignment. They then must indicate what

would improve each of the papers so that it was as effective as the #1 paper.

This was not well received when I had student names on the papers the first

year. However, during the last two years the papers have been anonymous and

the students editorial suggestions have been useful and usually consistent.

 

I then distribute the pile of responses back to the original authors and meet

with them to discuss their response to the critiques and how they can

improve their paper.

 

A drawback is that only 5 papers get critiqued per assignment and only

those 5 authors get the benefit of the peer reviews. However, they get a lot

of reviews and usually see a consistency of response from all of their

classmates. I avoid the idea of grading in the students mind by stressing

the ranking of effectiveness rather than a grade. They then have to justify

the ranking by suggesting specific improvements to each of the lower

ranked papers. They are thus in a position where they have to make

concrete suggestions in order to justify there ranking. The students know I

read the critiques and that this has an impact on my perception of their

professional involvement in the course.

 

I believe a major benefit is that each student gets to see (must see) a lot

of other student writing during a term and must decide what is more

effective and what is less effective writing. Often students never see much

poor writing, other than there own (and perhaps some of mine). This gives

them, with each essay assignment, 5 examples of other students attempt at

that essay. They get to see examples of some fine student writing and some

poor writing all related to the same essay topic. By the end of the term all

of their writing has improved.

 

Thanks for the good ideas,

Al

 

% Al Williams AAWilliams@Manchester.edu

% Biology Department Voice 219-982-5308

% Manchester College FAX 219-982-5043

% N. Manchester, IN 46962

 

 

I am curious in the example from Al Williams if he has ever tried having

students review papers as groups so that instead of everyone reviewing

only five papers, each group reviews a different set of five. I did

something similar to this in an ecology course and it seemed to work well.

Do you discuss these as a class? That might be good reason to use only

five overall, but if the same sorts of problems occur in many papers, the

review of more papers might be enlightening and each student would benefit

from the review. The groups of papers should change each time, but groups

of students might not need to change.

Janice

***********************************

Janice M. Glime, Professor

Department of Biological Sciences

Michigan Technological University

Houghton, MI 49931-1295

jmglime@mtu.edu

906-487-2546

FAX 906-487-3167

 

 

Janice --

 

I thought I would reply to the group since I had received a few other

similar questions.

 

My use of having the entire class rank and edit 5 student papers with each

assignment has been an add-on to a number of content courses (Vertebrate

Physiology, Vertebrate Anatomy). We wish to facilitate writing within the

discipline through all of our courses. However, in these courses I do not

dedicate any extended in class time to direct discussion of writing and did

not want a time consuming addition to the already existing writing

component of the course.

 

There are usually 3-4 essay assignments per term. Thus 3-4 times a term a

group of 5 of the student essays are distributed (via E-Mail) to the entire

class. Everyone must rank the 5 and specifically suggest how each paper

could be improved to be as effective as the best in the group.

 

My goal is to create a situation where the student must read a variety of

student papers, all dealing with the same topic, that range in quality from

good to poor, and have the students begin to critically evaluate quality.

The ranking was a fairly simple way to force the students to not be

superficial. Once they had established a rank order they seem compelled to

justify it by suggesting needed improvement. As this is repeated through

the course they get much more effective at picking up a paper and with one

reading identifying a number of key areas where the paper can be

improved. About half way through the course I reinforce the idea that they

should be incorporating this same critical evaluation into reading their own

work, that having others review drafts of their writing will obviously

improve their writing, and that in the final analysis there are usually many

good ways to write on any topic and a far greater number of poor ways.

 

Many students have never seen a variety of essays all dealing with the same

topic and thus don't have an intuitive feeling for how simply changing the

organization, or choice of examples, or sequence of ideas, can radically

change the effectiveness of an essay.

 

I like the suggestion of having groups do this rather than individuals. The

discussion would be a much more effective learning situation. However, it

seems that it would also become a much larger assignment involving either

dedicated class time or the students would have to prepare in advance and

schedule meeting times for the group discussion and preparation of the

final reviews outside of class. In discussing this assignment with students

my impression is that currently, once they understand what they are to do,

it takes about 45 minutes to read the 5 essays, determine their opinion of

the rank order, and type up a specific list of suggested improvements for all

5 papers. That type of time commitment lets me consider it a fairly small

assignment that can be done any evening. I believe I will try the group

approach in one of the courses next year.

 

Another aspect of the assignment is that the entire set of class responses to

the essays is made available to the actual authors of the original essays.

They get a great deal of peer review of their work. After they have

digested about 25 sets of comments/suggestions on their essay I meet with

them and we plan a revision of their original essay. By the end of the

course everyone has had at least one of their essays reviewed by the entire

class. I think they are very aware that I am not the only audience for their

writing.

 

Finally, this has not been very time consuming addition to the course for

either myself or the students. I was interested in an approach that would

force more serious thinking about writing without my having to actually read

more student writing.

 

Thanks,

Al Williams

 

% Al Williams AAWilliams@Manchester.edu

% Biology Department Voice 219-982-5308

% Manchester College FAX 219-982-5043

% N. Manchester, IN 46962

 

 

Well, I'm interested in "student writing assignments," but I'm not

interested in reading them. <head hanging in shame> I have 150-or-so

students, and my dedication just doesn't reach that far. For that reason,

I've been just skimming this thread, but I'm glad I took the time to read

all of your post. Perhaps by involving the students the way you suggest,

it's possible for them to have this valuable learning experience even with

a lazy professor like me!

 

Lane

----

Lane Lester / llester@athens.net / Athens, Georgia USA

----

 

 

 

You'd be amazed at how quickly reading and grading can go with a familiar

topic--one that you have devised and to which you have a clear expectation of

a correct answer. In the early stages of my intro. to bio. course, after I

have presented the concept of science as pure or applied, natural or social,

I ask: "Is medicine a natural or social science, and is it pure or applied?"

Students are directed to produce a one or two page essay. I can grade 50 of

these in less than an hour! It will also tell you a great deal about your

students. The ones who confuse "medicine the discipline" with "medicine the

pills that you take" need extra help and this is a good time and way to find

this out.

 

Another good one is: "Can social sciences be pure?" I have previously defined

"pure science" as science pursued for the sake of the knowledge alone.

Applied science, for our purposes, is science used to solve some

human-perceived problem. I define technology as pure (!!!) applied science,

just to make them laugh (Laughter is a good thing in a science classroom,

no?).

 

Later, after we have talked about biotic and abiotic factors in an ecosystem

and the students are reasonably well versed in the organic vs. inorganic

controversy, I ask: "Is the water, carbon dioxide, and oxygen which is inside

of cells biotic or abiotic?" "If biotic, how do they differ from the same

molecules in the air and water around the cells?" "If abiotic, how can it be

that they are vitally necessary for life?" I have a clear idea of the kind of

answer I am looking for here and can pick it up in seconds.

 

A word of caution. It is best to grade "easy" on these kinds of assignments.

Generally, I give at least an 80% award to any reasonable effort. These

assignments are meant to bolster the grade, not kill the students (that is

much too easy for professionals like us). Well written efforts get 90% and

when they hit it on the head (in my opinion, of course) they get a full point

award. I am usually dealing with anywhere from 5 to 20 points per assignment.

By showing them that they can get nearly full credit for a well constructed

argument regardless of the correctness of the answer you do them a truly

great favor.

 

I give less than 80% only for those papers in which the author has failed to

use the obvious resources at hand, such as the textbook, a good dictionary,

or the definitions which we have worked with previously in our class. Even

then I may stop at the 70% mark. The most difficult thing about this kind of

grading is to notice the really innovative answers--they can be easily

overlooked. I have learned a good bit from such responses (At least from the

ones which I have, in my haste to finish grading, noticed!).

 

Above all, respect your students and have fun! If you find teaching to be a

bore or drudgery, then get the hell out and let somebody else have a go at

it!

 

Speaking of hell, does anyone remember Hatlo's Inferno? It was a syndicated

cartoon feature in which a cartoonist named Jimmy Hatlo created cartoon

depictions of specially created scenarios in hell for especially irritating

people. I haven't seen it for years and have no idea what happened to Hatlo,

but I have a Hatloian inferno for teachers who hate students or who are bored

with teaching and continue to do it for the money and/or prestige. They are

forced to sit for all eternity while listening to an 18 year-old girl talk

incessantly about her friends and her make-up.

 

Dave Williams

Science Division

Anne Arundel Community College

Arnold, MD 21012

Email: profdhw@aol.com

Vmail: 410-541-2265

 

P.S. When we say that science is "natural" as opposed to "social", what do we

mean by "natural"?

 

 

PROFDHW@aol.com wrote:

 

>A word of caution. It is best to grade "easy" on these kinds of assignments.

>Generally, I give at least an 80% award to any reasonable effort. These

>assignments are meant to bolster the grade, not kill the students (that is

>much too easy for professionals like us).

 

A good point. Sometimes I think we're so concerned with our status as a

rigorous subject that we can be unreasonable and forget the level at which

our students are operating.

 

>Above all, respect your students and have fun! If you find teaching to be a

>bore or drudgery, then get the hell out and let somebody else have a go at

>it!

 

Teaching is fun, reading student papers is... <g>

 

Lane

----

Lane Lester / llester@athens.net / Athens, Georgia US

 

 

 

Just wanted to add a little to this. I give my upper level and graduate

students concept analysis writing projects. These involve them

considering some central question, read several pertinent articles then

put together a reasonable scientific argument relative to a stance on the

original question.

 

As I am trying not just to teaching them the content but also how to

think through it and how to express it scientifically, this turns into a

fairly significant project (they usually do 2 or 3 per semester). On the

due date, they turn them in. I read and critique (severely) then return

them for "fixing." They turn them in again. I read them again (MUCH

easier to do the second time) and sometimes give them back for further

fixes. I do not award a grade until the paper is A quality or they give up.

 

Interestingly, the 2nd and 3rd papers turn out much better on the initial

screening than the 1st was.

 

I explain a couple of things to my students in this process: 1) as

professional scientists we do not write, we rewrite. 2) I am doing this

to them because it causes them to focus on growth rather than just

turning something in and hoping for the best.

 

We have had some amazingly successful individuals come out of this

approach. Fellow professors have given me much positive feedback and the

students have taken the trouble to call, write, or come back for a visit

to let me know how much the method helped. For most of them it was the

first time a professor wouldn't allow them to get by with a low grade and

poor performance.

 

Best wishes,

Bill

 

----------------------------------------------------------------------------

William A. Hayes, II, Ph.D.

Professor of Biology

P.O.Box 3234

Delta State University

Cleveland, MS 38733

ph: 601-846-4247

fax: 601-846-4016

email: bhayes@dsu.deltast.edu

 

 

 

Dear Biolabbers,

 

I can't believe some of the stuff I am reading on the discussion

group lately. Can anyone spell "grade inflation"?? What do you

think we are setting our students up for? It's setting them up

to get fired! It's not our fault that the high school system can't

put together a decent program of education that stresses the

development of a rigorous work ethic. But we shouldn't perpetuate

that kind of attitude at the college/university level. We

shouldn't be patting them on the back for just showing up for

an assignment. Grading easy - don't make me sick!! Everyone here

knows that the best learning experience they had as an undegrad

or a grad student involved working out a problem, spending whatever

amount of time it took to get the job done. Our current crop

of students don't have a clue about the relationship between work

and productivity. The buck has got to stop somewhere. Let's forget

about faculty evaluations for minute and remember that we are

here for the best interest of the students. I am getting out the

asbestos suit for the flames I'll get for expressing this opinion

but somebody has got to say something. When the average grade

is an A- then what is average??

 

Jay Pitocchelli

 

--------------------------------------------------------------------------

Jay Pitocchelli Voice: 603-641-7397

Biology Department Fax: 603-641-7116

Saint Anselm College jpitocch@anselm.edu

Manchester, NH 03102-1310

--------------------------------------------------------------------------

WWW Home Page http://www.anselm.edu/homepage/jpitocch/welcome.html

--------------------------------------------------------------------------

Note: The preceeding message triggered a discussion on grade inflation. That thread is compiled separately.

Biolab Home Page