More details on federal testing

While it came out just before the President Bush's Commission on the Future of Higher Education's latest position papers changed the terms of debate, The Philadelphia Inquirer had a detailed article on the commission last week. Written by staff writer Patrick Kerkstra, it was published March 28 under the headline "Colleges Pushed to Prove Worth: Standardized Testing at the University Level." It fills in some of the gaps in what we've learned so far, both about the commission and the standardized test its chairman has touted, called the Collegiate Learning Assessment developed by RAND Corp., the military research and development contractor, and a RAND subsidiary known as the Council for Aid to Education.

Among other people on both sides of the issue, Kerkstra interviewed commission chair Charles Miller, its leading advocate of standardized testing. And his report suggests more details of Miller's background and current thinking than we had before:

"The pressures for accountability are everywhere," Miller, a former Bush-appointed leader of the University of Texas System Board of Regents, said in a recent interview. "Evidence of the need to improve student learning is pretty clear."

He offered a litany of examples: "softening curricula," "grade inflation," and insufficient literacy skills in half of all four-year college graduates, as detailed in a study funded by the Pew Charitable Trusts released in January. Meanwhile, annual tuition hikes are outpacing inflation.

To his critic's ears, Miller's case for collegiate testing has a familiar ring. They say similar arguments were used to turn the No Child Left Behind program into a federal fiat, mandating extensive testing in secondary and elementary grades. Miller, in fact, helped design a K-12 testing system in Texas for then-Gov. Bush that became the model for the federal program.

Miller dismissed the comparison. The states, not Washington, should take the lead on collegiate testing by requiring it at public universities, he said. Once the big state systems prove its value, he predicted, testing will be swept by market demand into private schools.

Also, unlike No Child Left Behind, federal funding would not be tied to test results, he said.

Miller has been saying this consistently. But both he and U.S. Education Secretary Margaret Spellings also say financial aid can be used to leverage accountablity. Adds the Inquirer's Kerkstra:

Money, however, is undeniably part of the issue. When the commission was formed, Spellings noted that the federal government provides a third of all higher-education funding and has a right to "maximize" its investment.

"We're missing valuable information on how the system works today," she said, "and what can be improved."

That baffles some ivory tower habitues who see higher education as too preoccupied with self-examination and ranking.

One is Pennsylvania State University president Graham Spanier. Kerkstra quoted him:

"There is no enterprise in America that I know of that assesses itself so carefully and so frequently," said Penn State's Spanier, calling it "both a science and an obsession."

He cited the arduous reviews that faculty members endure to make tenure, the accreditation process, the student-satisfaction surveys, and the monitoring of graduates' efforts to get jobs.

Better known to the public are the college comparisons made by popular publications such as U.S. News & World Report. Those rankings are based on such factors as faculty-to-student ratios, SAT scores, and alumni giving. But they say little, if anything, about how well students are learning in the classroom.

Kerkstra's citation of U.S. News & World Report rankings hardly inspires confidence in his knowledge of assessment issues, but he talked to at least some of the right people. He says:

The solution is not standardized testing, many academics say, but assessments that gauge each student's mastery of a discipline. For instance, a "capstone" course, or a senior-year research paper, or a portfolio of work covering a college career.

Trudy Banta, a professor of higher education at Indiana University and an assessment expert, said that such assignments - combined with satisfaction surveys and scores on graduate and professional school exams - are better indicators of student achievement.

"We all love simple, easy answers," she said. "But this isn't a simple, easy issue."

Kerkstra devoted the most space to a description of Lehigh University's experiment with the Collegiate Learning Assessment, the RAND Corp. standardized test that Miller favors. His discussion is of interest because it gives more details of the preferred test. He says:

Can standardized testing fill in the blanks?

That's what Lehigh University wanted to find out when it administered a standardized exam known as the Collegiate Learning Assessment for the first time last fall to about 100 randomly selected freshmen, according to Carl Moses, deputy provost for academic affairs.

Lehigh is the only Pennsylvania school to acknowledge experimenting with the assessment; none in New Jersey is known to be trying it. Most of the pilot schools are in states where college testing has become a prominent policy debate, such as Texas, New York and California.

The exam is made up of two 90-minute writing exercises. In one, students are given an opinionated statement and asked to compose an essay supporting or disputing it. The second is a real-life "performance task," such as producing a memo from newspaper clips and documents.

The test was developed by two think tanks, the Rand Corporation and the Council for Aid to Education. They employ graduate students to grade the task portion, but a software program called "e-rater" scores the essays. The same program is used to assess writing samples in the entrance exams for both business and graduate schools.

At Lehigh, it's too early to know whether the test has value, Moses said.

Kerkstra doesn't mention the connection, but the Council for Aid to Education is a RAND Corp. subsidiary set up to "enhance the effectiveness of corporate and other private sector support in improving education at all levels and to help education institutions more effectively acquire private support for their programs." All of this bears watching.

Kerkstra's story in The Inquirer also includes a brief description of the CLA and sample test questions, provided by the Council for Aid to Education:

The Collegiate Learning Assessment, a new standardized exam, is supposed to measure college students' communication and reasoning skills. It consists of two 90-minute writing tests. On one, students are presented with an opinionated statement, or "prompt," and asked to respond in an essay. On the other, they are given a real-life task, such as writing a memo based on documentation provided.

Sample Writing Prompt

Public figures such as actors, politicians and athletes should expect people to be interested in their private lives. When they seek a public role, they should expect that they will lose at least some of their privacy.

Sample Performance Task

You are the assistant to Pat Williams, the president of DynaTech, a company that makes precision electronic instruments and navigational equipment. Sally Evans, a member of DynaTech's sales force, recommended that DynaTech buy a small private plane (a SwiftAir 235) that she and other members of the sales force could use to visit customers. Pat was about to approve the purchase when there was an accident involving a SwiftAir 235.

You are provided with the following documentation:

1. Newspaper articles about the accident

2. Federal accident report on in-flight breakups in single-engine planes

3. Pat's e-mail to you and Sally's e-mail to Pat

4. Charts on SwiftAir's performance characteristics

5. Amateur Pilot article comparing SwiftAir 235 to similar planes

6. Pictures and description of SwiftAir Models 180 and 235

Please prepare a memo that addresses several questions, including what data support or refute the claim that the type of wing on the SwiftAir 235 leads to more in-flight breakups, what other factors might have contributed to the accident and should be taken into account, and your overall recommendation about whether DynaTech should purchase the plane.

Miller says the test measures college-level "critical thinking, analytic reasoning, problem solving, and written communications." But if the sample questions in the Inquirer are representative, I don't see how it can.

I guess a computerized "e-rater" might be programmed to score the essay, but the whole test strikes me as breathtakingly superficial. The first question obviously calls for a high school-level five-paragraph essay format (a paragraph each for actors, politicians and athletes, an introduction and a conclusion). And the second "Performance Task" question looks equally superficial, although it would make a dandy prompt for a high school essay with a paragraph each for technicial features of the airplane, liability issues and purchasing procedures.