CEAC

CEAC Update Vol. 1, Iss. 3 -- Now Available!!

2010-04-08T14:22:00.004-04:00

The newest edition of the CEAC Update is now available on our site! Click here to find out read a conversation between client and evaluator, as well as CEAC news and our Ask CEAC! feature.

CEAC Update Coming Soon!

2010-03-25T11:19:00.004-04:00

A new issue of CEAC Update is in the works!

This issue will include an enlightening conversation between evaluation practitioner and client, all about utilization of evaluation results.

Look for it soon!

If you haven't received previous CEAC Update newsletters from us and would like to subscribe, send us an email to subscribe ceac@pitt.edu.

Pennsylvania Assessments

2009-11-23T13:18:00.001-05:00

Assessment in Pennsylvania is taking a turn from the strictly mandated annual assessments to a system of assessments that include summative (end-of) assessments, benchmark (a little check-up) assessments and more classroom oriented, content-focused assessments.

The summative assessments are the No Child Left Behind (NCLB) and other federal- and state-mandated assessments which include the Pennsylvania System of School Assessment (PSSA) and its alternate (PASA) and modified (PSSA-M) versions, the ACCESS for ELLs, NOCTI, and NAEP. The PSSA is an annual assessment in Reading and Math (grades 3-8 and 11), Science (grades 4, 8, and 11), and Writing (grades 5, 8, and 11). Only the PSSA Reading and PSSA Math results contribute to a school and district’s AYP status. While NCLB mandates the administration of a science assessment, the law is silent on its use in determining AYP. The PSSA Writing assessment is state-mandated.

The ACCESS for ELLs is an English proficiency assessment administered annual in grades K-12 and is used for accountability of Title III programs. The NOCTI (National Occupational Competency Testing Institute) is administered to Career and Technical Education completers in their area of preparation, such as Automobile/Automotive Mechanics Technology/Technician and Electrical and Power Transmission Installers. The NAEP (National Assessment of Education Progress) is administered to a representative sample of schools in the state and across the nation. The subjects and grades assessed each year vary. The NAEP results are the basis of the “National Report Card.”

Benchmark assessments mirror the content of the PSSA. Their purpose is to gauge performance or readiness for success on the PSSA. The most frequently used benchmark assessments among Pennsylvania school districts are 4Sight, Acuity, and Performance Series.

Formative assessments are very brief assessment, administered very naturally during the course of instruction to gauge whether students understand the content being presented.

Within the past year, the PA Department of Education launched the SAS (Standards Aligned System) website. The six components (or circles) of a Standards Aligned System are: Clear Standards, Fair Assessments, Curriculum Framework, Instruction, Materials and Resources, and Interventions. Over the past year, various content areas have been populated into this site.

On December 7, 2009, there will be a new launch of the SAS website. The new site will be much more integrated in that teachers will be able to go into the system and search for a topic, such as fractions. Then, all of the six circles will be available and linked to their particular area of fractions. So, if teachers wanted to know what the standard actually states for fractions, they will then be able to see instructional resources for fraction, sample lesson plans for fractions, assessments for fractions, and so forth. In the past, users would enter by selecting one of the six circles. Now they’ll enter with a topic and will have access to all six circles related to that topical content area.

When the site launches, only a sampling of materials will be active on the site. Lessons plans will be built first. Then educators will be encouraged to submit their own materials to the site. Procedures for submitting materials for inclusion on the site as well as procedures for the vetting of all submitted materials will be forthcoming. PDE will be providing training on the website throughout the Commonwealth in early 2010. Trainings are planned for the Western Pennsylvania area.

--contributed by Dr. Shula Nedley

Some thoughts on "Evaluation perspective: Outside or inside"

2009-10-14T14:43:00.004-04:00

On such issues I go to Michael Crotty’s The Foundations of Social Research (2003). I think formative evaluation obviously aligns with constructionist epistemology, but even more specifically Patton’s Utilization-Focused evaluation “wreaks” of evaluators and programs “emerg[ing] as partners in the generation of meaning” (p. 9). I think it would be interesting to expand this piece and consider evaluations that are steeped in objectivist and subjectivist epistemology, which I think you allude to in the last two paragraphs. Summative evaluation of the external audit persuasion ties nicely to an objectivist theory of knowledge where the expert need only cast off any cultural baggage and look hard enough and in the right places to find the answer to whether or not the social intervention has yield predetermined objectives. To me, external audits fit nicely with objectivism. However, for me the really interesting connection is between subjectivist epistemology and hierarchical formula driven evaluation, as exemplified by Adequate Yearly Progress. In this context policy-makers, who often are neither “experts” in the social intervention (education) or evaluation, inform the program of the meaning that has been made by the powers that be, what knowledge the program is responsible for creating, and how it is to be measured. Basically, a formula for success is applied to the program by policy-makers. In this case “meaning does not come out of an interplay between subject and object but is imposed on the object by the subject. Here the object as such makes no contribution to the generation of meaning” (p. 9). Thus educators are reduced to technicians applying a grand algorithm to produce social results. If only it were that easy. It becomes a reductionist approach to the socially (e.g. culturallly, politically, and economically) complex undertaking that is public schooling.

Evaluation Perspective: Outside or Inside?

2009-10-14T11:02:00.000-04:00

Evaluators often debate about the role or stance that an evaluator takes with clients. Some would suggest that evaluators can only serve a client well if they stand somewhat distant from the client and program, a dispassionate observer. Other suggest that evaluators are of most use when they are more closely aligned, perhaps even immersed, in the project.

The debate centers on our philosophies of knowledge --- how we recognize and value what counts as knowledge. Some align with a more objective view of the world from an external source, others assign primary value to the lived experience from more of an insider’s viewpoint. Variations of this debate have flourished in academic and evaluation circles for many years and no doubt will continue for many more. The persistence of the issues across time speak to their importance. Clearly, both positions in the debate, and both of the perspectives or viewpoints that evaluators can offer, have merit. Often internal and external evaluators, or program stakeholders and evaluators, work together to bring both the external objectivity and the internal experiential views to the table to better inform decision-making.

Recently, evaluation experts discussed this topic during part of a forum from the American Evaluation Association. John Mayne, an evaluator from Ottawa, Canada, offered: “One thing an evaluator ought to be passionate about is evidence: the need for robust evidence to inform decisions on and beliefs about interventions, including those one believes in.”

Evidence is the basis for evaluative inquiry. Evaluation repeatedly poses the critical question: What evidence supports the contention made? The “rub,” of course, is in what constitutes “evidence.” Rather than a simple assumption, it may be important to realize that what counts as evidence for one person or group may not be as highly valued for another. Evaluators and clients/stakeholders must consider a further question: What is the burden of evidence needed and valued to assure use of the evaluation? The answer to this question may require multiple sources and types of evidence that can provide a credible foundation for discussion among varied stakeholders.

Qualified evaluators are equipped to navigate and guide these discussions and decisions, assuring that the best forms of evidence are explored to meet the needs of decision-makers and stakeholders. This is not tampering with the evidence or “cooking the data” --- it is the ethical responsibility of the evaluator to fully and accurately represent the actual data --- but making sure the appropriate data sources are tapped, and matching them with the needs of the decision circumstances (people and perspectives, timing, availability, burden to participants, cost, etc.) --- is a crucial and often overlooked part of evaluation planning.

Welcome!

2009-09-29T12:15:00.000-04:00

Welcome to the Collaborative for Evaluation and Assessment Capacity's new blog!