Current, common methods used by #highered #assessment professionals when sampling and evaluating student work for general education or program level outcomes assessment projects may not provide results that can be reliably generalized. Even if sampling work from 1500 students in an institution with only 4500 students total, the results may not be indicative of the institution’s true level of performance in teaching general education or program outcomes.

The research reported by Hathcoat and Penn in Volume 7, Winter 2012 of Research and Practice in Assessment analyzed results of an assessment project to consider potential sources of variance in the outcomes for students that had more than one artifact evaluated.

. . . it is of particular interest that 77% of the error variance derives from differences within a single person across each task.

When two different artifacts by the same student yielded different results, very little of the variation could be attributed to inter-rater reliability (12%). The differences were more a result of the variation in the nature of the tasks themselves or of the variation in student performance across multiple tasks.

This study suggests that if researchers want to make comparisons about students’ performance from authentic assessments between institutions or within an institution, they should greatly increase the number of tasks that are sampled for each student, establish statistical controls based on variables that are shown to impact students’ performance (such as motivation), or take steps to standardize some task characteristics (which may not be palatable for users of authentic assessment).

Certainly, there are limitations of the study’s results which are, of course, noted in the article. However, the implications of even the limited results are significant for institutional assessment and research professionals. Outcome centered assessment projects that randomly sample student work aligned to the outcome likely often have a limited number of samples – perhaps only one – from individual students. However, the results for a single student on a single task may not be validly generalized; thus, drawing conclusions regarding institutional effectiveness and accountability based on current, common sampling methods may face serious challenges.

Student centered outcomes assessment projects that sample multiple tasks from individual students (graduates, ostensibly) and generalize conclusions regarding institutional effectiveness based on a representative sample of the student population may be more valid and reliable. (e.g. sampling multiple artifacts from 30% of current year graduates).

A recent article published by Inside Higher Ed focuses on a study that suggests student motivation on low-stakes, standardized exams used for institutional assessment may impact the reliability of results.  That has serious implications for institutions using the results of those exams to report institutional effectiveness regarding student achievement of institutional outcomes.  That article and study is an important read for institutional assessment professionals.

“But my institution relies more heavily on course-embedded assessment, so it’s not as relevant to my institution.”  Not so fast…  I believe the results of that study also have implications for institutional administration of outcomes assessment projects relying on course-embedded assessments.

My institution relies almost exclusively on course-embedded assessment (of the first type I noted in a previous post regarding how “course-embedded assessment” may be interpreted in different ways).  We only use indirect methods to supplement our assessment efforts, and we do not use standardized exams in any manner to assess general education outcomes.  However, the study has serious implications for our administration of our outcomes assessment projects relying on course-embedded assessments.

This past fall, we identified five courses that were most frequently taken by students during their last semester before graduation.  Faculty teaching those courses collaborated to develop an assignment for the respective course that was aligned to our critical thinking general education outcome; the departmental assignments require students to think critically within the discipline.  The institutionally defined procedure was that each faculty member teaching the course would include the departmentally defined assignment as a regular assignment in their course: it was to be “course-embedded.”  Considering the results of the study, the “course-embedded” assignments would not encounter the student motivation issues that, according to the results of the study, occur when students take exams that do not effect their personal achievement within courses.

But for the students taking the exams, the tests tend to be low stakes — no one must pass or achieve a certain score to graduate, gain honors or to do pretty much anything.

I have a speaking engagement coming up next Friday at Howard College in San Angelo.  I’ve been developing a custom presentation to address the specific needs described in my conversations with the colleague that invited me to present.  As usual, the development process takes on a life of it’s own and the presentation slowly emerges during the weeks that I spend preparing it.

Most of the topics I’ve planned to include have been in place for some time, but the organization of it has been evolving quite a bit.  At the moment, the presentation will be a series of issues and challenges I’ll pose to faculty to improve assessment in courses.  Of course, there will be interaction expected; there’s a question and pause every fourth slide or so…  It’s more about them than it is me.

Current assessment issues to be included are below along with questions I may ask along with a short summary or a link to a previous blog post where I’ve discussed the issue.  I’m always open to discussion and comments.

Being primarily responsible at my institution for general education, program, and course level outcomes assessment, a project I helped to initiate this academic year has been the implementation of Blackboard Outcomes that integrates deeply with the Blackboard Learn LMS.  In short, Blackboard Outcomes makes it possible to collect electronically samples of student work so that they may be evaluated – also electronically – against a rubric (e.g. AAC&U LEAP Value rubrics) as part of an institutional or programmatic outcomes assessment project.  With the evidence collection and evaluation process occurring electronically, the reporting process is also greatly streamlined.  Naturally, as we’ve implemented the tool, I’ve encountered a few features that I’d like to have that are not currently available.  One feature that is needed is for Blackboard Outcomes to be able to collect samples of student work submitted to Turnitin assignments.  The full product enhancement suggestion I submitted is below.  If you work with Blackboard Outcomes, I’m interested in your feedback, and your also making the same suggestion to Blackboard.

——

Currently, Turnitin assignments are not available to Blackboard Outcomes to sample student work in an evidence assessment project, and a further complication, as I understand it, is that student work submitted to a Turnitin assignment is stored on Turnitin servers.  This suggestion remedies that issue and makes it possible for student work to Turnitin assignments to be included in evidence collection.

Add the “Create Alignment” option to Turnitin assignments.  When a Turnitin assignment is aligned, Blackboard Learn automatically and seamlessly retains a copy on the Blackboard server of student submissions to the Turnitin assignment. The workflow would be: (a) faculty creates the turnitin assignment, (b) faculty aligns the assignment to an outcome, (c) student uploads their submission, (d) Bb Learn saves a copy to the Bb Learn servers for use by Outcomes, and (e) Bb Learn transmits the student submission to the Turnitin server as usual.  Steps (d) and (e) occur simultaneously and occur without further actions by faculty or students.

Being primarily responsible at my institution for general education, program, and course level outcomes assessment, a project I helped to initiate this academic year has been the implementation of Blackboard Outcomes that integrates deeply with the Blackboard Learn LMS.  In short, Blackboard Outcomes makes it possible to collect electronically samples of student work so that they may be evaluated – also electronically – against a rubric (e.g. AAC&U LEAP Value rubrics) as part of an institutional or programmatic outcomes assessment project.  With the evidence collection and evaluation process occurring electronically, the reporting process is also greatly streamlined.  Naturally, as we’ve implemented the tool, I’ve encountered a few features that I’d like to have that are not currently available.  The most significant of those for Blackboard Outcomes is the ability to specify students from which samples will be collected.  The full product enhancement suggestion I submitted is below.  If you work with Blackboard Outcomes, I’m interested in your feedback, and your also making the same suggestion to Blackboard

—

SUBJECT: Identifying specific students from which to sample work in an evidence assessment project.

AUDIENCE: This would be primarily for assessment managers/coordinators responsible for creating and managing evidence assessment projects.

FEATURE: Make it possible for an evidence assessment project to be initially defined by a list of students from which samples of work will be drawn; that’s in comparison to the current method of specifying first which outcome will be assessed. For example, allow a sample to be defined first by a list of students based on their student ID or other criteria before specifying the outcome(s) to be assessed.  If implemented, there should be an option to set criteria for how many artifacts are collected per student.  For example, assessment coordinators may want one artifact per user if assessing only one outcome.  Or, if assessing multiple outcomes, the fewest number of artifacts possible for an individual student should be selected while covering all outcomes to be assessed (that assumes that one artifact may be aligned to more than one outcome being assessed).  If implemented, reports by individual user per outcome will be needed as well as the aggregate performance of the entire group.

USE CASE 1: Assessing Graduates
We would like to be able to better focus our general education outcomes assessment on the work of our graduates.  The most appropriate time to do a summative evaluation of how well a 2-year institution is teaching students the general education outcomes is to measure the work of graduates in their last semester – maybe their last two semesters.  Measuring earlier than that – in the student’s first year of courses – does not afford the college or the students adequate time to develop skills that would make a summative evaluation valid.  Many two year institutions are limiting their assessment to students that have completed at least 45 hours, but those institutions don’t necessarily have Blackboard and Blackboard Outcomes.  I believe Bb Outcomes offers a unique opportunity that I’d like to leverage, but it requires being able to sample from a particular list of students.

Out institution keeps the last year of courses in the LMS; after that year, the courses are archived.  But, that means that if we wanted to take samples of work from May graduates, we will have available their coursework in Blackboard from Spring and the previous Fall and Summer.  Likewise, if we wanted to take samples of work from December graduates, we will have available their work from that Fall and the previous Summer and Spring.

What we’d like to be able to do is to take a list of our December 2013 graduates into Blackboard Outcomes and set up assessment projects for 2-3 of our general education outcomes that sample work from JUST those students.  If we have 1200 students graduating, I’d like to sample about 300 artifacts for each of the 2-3 outcomes.  That would give us a 25% sample, which we’ve heard is adequate per SACS, and I believe the results from that sample would be a higher quality, more purposeful evaluation of how well we are assessing our general education outcomes.  And, we’d accomplish that with a much smaller collection of artifacts than if we are trying to do what most institutions are attempting now which is a 25-30% sample of all students or of students that have not yet graduated or with a particular number of earned credit hours.

USE CASE 2: Retroactive Portfolio-like Assessment
This one is a secondary angle on the same idea, but I think it’s potentially more powerful, and it might be unique: retroactive portfolio-like assessment.  With 1200 graduates, take a list of 300 students into outcomes and create a project around those 300 students and include every general education outcome.  Blackboard Outcomes should minimize the number of artifacts selected by identifying the least number of artifacts per student necessary to cover all outcomes, e.g. if a student has one assignment aligned to 3 of 7 outcomes and another assignment aligned to the other 4, only two artifacts would need to be sampled.

The manner in which the evaluations are conducted/segmented should be at institution discretion.  (1) One option would be to segment the evaluations by outcome; evaluators could be assigned to evaluate artifacts by a particular rubric.  In that instance, the evaluator would be evaluating artifacts aligned to the chosen outcome; one of the two artifacts mentioned above for a student would be used multiple times: once for each outcome being assessed.  (2) A second option would be to segment the evaluation by artifact; evaluators are could be assigned to evaluate a certain number of artifacts.  In that instance, the evaluator would have multiple rubrics to complete per artifact.  (3) A third option would be to segment the evaluation by student; evaluators could be assigned to evaluate artifacts submitted by a single student.  In that instance, the evaluator would have a body of work to evaluate by a specific student; as they evaluated, they would have all of the artifacts – two in the example mentioned above – for a student and have multiple rubrics to apply to each.

The reports could be segmented in a similar fashion by outcome (similar to the current rollup reports) or by student.  The real value will be in the by student reports.  That report would indicate the success on each outcome for each of the 300 students.  How many were successful on all 7 outcomes?  6 outcomes? etc. With drill down into the individual rubric criteria.  With that report, we get a view of institutional general education outcomes from a portfolio-like perspective — a retroactive portfolio-like assessment.

I’m sure we will identify additional possibilities and use cases that could enhance the value and impact of outcomes assessments by being able to specify students to be included in the sample.  And, the only type of tool that can make that possible is one like Bb Outcomes – an outcomes assessment tool that is deeply integrated with the LMS and that can sample student work from the past year or more (as long as the courses are active in the LMS).

I have suggested previously that NO college level course should have learning outcomes that are written at the lower cognitive levels.  Working from Bloom’s Taxonomy, a college level course learning outcome should NOT be to define, explain, describe, discuss, list or identify.  College level courses should require students to analyze, apply, synthesize and evaluate new knowledge and concepts.  They should be expected to use new knowledge, concepts and skills not simply remember or comprehend them.

The question or issue typically raised is almost always similar to, “Students must acquire basic skills and knowledge before they can operate at higher cognitive levels regarding that content.”  That is absolutely true.  However, my argument is that basic skills and knowledge are pre/requisite to achieving higher order outcomes; students that achieve higher order outcomes of a course will have implicitly demonstrated mastery of the pre/requisite skills.  So, listing the lower order skills as outcomes is both unnecessary, and from an assessment perspective, undesirable.

A theoretical example.  If I am teaching a simple three hour “Basic Baking” course, two appropriate learning outcomes might be to:

• Bake and decorate a cake “from scratch.”
• Evaluate the quality of a cake baked and decorated “from scratch.”

Both of those outcomes require higher order skills.  Students must be able to synthesize and apply a range of simpler skills to effectively bake and decorate a cake, and based on what they learn about the process, they must evaluate the quality of a cake. There are MANY requisite, simpler skills that are required of students if they are to achieve those two outcomes; a few of those may include:

• Use the English measuring system to measure ingredients.
• Use fundamental kitchen equipment such as an oven, a stand mixer, baking pans, or common utensils.
• Identify standards for kitchen cleanliness and safety.
• Describe different methods for determining when a cake has cooked completely.

I contend that listing the requisite skills as outcomes is both unnecessary and undesirable.  First, I do not need to list those as outcomes because the skills will have been taught to and demonstrated by students if they achieve the two higher order outcomes noted above.  And, the requisite skills are NOT the desired outcome of the course; the goals for the course are to bake and decorate a cake from scratch and to evaluate the quality of a cake.  Second, from an assessment perspective, it is undesirable to list the requisite skills as outcomes.  In Texas, state and regional agencies suggest and many institutions are requiring locally the reporting of assessment results at the course level.  Thus, if pre/requisite skills and objectives are listed as course outcomes, assessment results for those outcomes must be reported.  That creates an unnecessary burden on faculty, department chairs, and deans.  In the example above, it is much more efficient and absolutely adequate to report assessment results for the TWO higher order outcomes.  Those two higher order outcomes are the ultimate goals for the course.

Practical Example: GOVT 2301

The Spring 2012 version of the Academic Course Guide Manual published by the Texas Higher Education Coordinating Board includes (but is not limited to) these three outcomes for GOVT 2301 American Government I (page 107 of the PDF).

• Demonstrate knowledge of the legislative, executive, and judicial branches of federal and state government.
• Demonstrate an understanding of federalism.
• Analyze state and local political systems and their relationship with the federal government.

The first two outcomes listed here are lower order, per Bloom’s Taxonomy (setting aside for a moment, the ineffectiveness of “demonstrate” as a verb within an outcome).  The third is a higher order outcome.  Even with my somewhat cursory knowledge of the content for GOVT 2301, I firmly believe that a student capable of analyzing state and local political systems and their relationship with the federal government will have gained a clear understanding of federalism and firm knowledge of the branches of federal and state government.  And, as an assessment specialist and instructional designer, I know I can develop an assessment method for the last outcome that would absolutely require students to achieve the first two outcomes.  Given that, I do not need to list the first two as outcomes because they are implicit in the third, and as a faculty, department chair, or dean, I would not want to have to report assessment results on all three when the results of just the third would be sufficient to meet local/state/regional assessment requirements.

Practical Example: CHEM 1112

The Spring 2012 version of the Academic Course Guide Manual published by the Texas Higher Education Coordinating Board includes these outcomes for CHEM 1411 General Chemistry II Lab (page 42 of the PDF).

1. Use basic apparatus and apply experimental methodologies used in the chemistry laboratory.
2. Demonstrate safe and proper handling of laboratory equipment and chemicals.
3. Conduct basic laboratory experiments with proper laboratory techniques.
4. Make careful and accurate experimental observations.
5. Relate physical observations and measurements to theoretical principles.
6. Interpret laboratory results and experimental data, and reach logical conclusions.
7. Record experimental work completely and accurately in laboratory notebooks and communicate experimental results clearly in written reports.
8. Design fundamental experiments involving principles of chemistry and chemical instrumentation.
9. Identify appropriate sources of information for conducting laboratory experiments involving principles of chemistry.

Developing more effective higher order outcomes would be more effective.  Again, based on my cursory knowledge of the content proper, I believe there could be a single outcome for this lab course that represents a more efficient wording of outcomes 6 and 7.

• After designing and conducting a laboratory experiment, communicate experimental results and conclusions in a written laboratory report.

Achieving that outcome will require students to achieve every outcome currently listed in the ACGM.  As students design and conduct the laboratory experiment, using basic apparatus and applying experimental methodologies will be absolutely necessary – so will demonstrating safe and proper handling of laboratory equipment and chemicals and using proper laboratory techniques.  The same goes for outcome 4, 5, and 6.  Faculty assessing student ability to communicate experimental results and conclusions in a written laboratory report are inherently going to require and measure whether the student designed the experiment based on chemistry principles and instrumentation, used appropriate sources of information, made careful and accurate experimental observations, related physical observations and measurements to theoretical principles, interpreted results, reached logical conclusions, and communicated those results and conclusions.  Why?  Because that’s what chemists do as they communicate experimental results and conclusions in a written report after designing and conducting a laboratory experiment.

ONE OUTCOME ASSESSED EFFECTIVELY REQUIRES ONE RESULT TO BE REPORTED, and students will have achieved every outcome listed currently in the ACGM.  As faculty, a department chair, or a dean, I would much prefer to report the results from the single outcome rather than trying to report students results on each and every implicit process or skill required by that single outcome.

The difference between an outcome and an objective is critical.  I have argued before that indifference to the distinction could present significant issues.  Without the clarity between the two concepts, the development process could yield a long mish-mash of “outcomes” for a course that both complicate institutional efforts to report assessment outcomes at the course level and potentially erode the academic freedom, creativity and responsibility of faculty.  Consider an example . . .

For the COSC 1401 Introduction to Computers course I teach, the course outcomes are:

1. Identify the components of a computer system.
2. Use common applications.
3. Explain the impact of computers on society.
4. Identify computer careers.
5. Identify fundamental programming structures.
6. Identify ethical use of computers.
7. Use basic operating system functions.
8. Discuss current issues associated with security, ethics, and legal issues.
9. Describe uses of the Internet and design a simplistic web page using HTML coding.
10. Demonstrate proficiency in the use of word processing, spreadsheet, database, and presentation applications.
11. Demonstrate proficiency in the use of email.
12. Discuss current issues such as viruses and other related topics.
13. Demonstrate basic understanding of network topology and connections.

For brevity’s sake, I’m going to assume/assert that some of the listed outcomes are more akin to objectives (e.g. 3, 8, 9, 11, 13) and that two or more could be rewritten into a single outcome (e.g. 2, 7 & 10); and I’m not going to get into the discussion as to whether each of them are appropriate for a college-level course (e.g. 1, 4, 5, 6, 12).  For this post, the issues are the difficulty this list of outcomes creates for academic freedom and for institutional reporting of assessment outcomes at the course level.

To meaningfully report whether students, college-wide, are achieving any single outcome, it is important if not entirely necessary that all sections of the course use at least one, agreed upon common assignment that effectively measures the outcome.  While this conversation may need to be much longer at some point, I’ll simply state, for the moment, that it becomes quite cumbersome if not altogether impossible to summarize the extent to which students are achieving an outcome if there are as many different assignments and assessment instruments as there are faculty. (That can be debated in another post, if needed).

In this example, if the institution ends up using one assignment and assessment instrument per outcome, there will be 13 assignments that faculty are required to use in their course.  Reporting on 13 outcomes is cumbersome; collecting and summarizing results for that many outcomes for a single class, in my opinion, will likely overwhelm faculty, department chairs, deans, and assessment professionals.  Further, do 13 required assignments not have a negative impact on faculty creativity, academic freedom, and individual responsibility within the classroom?  After incorporating 13 required assignments into my course, how any other assignments will remain?  How many do I design, develop, and use – as the faculty teaching the course?  Certainly, I will be able to include assignments of my choice, but the required assignments, in my opinion, will trade off with ones I might like to teach.

Do *not* misunderstand me.  I am *not* suggesting that faculty do not have a responsibility to report course assessment outcomes.  They absolutely, 100% do have that responsibility to the institution, the students, and the profession. Academic freedom, faculty independence and creativity do *not* outweigh in importance the institution’s accreditation and accountability requirements.

My argument is that the institution and the profession have a responsibility and a mutual interest with faculty in differentiating between outcomes and objectives so as to have 4-5 outcomes for a course rather than 13.  That leads to gains in efficiency for the institution in reporting classroom assessment results, and in my opinion, it has critical implications for the preservation of academic freedom for faculty.

The previous post highlighted differences between Outcomes vs. Objectives for Institutional Assessment.  As promised, I hope to further clarify the differences by engaging the next step in the institutional assessment planning process: defining and planning assessment methods for a previously developed outcome.  Describing the assessment methods and measures, that may be used will further distinguish between outcomes, assessment, and objectives/tasks within institutional assessment.

For the purpose of discussion, we are focused on an administrative unit responsible for developing articulation agreements for transfer from a community college to a university. The previous post described current practice and common “outcomes” (which are more akin to objectives) and how they are measured.  A more appropriate outcome was developed by asking and answering a question analogous to those used when developing outcomes for instructional purposes:

Question(s):
What are the overarching, high level goals that are relevant to this unit (course) and aligned to institutional (program) goals and mission?  What do we really want stakeholders (students) to accomplish or be able to do because of the work done (curriculum) in this administrative unit (course)?

Answer:
To develop articulation agreements that enable students to transfer successfully to the institution of their choice.

Now, the question is, “What’s an appropriate method for assessing whether students are successfully transferring to institutions of their choice?” More questions will help us identify what we need to know or find out.

• How do we know – directly or indirectly – that students are able to transfer successfully to the institution of their choice?
• How do we know that articulation agreements are helping students transfer successfully?
• Is there a difference between “transferring” and “transferring successfully?”
• Feel free to add others in the comments…

From those questions, we can begin to focus on data or information that may help us determine if articulation agreements are helping students to transfer successfully.

1. How many students transferred?
2. Were there any students that attempted to transfer but could not?
3. Did students lose any hours in transfer?
4. Would students describe their transfer as successful?  Was it easy for them to transfer?
5. Did the hours at our institution transfer to the receiving institution as the student expected?
6. Were students aware of an articulation agreement?
7. Were there particular elements of an articulation agreement that helped them transfer?
8. Did students transfer to a university in general, or did they transfer into a specific program/school?
9. How well did students perform after they transferred?  (“success after transfer”)
10. Did students complete their degree in a reasonable time after transferring? (“completion after transfer”)

With those questions in mind, it then is possible to identify data that may be collected to assess the results for the outcome.  I’m not suggesting that either of these are easy to accomplish or that the data is readily accessible.  There may be logistical issues with these, BUT that does not undermine the appropriateness of the assessment tools.

• A survey of recently transferred students can provide indirect evidence of transfer success.  The survey could address questions 2 through 10, indirectly based on student reports.  Student reported data is more useful for some of those questions than others.
• Data from receiving institutions regarding the number of transfers (perhaps via National Clearinghouse), success after transfer, completion after transfer, and hours applied will be useful to directly assess the outcome.
• A survey of transfer students that earned a four year degree could provide additional, indirect information – perhaps as a follow up to the other survey.
• Reports from educational planners and website analytics for the “transfer” page of the institutional website may provide further direct data regarding student awareness and use of articulation agreements.  Better yet, providing a “point of service” survey ON the page for each articulation agreement could provide additional information, e.g. asking students if the information was understandable, useable, helpful.
• Feel free to add others in the comments…

With those ideas, we can develop a specific protocol for assessing the outcome.

Outcome
To develop articulation agreements that enable students to transfer successfully to the institution of their choice.

Assessment Instruments
Survey of Recent Transfer Students
Transfer Success & Completion Data from Receiving Institutions
Website point of service survey data and transfer page analytics

Measurement
The data from those instruments may be aligned to and provide insight regarding:
(a) awareness and use of articulation agreements,
(b) student reported ease and success of transfer,
(c) quantitative institutional data regarding hours applied, and success/completion after transfer.

The measurement will allow benchmarks to be set and goals to be identified year over year in an effort to continuously improve the results achieved by the institution regarding the articulation agreements it develops with other institutions.  The measures will not change substantially year to year, and the instruments will not change substantially year to year (only improvements that are devised/made).  There should be stability in the outcomes, assessment methods, and measurements to allow consistent evaluation of institutional performance and improvement from year to year.

So what happen to the statements and activities that are currently labeled as “outcomes?”  The statements like:

• Increase the number of articulation agreements with four year institutions.
• Increase the number of students transferring to four year institutions.

Those are now more clearly, annual objectives or strategies to improve our performance on the outcomes.  Of course, as noted in the previous post, there are better objectives to be written than those two.  In the last post in this three part, series, I will focus on how the outcome, assessment methods, and measures may guide the development of better objectives/annual strategies, and how those objectives/annual strategies may be used properly within the institution (rather than as outcomes).

“Measuring Time” by Louise Docker (aussiegall @Flickr)

I’ve written previously about the difference between outcomes and objectives.  During a recent institutional effectiveness workshop, I’ve come to the conclusion that the same issue exists when assessing administrative units.  From what I’ve experienced and seen, current institutional effectiveness and continuous improvement practices for the administrative assessment efforts lead to the same blending of outcomes and objectives that I’ve observed and noted on the instructional side of the house.  A few examples and an explanation of how to approach it differently . . .

For purpose of discussion, I’ll use an area in which I work as an example: developing articulation agreements for transfer from a community college to a university. A common outcome currently used to measure the effectiveness of an administrative unit that develops those agreements might be:

• Increase the number of articulation agreements with four year institutions.
  or
• Increase the number of students transferring successfully to universities.

In current practice, the current number of articulation agreements and the current number of students transferring would be used as benchmarks, and a goal might be set for the administrative unit to develop a specific number of new agreements in the academic year to facilitate a greater number of transfers.  The unit would then be “assessed” at the end of the year by determining the extent to which the goals were achieved.

As currently used, there are two concerns to note regarding these “outcomes.”

First, each example is more akin to an objective rather than an outcome, in my opinion.  As I’ve defined it, “an objective is much more specific and serves to clarify, in part, what the outcome means.”   The objective provides “a very good idea of the specific activities” to be engaged by the unit.  An “outcome, in contrast, is more of a high level outcome that points everyone in the same direction but does not provide specific activities to be engaged.  Given that contrast, the two items listed are very specific in nature; they are objectives, not outcomes.

Second, the two objectives are substantially different in nature which I believe creates a further lack of clarity in institutional assessment.  The first is an action taken by the articulation development unit.  The second is a result of actions taken by the unit.  Objectives should focus on actions taken by the unit rather than the result of their actions; the results should be incorporated into the assessment methods.  Blending the two as objectives creates a lack of clarity in institutional assessment.

So, what would be an effective outcome for the administrative unit?  And, what’s the process for developing the outcome?

I believe we can borrow from the instructional side for this process.  Outcomes are developed at the curricular/institutional level, and objectives can be developed at the course/yearly level.  Outcomes in a course do not change regularly; they are more static.  They may be updated occasionally, but by and large, there’s a great deal of consistency.  Similarly, the high level outcomes and efforts of an administrative unit, and the manner in which they should be assessed, should also be somewhat static.

The question for the administrative unit is one that may be asked when developing outcomes for a course, “What are the overarching, high level goals that are relevant to this unit (course) and aligned to institutional (program) goals and mission?  What do we really want stakeholders (students) to accomplish or be able to do because of the work done (curriculum) in this administrative unit (course)?”

A desired outcome by which the unit could be assessed is . . . To develop articulation agreements that enable students to transfer successfully to the institution of their choice.

That statement provides a high level perspective of what the unit does; that is the core business of that function within the institution.  That statement is an outcome.  It points all activities and personnel in the same direction and provides greater clarity of purpose.  Using it as the centerpiece of objectives planning will improve the objectives developed.  For example, consider one of the two objectives noted earlier.

• Increase the number of articulation agreements with four year institutions.

The outcome developed requires a more informed objective.  To align properly to the outcome, the objective should not be to only increase the number of articulation agreements with four year institutions; it needs to be more explicit by focusing on the institutions students transfer to more frequently.  That may require a pre-requisite objective be included.  Perhaps the objective becomes,

• Identify and increase the number of articulation agreements with the 5 institutions to which our students most frequently transfer.

The outcome has guided us to developing an objective that better meets the needs of our students.  Certainly, folks working on the agreements may be doing it intuitively, but if the planning processes do not get to the point of specifying “the 5 institutions to which our students most frequently transfer,” it’s not as effective as it could be.

What’s the next step?

With the effective outcome developed and before developing the objectives, the next step is to determine how the outcomes should be assessed.  The assessment metrics will be used year to year and not change on a regular basis.  That forms the foundation of institutional assessment and effectiveness for the department.  With that foundation laid, departmental planning becomes more effective – more focused and more efficient – by focusing specifically on the objectives necessary to achieve the outcomes.  Unfortunately, current institutional effectiveness efforts seem to focus on developing objectives and measures every year.

I will address the next steps – identifying assessment methods and developing objectives – in a subsequent post.

CHANGE!  My online, professional digital footprint is changing, evolving, updating.  EdTechatouille, the focus of the site for the past 6+ years, is disappearing into the background.  I have given the site a slight makeover to highlight my current professional focus and interest: learning and assessment in higher education. It’s a subtle change to the presentation of the site, but the substantive change is significant.

Like more recent posts I’ve written, the emphasis from this point forward will be on classroom assessment, program review and evaluation, assessment of general education outcomes, curriculum development, general pedagogy, institutional effectiveness, strategic planning, institutional assessment, or accreditation related news and issues. Occasionally, there may be posts about my own teaching activities, educational technology, emerging instructional technologies, computer literacy of college learners, distance learning theory and practice, or general education and technology news of interest.

I believe the change to the site provides a renewed sense of purpose in writing and sharing online.  It’s almost seemed as though the EdTechatouille branding/purpose of the site has inhibited my writing and sharing process over the past 2 years or so that I’ve become focused on assessment and accreditation.  I intend to write and share more about learning & assessment in higher education, and I hope that brings to this site and to my professional network new colleagues and friends.

Bloom’s Taxonomy is exactly that – a taxonomy, not a heirarchy.  And, students of many ages are capable of higher order thinking skills (e.g. my then 9yo daughter) like application, analysis, synthesis and evaluation; of course, students will exhibit different levels of proficiency and different levels of complexity in their thinking.

Here’s my question… Should there be any college level course that has learning outcomes that are predominantly, or worse yet, entirely at the lower levels of thinking per Bloom’s Taxonomy?  Is there any instance in which the outcome of the course should not be an ability to analyze, apply, synthesize or evaluate content related to the discipline?  Certainly, there will be objectives within the course (which are different from learning outcomes of the course) that will focus on lower order thinking skills as a matter of introducing and scaffolding the learning process for learners.  However, is it  appropriate for a college level course to ONLY require students to comprehend content by describing, identifying, listing etc?

For example, BUSG 1301 Introduction to Business within Texas’ Workforce Education Course Manual (WECM) is a course focused on “Fundamental business principles including structure, functions, resources, and operational processes.”  The learning outcomes for the course are

• Describe the scope of global business enterprise;
• identify business functions of accounting, management, marketing, and economics; and
• describe the relationships of social responsibility, ethics, and law.

Even in an introductory course, learners could more meaningfully engage the content through outcomes focused on higher order thinking skills.  I’m certain we (subject matter experts and curriculum/instructional design experts) could collaboratively develop more effective learning outcomes focused on higher order thinking skills that would be more appropriate for a college level course (e.g. Analyze the social, ethical and legal implications of business decisions).  Any suggestions?

A second example, GOVT 2301 American Government I from Texas’ Academic Course Guide Manual (ACGM, Spring 2012 revision) is a course focused on “Origin and development of the U.S. and Texas constitutions, structure and powers of national, state and local government including the legislative, executive, and judicial branches, and federalism.”  The learning outcomes are:

• Explain the origin and development of constitutional democracy in the United States.
• Explain the origin and development of the Texas constitution.
• Describe separation of powers and checks and balances in both theory and practice.
• Demonstrate knowledge of the legislative, executive, and judicial branches of federal and state government.
• Analyze state and local political systems and their relationship with the federal government.
• Demonstrate an understanding of federalism.

Only one of the six learning outcomes for that sophomore, college-level course is at a level beyond knowledge and comprehension.  Certainly, to engage more in depth thought and learning regarding American government, it is necessary to comprehend the origin and development of state and local government, the separation of powers etc.  However, is that an appropriate outcome for a college level course?  Would it not be more desirable and appropriate for college students to use that knowledge to analyze a current political issue within the context of separation of powers or checks and balances?  Or to analyze the roles engaged by the different branches regarding a particular issue or piece of legislation?

The Chronicle of Higher Education asked,

Is it time for more widespread reform of college teaching?

This series explores the state of the college lecture, and how technologies point to new models of undergraduate education.

Last month, we began inviting students across the countries to fire up their Web cameras or camera-phones to send us video commentaries about whether lectures work for them.

Chronicle.com/LectureFail displays a number of student comments, including a compilation, along with several faculty responses.

As a faculty member, as I watched several of the videos, I found my beliefs and attitudes to be more in line with the students than my faculty colleagues.  Personally, lectures are boring… for me… as a faculty member. I don’t like them, and pedagogically and historically, I find them to be an outmoded approach to teaching and learning. Why?

One caveat before a few thoughts… As I read/watch comments, I did interpret the question of #lecturefail to focus on learning experiences/courses which rely heavily or exclusively on lecture.  It does not seem that students are commenting on nor faculty are defending a moderate use of lecture as a learning tool.  The comments from both groups do not suggest a class in which a 30 minute lecture is followed by 30 minutes of discussion or project work, or a course experience that includes a lecture or two to support a 3 hour field experience; all comments seem to suggest a course which is entirely lecture based.

First, historically.  At the very least, lectures serve a once-relevant but now obsolete purpose: to disseminate information and knowledge.  Dating back at least to medieval universities, the purpose of the lecture was to convey information from an original source to learners; there was ONE paper source of a particular text, so the professor read from the text and learners took notes – transcribing their own copy.  That purpose and function of the lecture was, at least, diminished with the advent of the printing press, and with the development of the internet and the ubiquitous availability of and access to information, it is entirely  unnecessary to disseminate information to learners.  They already have access to more information than can be consumed in a lifetime.  The range of available technologies makes more than a few traditional classroom learning methods obsolete, and in some cases, inconsistent with established and accepted theories and practices of teaching and learning.

Second, pedagogically.  I have firm beliefs regarding what constitutes effective teaching and learning; it can be defined explicitly and succinctly. Those beliefs are based on a thorough knowledge of learning theory and practice gained through my own teaching and classroom experiences – as a professor and as a student.  I do not believe a lecture inherently precludes an active, collaborative and authentic learning experience, but I believe it may certainly inhibit it.

For a lecture to be a meaningful and effective learning experience, students must have the ability and habits of a scholar’s mind to engage the content; they must be able independently to engage the lecture by actively processing new information, by relating and applying concepts to situations relevant to their academic or future professional career, by choosing to engage classmates or others in the learning process.  If they do not possess those skills, a lecture ends up being a very passive, solitary and uniquely academic experience – everything that is NOT effective teaching and learning.  Is it a 18-21 year old undergraduate student’s fault if they do not yet possess the habits or skills of a scholar?  It could be; there are certainly students that just don’t care and don’t make the effort.  However, a lecture focused on content does nothing to model, instruct or otherwise facilitate learners engaging a scholarly process of interacting with and applying new knowledge.  If a learner does not possess the meta-cognitive awareness to manage and direct their own learning experience during and following a lecture, that is not their failure alone.

My own experience was that I did not fully develop my own meta-cognitive awareness until I was several years into my graduate career; looking back, I had started engaging lectures more independently during the last year or two of my undergraduate education, but I was not fully aware of what I was doing or the process in which I was engaging.  During those times though, I was working with and learning from faculty that were deliberately teaching scholarly, academic habits of mind; they weren’t just delivering content.  Given that experience, I believe fewer undergraduate learners than faculty think or believe have progressed far enough into their academic careers to be fully prepared to make a lecture the learning experience it could be.  Further complicating the issue, perhaps, is that faculty DO possess the skills for a lecture to be truly and deeply meaningful, so the lecture, as an instructional tool, seems much more effective than it actually may be.

Professionally and personally, I’d prefer to assume that most of my undergraduate students have not yet had an opportunity to fully develop the requisite skills to learn actively, collaboratively, and authentically from a lecture.  For me, methods other than lecture are much more likely to facilitate an active, collaborative and authentic learning experience.  I believe other methods are more likely to be effective even though a lecture is much less work, on my part, than designing, developing, implementing and effectively facilitating an active, collaborative, and authentic learning experience for my students.

I work closely with end of course evaluation surveys.  At one institution, I administer the online survey system through which we survey students, and for the other institution, I rely heavily and place high value on feedback from students to help me continuously improve the course.  My question is, “How much is that feedback worth?”Based on research in the field and discussions with colleagues, the greatest influence on student motivation to complete end of course evaluations is the student’s belief that the instructor/faculty values the feedback and will actually use the information they provide.  The second strongest influence on student motivation to complete a course survey is a local incentive; an incentive specific to the course like extra points on an assignment or as a separate class grade.  Other factors could be global incentives like entry into a college wide drawing for a particular prize; many institutions have given away iPads or netbooks or other gadgets; one with which I’m familiar has used scholarships for the next semester.

In discussion with colleagues, faculty have differing opinions regarding the validity of local incentives that have an impact on the course grade.  The argument that completing an end of course survey is not a valid assessment is quite accurate; in that instance, a portion of the student’s grade is based on something other than their performance in the classroom.  I absolutely understand and respect the argument.

With that said, if I can go from a 16% response rate to a 90%+ response rate just by offering extra points on the final exam, is the feedback for me not worth giving students a bump on the final exam?  If the extra points amount to a 1 point bonus on the course average? a half point?  1.5 points? 2 points?  What’s too much?  We are talking about a significant source of “quality control” for the course?  The teaching and learning environment I provide and facilitate is the service I’m providing; the end of course evaluations are a source of quality control for that service.  How many extra points is it worth to get feedback in a single section from 24 students rather than 4?

Your thoughts?

With a significant focus on the evaluation of our institutional general education curriculum/program, one concept I’ve encountered frequently of late is “course embedded assessment.”  However, I’ve discovered at least two different interpretations of the concept. 

Back up a small step . . .

When evaluating the effectiveness of instruction, there are direct and indirect methods; the Texas A&M Office of Institutional Assessment defines those two concepts fairly well, for reference as needed.  Direct methods are those which “require students to produce work so that reviewers can assess how well students meet expectations.”  As a direct measure, “course embedded assessments” are those that students complete during a course; that is in contrast to assessment methods like certification/licensure exams, other standardized exams, student portfolios, or other work that students may do – in addition to coursework – for the purpose of assessment and evaluation.  However, in conversations focused on institutional evaluation of general education outcomes, I have noticed two different levels of the extent to which assessments are embedded.

First, some institutions are leveraging only the artifacts students produce during a course; they sample a number of sections of a particular course and all student work from a particular assignment.  For example, to evaluate general education outcomes, a lab report assignment from 15 different sections of a microbiology course may be sampled.  Those assignments are collected and graded by a group of faculty (other than the faculty teaching the course) against a rubric measuring level of student attainment regarding critical thinking.

Second, other institutions are leveraging the artifacts students produce during a course AND the assessment of the work also occurred during the course.  For example, to evaluate general education outcomes, the same lab report assignment from 15 different sections of a microbiology course may be sampled.  Those assignments AND the assessment of the work by the faculty member – against rubric criteria or an entirely separate rubric measuring level of students attainment regarding critical thinking – are aggregated.  In short, the assignment is not regraded outside the scope of the course; the institutional assessment effort relies on the grading of work by faculty teaching the course.

If you work with evaluation of general education program/outcomes in your institution, what is your understanding of “course embedded assessment?”  Which interpretation represents your institution’s approach?  What are the pros and cons of implementing each approach?

After three iterations of the course I’m teaching, I’m revisiting and potentially revising the grading rubric I’m using to assess learner participation in discussion forums.  Back in August, I described the types of discussions in which my students in COSC 1401 Introduction to Computers are asked to participate and posted the grading rubric for assessing their participation.  I have been using that rubric the last three terms (I’m teaching primarily 8 week terms; two last fall and one so far this spring).  But, it’s not quite a perfect fit to how the discussions have progressed and how I want to grade them.  So, I’m revising.  I’m interested in your thoughts on this rubric.