The plea is simple.  Please find your way to becoming an education company and not just an educational technology and distance learning company; I believe your value (and profitability) will increase dramatically within, at least, the higher education segment of the industry. The effect you could have on the quality of teaching and learning and the future of higher education is infinitely greater if you were to focus on assessment across all delivery methods rather than the latest and greatest technology features, tools, and and user interfaces. Why? Assessment is the single greatest issue facing every institution; but currently, Blackboard appears to be “missing the boat” regarding assessment even though there’s several, readily accessible corporate strategies that could rectify that issue.

For higher education, the single greatest issue facing most every institution is assessment. At the core, assessment is education; without improving assessment methods, the effect and influence of other pedagogical improvements is inherently limited. The most “fantastically innovative” use of blogs, wikis, web conferences, collaborative tools, mashups, or any other technology tool becomes essentially useless if it ends with a tired, well-worn and often mis-aligned assessment method. Further, accreditors and every educationally-centered foundation in North America seems to be focused on assessment within the national discourse surrounding higher education: regional accreditors, Lumina’s Degree Qualifications Profile, Gates Foundation Post-Secondary Success. Assessment and the subsequent continuous improvement efforts are absolutely critical across the entire institution, not just in the classroom; the methods and tools for addressing assessment and institutional effectiveness across the entire institution must improve. Finally, competency-based education appears as though it will continue to disrupt the industry; assessment as a means of measuring student credentials – rather than seat time and the carnegie semester credit hour – underpins the entire competency-based education movement.

Blackboard, from my admittedly outsider’s perspective, appears to be “missing the boat” regarding assessment. The assessment tools that do exist in the Blackboard LMS appear to be limited to a “distance learning only” type framework. Rubric assessment is possible, but ONLY if a student submits something to the system. If Bb had all of education and assessment in mind – classroom included – it would be possible to create an “observation” assessment item that would allow a rubric to be scored without a student submission being required; that would enable classroom observations to be scored within the Bb environment. Further, given my 3-4 years of experience with Blackboard Outcomes and 10+ years of experience with the assessment tools in the LMS, the advances made in assessment seem marginal at best in comparison to changes and “improvements” to the technology and interface tools. The current focus appears to be on revising the user interface, and that may be at the expense of further development of assessment tools. The user interface is “nice” but it does not address assessment. There are a number of features that are desperately needed and have been “on the roadmap” for several years that have not been realized. Blackboard Outcomes and LMS Assessment tools appear to be entirely absent from assessment and related (accreditation) conferences. Colleagues specifically looked for but did not find a Blackboard presence at the December SACS meeting attended by almost 5,000 higher education professionals. Worse yet, Blackboard Outcomes and LMS Assessment tools appear to be entirely absent within their own ecosystem of Blackboard World: the call for proposals for Blackboard World 2015 had only one use of the word “outcomes” and that was at a classroom level rather than an institutional one AND Blackboard Outcomes was omitted entirely from the list of Bb products to which a proposal may be related.

What do I believe could or should be done to rectify this issue, including some very specific features? Prioritize development resources around assessment tools. Give Blackboard Outcomes and LMS Assessment tools each (a) dedicated leadership and organizational structures within the company, (b) independent development/programming resources that are not shared with other projects or initiatives, (c) dedicated and knowledgeable support and documentation teams that specialize in all things assessment, and (d) the resources to engage clients through all available channels (including client advisory boards and assessment conference centered travel). Second, revisit institutional effectiveness and assessment tools that were deprecated from Outcomes Assessment. If there’s one non-instructional related feature that’s justifiable within an instructionally-centered tool, it is a tool that addresses assessment and institutional effectiveness for administrative units; those tools existed previously in Blackboard Outcomes but were deprecated and removed. There is a group of Outcomes client self-organizing to meet-up at Blackboard World. Come join us and engage the conversation.

What’s the benefit for Blackboard? At the very least, attending to the assessment needs within higher education may radically improve the Blackboard ecosystem as a product solution. However, having been the Outcomes Administrator for my institution for the past 3+ years, I believe Outcomes is a distinguishing feature within the LMS landscape. What other LMS ecosystem can enable an office of one assessment specialist within a single year to collect 27,000+ student documents, to sample 1,200 of those documents, to facilitate evaluation sessions including 175+ faculty evaluators, and to aggregate and report the results that lead to the formulation of continuous improvement recommendations regarding student attainment of three general education outcomes? And, that doesn’t begin to address program level assessment.

In my opinion, Blackboard should focus its effort on sharpening an already incredible tool that sets the Blackboard solution apart from others and on taking full advantage of that tool within the marketplace.

How?

The model to which I linked above focuses on the OUTPUTS of higher education, the effect of higher education on the community at-large: skilled graduates prepared successfully to enter the workforce or to transfer (and progress toward those). Those outputs are what matters most to the tax-paying public and to state legislators.  Those are indicators that suggest successful teaching and learning is occurring in higher education institutions. Focusing on those data provide a means by which legislators and the public may measure higher education institutions, but it does so without intervening into the classroom.

The model does NOT focus on are what were often labeled as “authentic OUTCOMES” or learning outcomes or specific definitions or methods of learning, and higher education professionals should STOP demanding that performance-based funding measure those outcomes because those types of measures would require legislators and the public interest to intervene into the classroom.  Measuring “authentic outcomes” for performance-based funding would require legislators to to define those outcomes and to establish specific methods for assessment.  In my opinion, that most likely leads to state-wide implementation of the state legislature’s version of something like the DQP accompanied by standardized exams to measure student attainment of outcomes.

Currently, the outcomes and means by which they are assessed are at the discretion of each institution and its faculty; the institutions are held accountable by regional and other accreditors regarding integrity of those processes and the continuous improvement of student attainment of OUTCOMES and by state and federal legislation and programs regarding the effect of those processes (OUTPUTS).  It should remain that way.

Second, deans for our technical programs have been collaborating to review assessment methods for end-of-program external learning experiences including student and supervisor surveys.  We have discussed more systematic and college-wide assessment of work readiness skills along with the assessment of program-specific outcomes and skills.

Third, we are working to improve the delivery of indirect assessment including a battery of surveys: graduate (at time of graduation), employer (of regional employers that employ our graduates), employment (of alumni 6-12 months after graduation).  We have initiated early conversations with a newly developing alumni association to collaborate regarding the process of maintaining contact with alumni.  I believe the annual process of updating alumni information could ideally include employment and career information; that would provide valuable longitudinal data regarding our students’ success-after-graduation.

Higher education inevitably will have to figure out competency-based education; it’s the logical conclusion to the national accountability discourse.  Several examples from this article that illustrate perhaps that we’re further away than we might hope:

” . . . raised questions about the faculty role in direct assessment.”  Teaching and learning IS assessment.  If we have questions about the faculty role in direct assessment, that’s a clear indication we have issues in our understanding of teaching and learning being something other than simple “seat time.”

” . . . whether students might receive federal aid for “life experience.”  Credit for prior learning carries significant negative baggage because many paper mills used credit for “life experience” fraudulently.  When competency based education is done well though, “life experience” and prior learning is seamlessly integrated.  Hypothetically, a 45-year old student enrolling in a competency based program will complete all of the same work as a traditional 18-year old student, but because the 45-year old has 27 years of informal learning experiences, the 45-year old may demonstrate competency in a shorter period of time – though in the exact same manner as the 18-year old.  The learning and assessment does not change; credit is not granted to the 45-year old using different mechanisms (a credit by exam) than those by which the 18-year old is (48 contact hours in a course).  In true competency-based education, the notion of “credit for life experience” becomes an inherent but consistently measured aspect of the credit-granting process.

“. . . should do more to ensure that direct-assessment programs are properly tied to credit-hour ‘equivalencies’ . . .”  We are constraining what should be by what already is; competency-based education is and will be extremely disruptive.  Doing competency based education well will, I think, require an overhaul of the financial aid system.  As long as we continue to try to map competency-based assessment to the credit-hour construct, the competency-based system will have limited effectiveness.

“. . . application did not adequately explain whether its coaches, who help guide students through their studies, were actually faculty members or had subject matter experience.”  Is the manner in which we credential faculty a by-product of the credit-hour construct?  Do we credential faculty the way we do to insure that students spend their required seat-time (contact hours) with an expert in the field?  Do a “coach’s” (or facilitator’s or faculty’s) credentials become secondary to their ability to support student attainment of the outcomes?  IF the assessment is valid and reliable, is that not a check against the credentials of the individual mentoring a student completing the assessment?

First, I described previously the robust work done by the AAC&U to create the LEAP Value Rubrics; revising that work is not likely to add significant value to the institutional assessment effort. The work began with, essentially, a rather comprehensive meta-analysis of local campus rubrics related to the respective outcomes. The research process clearly established the validity of the rubrics’ measurement of the constructs, and reliability testing was conducted. Taking the time and effort to modify the rubrics likely will not break new ground; it is quite likely that the consideration given to the research and development process originally has addressed issues or suggestions that local faculty may consider.  In short, and referencing my previous comments, revising the rubrics locally would be the just as much the epitome of “re-creating the wheel” as developing new rubrics would be.  Certainly the research effort would likely be insightful, but the end product likely would not be markedly different or necessarily an improvement; any minor gains will be offset by the time invested.  Exacerbating that issue is that beginning the revision process may open a Pandora’s Box of sorts; many of the constructs may be interpreted a variety of ways; revising the rubrics could be an endless process for an institution.  Locally, institutions may better invest time in developing and facilitating a robust process for the application of the rubrics and the continuous improvement effort that follows the results.

Second, expending institutional time and effort to modify the LEAP Value rubrics may nullify the benefits of adopting the rubrics in the first place.  Initially, modifying the criterion, dimensions, or descriptors of one of the LEAP Value rubrics potentially compromises the integrity of the rubric in regards to its validity and reliability.  All of the prior research on the rubrics conducted by the AAC&U projects is dependent upon and assumes the rubrics as they are written.  Derivatives of the rubric benefit from that research and, to some extent, leverage the validity and reliability.  However, substantive changes require additional effort to re-establish the validity and reliability of the new rubric, and even semantic changes to the LEAP Value rubrics can compromise the validity and reliability.  Further, modifying the rubrics would, in the same manner, nullify the broad acceptance that the LEAP Value rubrics have achieved; that would not be an advantage revised rubrics would maintain.

Third, and perhaps most importantly, modifying a LEAP Value Rubric likely precludes any future potential to compare and to benchmark local results to comparable institutions at the regional, state, and national levels.  The rubrics, as originally designed and developed, offer higher education institutions an alternative to standardized exams as an answer to legislative and accreditation calls for greater transparency and accountability.  However, that alternative is largely dependent on the comparability of results across institutions.  There is value and additional insights to be gained by community colleges comparing results to comparable institutions and to universities; student tracking becomes more valuable with the comparability of results on the same set of rubrics implemented at multiple institutions.

First, the AAC&U LEAP Value initiative has already done the work and research our faculty have interest in doing and has done it at a much broader scale than any single institution can manage.  Typically, the suggestion by faculty to explore other rubrics stems from an inherent curiosity and interest in researching potential solutions.  The AAC&U approached the development of the LEAP Value rubrics from that perspective.  Teams of “faculty, academic and student affairs professionals, and other experts from public and private, two-year and four-year higher education institutions across the United States . . . gathered, analyzed, and synthesized institutional-level rubrics (and related materials) . . . related to the LEAP Essential Learning Outcomes.”  In short, the AAC&U project conducted a very broad study regarding the methods/rubrics by which each construct could be measured. Our institution could certainly engage that same process, but the results would likely be very similar, and we would then need to go through the extensive process to further validate the rubrics and test their reliability.  That work has already been done as well.  Developing our own rubrics would be the epitome of “re-creating the wheel.”

Second, the LEAP Rubrics have strong validity and have been thoroughly tested for reliability.  Because the rubrics were created by faculty, nation-wide across a broad range of institutions, the LEAP Value rubrics have a high degree of face validity.  Further, among the faculty and higher education professionals on the LEAP Value rubric development teams were experts in the different outcome areas; that contributes greatly to the content validity of the rubrics; there’s a high degree of certainty that the rubrics measure the intended construct.  The rubrics also align with expectations for the skills that college graduates should exhibit and provide a more robust method for measuring student learning across the range of outcomes than other available methods (e.g. standardized exams).  In addition to the validity, the rubrics have also been tested and validated by faculty at more than 100 institutions, and the AAC&U conducted a national reliability test.  In the first reliability study, without a calibration/norming session, faculty scorers were found to have perfect agreement on scores nearly one-third of the time (or 32 percent, on average); using other methods of measuring reliability and after further calibration, the inter-rater reliability was measured at more than 60%.  Finally, reliability analyses conducted by various institutions during their implementations and use of the rubrics have consistently indicated high levels of inter-rater reliability.

Third, the LEAP Value rubrics have achieved a very broad acceptance that should not be discounted or overlooked; locally developed rubrics are not able to leverage that acceptance. Since Fall 2010, the rubrics have been downloaded and used at more than 4,000 different, world-wide institutions.  They have been adopted by entire university systems, multi-state consortia, and national higher education initiatives as a recommended method for measuring student learning (e.g. the Lumina Foundation’s Degree Qualifications Profile (DQP) includes the LEAP Value Rubric as a recommended strand).  Perhaps most importantly, the VALUE rubrics have been “embraced by all the regional accrediting bodies as one acceptable approach for institutions to use in assessing student learning . . . [those] accrediting organizations are recommending that most of the core elements and performance descriptors of the VALUE rubrics be retained for use in the assessment of student learning.”

Fourth, with the broad acceptance, the future will likely hold opportunities for comparability and benchmarking at state and national levels.  The “AAC&U is working to develop a national repository of VALUE rubric assessment results that will eventually permit national benchmarking similar to that used for the National Survey of Student Engagement and the Community College Survey of Student Engagement.”  Within Texas, a voluntary consortia of higher education institutions has formed to foster and to support use of the LEAP Value Rubrics across the state: LEAP Texas.  As of January 2014, 62 higher education institutions in Texas had officially joined the initiative.  One potential outgrowth of that consortia in the future may be comparison and benchmarking of results across the state and by comparable institutions.

Certainly, developing rubrics locally within an institution may have a variety of benefits, but using the LEAP Value rubrics allows an institution to leverage all of the work already completed by and potential benefits of the AAC&U’s LEAP Value Rubrics.

Related to our general education outcomes assessment project, I’ve been extremely busy with “rubric norming sessions” the last several weeks. Working with volunteer faculty evaluators, we have been going through a process to help foster consistency in how we, as an institution and as a faculty, apply different rubrics as we assess general education outcomes. I’ll describe that process, and I’m interested in hearing from others perhaps engaging a similar process and/or doing it differently.

We are reporting out this year on three outcomes: critical thinking, empirical and quantitative reasoning, and teamwork. As an institution, we decided to leverage the AAC&U’s LEAP Value Rubrics, and our faculty assessment committee had previously identified four different rubrics related to three outcomes.

* LEAP Critical Thinking to assess Critical Thinking
* LEAP Teamwork to assess Teamwork
* LEAP Quantitative Literacy to assess Critical Thinking and Empirical & Quantitative Reasoning within Math
* LEAP Problem Solving to assess CT and EQR within Sciences

I’ve been conducting different norming sessions, each focused on one of the rubrics. Our volunteer evaluators for the general education outcomes assessment project are asked to attend a norming session for each outcome for which they plan to assess student documents submitted in response to select course assignments.

The norming sessions go something like this…

We discuss the background of general education re/development that’s occurred over the past several years including efforts at AAC&U, THECB, and locally within our institution. That provides context for the new core curriculum that goes into effect this Fall 2014, as it does for all public institutions of higher education in Texas per THECB regulations. All of that is the precursor to improved and more systematic methods and approaches for assessing general education outcomes. We then turn and focus on the specific rubric for the session at hand.

We review the elements of the rubric and discuss what it does assess and what it does not assess. What the rubric assesses is described by the rubric itself; what it does not assess are those confounding variables we may tend to insert into our assessment.

For example, when assessing critical thinking, that rubric does NOT assess writing and composition skills. Certainly, there may be a correlation between the two, but they can be quite independent of one another. A student can write in fragments and run-ons and still be capable of demonstrating critical thinking skills. Also, a student may not be terribly effective at organizing their response in a traditional manner follow the common conventions of composition; they may do a better job in the conclusion than in the introduction of providing their thesis statement and introducing their position.

For example, the rubrics are not assessing from a grading perspective. When assessing student work for the purposes of grading in a course, the student response would be assessed against the requirements for the assignment. Students can respond quite well to an assignment but still only performing at a lower level in regards to critical thinking – particularly if the assignment is not aligned very well to the outcome. In those instances, it is important to determine which level of performance is exhibited by students and to not adjust upward on the rubric based on the student’s effectiveness in responding to the assignment.

That part of the discussion then concludes with a conversation about the institutional knowledge base that exists around the rubric – through prior years and through other, current year norming sessions. How do we apply it in specific instances or certain circumstances? Where there’s been confusion or differences between raters regarding the interpretation of the rubric criteria, how have we decided as an institution to apply the rubric?

Evaluators then apply the rubric to 4-5 samples of student work. We collect the results across all evaluators in the norming session (10-15 faculty) and begin analyzing and considering those results.

We look at the agreement between raters regarding success and not success. Student’s achieving a Level 2 or better on all criteria of the rubric are labeled as having successfully demonstrated that skill. Students with a 0 or 1 on any single criteria are identified as not successful regarding the outcome. Typically, our evaluators typically achieve an inter-rater agreement of 75-90% when considering success and not success – with some variance from session to session and student sample to student sample.

We focus the conversations for the norming sessions though on any particular noticeable differences between raters when it comes to the specific criteria. For example, if on “Evidence” criteria of the critical thinking rubric, if there are five raters that assigned a 2, 3 or 4 on the criteria, and 3 raters that rated a 0 or 1, we dive deeper into that criteria to ferret out why the raters varied to the extent they did. As we do that, we begin to identify subtle differences in how the rubric is interpreted and applied, and in doing so, we can establish how we want, as an institution, to interpret and apply it. That information then feeds back into and becomes part of the institutional knowledge base regarding the rubric.

After further conversation around those “hot spots,” evaluators then apply the rubric to another 3-4 samples of student work, and we repeat the process. Ideally, we see an increase in the agreement within each criteria.

These norming sessions prepare us to conduct evaluation sessions that are entirely online and asynchronous, rather than conducting synchronous, all-day “scoring sessions” that present a number of challenges when trying to “go to scale” and implementing juried assessments in a multi campus institution, with 30,000 students, and 800 faculty.

Current, common methods used by #highered #assessment professionals when sampling and evaluating student work for general education or program level outcomes assessment projects may not provide results that can be reliably generalized. Even if sampling work from 1500 students in an institution with only 4500 students total, the results may not be indicative of the institution’s true level of performance in teaching general education or program outcomes.

The research reported by Hathcoat and Penn in Volume 7, Winter 2012 of Research and Practice in Assessment analyzed results of an assessment project to consider potential sources of variance in the outcomes for students that had more than one artifact evaluated.

. . . it is of particular interest that 77% of the error variance derives from differences within a single person across each task.

When two different artifacts by the same student yielded different results, very little of the variation could be attributed to inter-rater reliability (12%). The differences were more a result of the variation in the nature of the tasks themselves or of the variation in student performance across multiple tasks.

This study suggests that if researchers want to make comparisons about students’ performance from authentic assessments between institutions or within an institution, they should greatly increase the number of tasks that are sampled for each student, establish statistical controls based on variables that are shown to impact students’ performance (such as motivation), or take steps to standardize some task characteristics (which may not be palatable for users of authentic assessment).

Certainly, there are limitations of the study’s results which are, of course, noted in the article. However, the implications of even the limited results are significant for institutional assessment and research professionals. Outcome centered assessment projects that randomly sample student work aligned to the outcome likely often have a limited number of samples – perhaps only one – from individual students. However, the results for a single student on a single task may not be validly generalized; thus, drawing conclusions regarding institutional effectiveness and accountability based on current, common sampling methods may face serious challenges.

Student centered outcomes assessment projects that sample multiple tasks from individual students (graduates, ostensibly) and generalize conclusions regarding institutional effectiveness based on a representative sample of the student population may be more valid and reliable. (e.g. sampling multiple artifacts from 30% of current year graduates).

“But my institution relies more heavily on course-embedded assessment, so it’s not as relevant to my institution.”  Not so fast…  I believe the results of that study also have implications for institutional administration of outcomes assessment projects relying on course-embedded assessments.

My institution relies almost exclusively on course-embedded assessment (of the first type I noted in a previous post regarding how “course-embedded assessment” may be interpreted in different ways).  We only use indirect methods to supplement our assessment efforts, and we do not use standardized exams in any manner to assess general education outcomes.  However, the study has serious implications for our administration of our outcomes assessment projects relying on course-embedded assessments.

This past fall, we identified five courses that were most frequently taken by students during their last semester before graduation.  Faculty teaching those courses collaborated to develop an assignment for the respective course that was aligned to our critical thinking general education outcome; the departmental assignments require students to think critically within the discipline.  The institutionally defined procedure was that each faculty member teaching the course would include the departmentally defined assignment as a regular assignment in their course: it was to be “course-embedded.”  Considering the results of the study, the “course-embedded” assignments would not encounter the student motivation issues that, according to the results of the study, occur when students take exams that do not effect their personal achievement within courses.

But for the students taking the exams, the tests tend to be low stakes — no one must pass or achieve a certain score to graduate, gain honors or to do pretty much anything.

Most of the topics I’ve planned to include have been in place for some time, but the organization of it has been evolving quite a bit.  At the moment, the presentation will be a series of issues and challenges I’ll pose to faculty to improve assessment in courses.  Of course, there will be interaction expected; there’s a question and pause every fourth slide or so…  It’s more about them than it is me.

Current assessment issues to be included are below along with questions I may ask along with a short summary or a link to a previous blog post where I’ve discussed the issue.  I’m always open to discussion and comments.