Two Compatible Paths — Take Both

There is little good news in current and projected college enrollments. Setting community colleges aside as a special case, independent colleges and universities are the hardest hit and will continue to be so for several years. While a few atypical independents are growing, most are struggling to meet enrollment targets. Some come close but fall victim to the silent killer: loss of total market share. Others are making budgetary adjustments to budget for a 5-10% decline. (To those adjusting budgets down: please do not cut your marketing, lead generation, and enrollment budgets out of a misplaced notion of equity.) Going forward, a key point often missed is that a large portion of enrollment growth — all of it for some institutions — can only come from market share taken from competitors.

The question I hear most from our clients is, “How do we succeed (grow) in this era of declining demographics and increasing competition from well-branded and cash-rich schools?”

With apologies to the complexities we deal with daily, the short answer is:

• Become much better at the complex process of attracting and enrolling uncommitted students

• Offer fresh programs that are responsive to the needs of your market

Specifically, growing in this environment will require new programs aligned with emerging needs and improved marketing and enrollment processes to attract and enroll students. Instruction based on modern learning sciences and precision managed retention play important roles as well but they account for less total success than enrollment (do it better and before your competition takes the prospective student off the market) and programming (offer the programs for which unmet need is highest). Expect to see outsized growth over the next few years in the market share of universities that enroll well, offer the right programs in an engaging structure that aligns with the learning sciences, and support students through structured retention services.

Put Enrollment Before Programs

Most of us would agree that it is bad practice to place a great product in the hands of a bad sales team. But the message gets lost when we change “sales” to “enrollment.” While most institutions have a decent if improvable catalog of programs, only 10-15% can make that claim about their enrollment function. I have more than two decades of hard data showing that the typical enrollment counselor turns away or turns off uncommitted prospective students by ignoring them, by being materially unresponsive to their needs, by seeking to enroll them before seeking to understand their needs and constraints, and so on through a dozen well documented points.

To a senior administrator’s eye, guided by the wrong metrics, enrollment teams may appear to be doing a good job because they have become proficient in enrolling prospective students who made up their mind to attend the university before inquiring. If these students (called “bluebirds” because they fly in the window under their own power) were ever enough to meet your enrollment targets, they will not be sufficient in the future.

Enrolling bluebirds requires low-level order taking skills. Being cordial is generally sufficient. Committed prospective students will do the rest. Enrollment counselors who bring these skills to uncommitted prospective students fail to convert enough or sometimes any of them. Uncommitted prospective students are comparison shopping and looking for a reason to attend based on their unique goals and circumstances. They need to know that your institution understands these circumstance and has a plan that will address them.

A Few Things You Can Do

Check your metrics. What percent of uncommitted shoppers should your enrollment team be able to enroll? The answer depends on the program, the program level, the competition, tuition level in relation to de facto competitors, time-to-degree, and other factors but a simple rule of thumb might be that your team should enroll nearly half of the uncommitted shoppers who contact your university via your website, email, or telephone (organic leads, 40-50% minimum conversion). If your metrics do not support the distinction between bluebirds and uncommitted shoppers, change them today. Separate the bluebirds and focus your assessment, coaching, training, and collateral on how well you are doing with uncommitted prospective students.

Get an audit. It is a good idea to check your equipment before setting out on a journey. This is why an audit of lead generation, website, and enrollment systems and processes is one of the most valuable services we provide. As you might expect, I have good reason to believe our audits are the best but speed is important here. Find someone to provide an expert and impartial assessment of your enrollment function. An audit should examine the structural and functional integration between branding, lead generation, enrollment, collateral, and pre-matriculation services. It should examine policy and business rules, and the relationships between role definitions, performance standards, and actual daily behavior. It should pay special attention to the metrics by which enrollment performance is evaluated and guided. Virtually every independent college we have audited over more than two decades, was found to be managing with metrics that were counterproductive to the institution’s goals. Things get better quickly when you develop, gather, and act on the right metrics. Above all, a good audit is as enjoyable as it is informative; it places detailed actionable recommendations in the context of best practice as adapted to the institution’s goals and its place in the markets it serves.

Our institutional growth audits often lead to recommending that the university temporarily defer launching new programs until the enrollment team can consistently convert uncommitted prospective students.

Bottom line: Our findings, consistent across 26 years of research, show that enrollment is most often the place to initiate a growth plan.

Enter the New Programs

If good enrollment practices are the first and best way to increase enrollments, launching the right new programs is the second most impactful path. The new program path is more risky, however, because of the time and cost required to fill the first seat and the possibility that not enough people will show up to make the program viable. These risks elevate the importance of making the right program decision the first time.

A recent study found that 48% of university programs generate fewer than 10 completions per year, meaning that 48% or more of the programs are probably operating in the red. Losing money on a program does not necessarily represent a failure if the loss aligns with a strategic decision and the program is managed to achieve whatever negative margin is deemed acceptable. Unfortunately, most of these programs are not intentional or mission-related losses. The majority of losses come from a fundamental misunderstanding of the nature of risk mitigation when determining what new programs are needed, by whom, in what numbers, in what format, and whether the institution is or would be viewed as a desirable provider.

The most common mistake made in mitigating new program risk is selecting new programs based on gross demand rather than net demand understood in the context of institutional value proposition.

Gross demand reports are widely available from the large marketing companies. Objective data and market logic show that gross demand may or may not align with program success. Moreover, the correlation coefficient between gross demand and program success, never high, is well below chance level in competitive markets. Flipping a coin might be an equivalent alternative.

A more longstanding mistake is selecting new programs based on demographic and psychographic attributes of theoretically prospective students. There is some merit in understanding target market demographics but that kind of information is best understood as a modulating variable in estimating enrollments after program feasibility and success have been determined by much more precise research methodologies. A few other sources of error in selecting for program success include launching programs deemed needed by faculty or those suggested by currently enrolled students.

How To (and how not to) Mitigate Risk in Selecting New Programs

Unless you are fortunate enough to have no competitors combined with large unmet demand for a new program, you need research-driven answers to 5-10 questions before making the decision to launch a particular program or move on to another possibility.

Together, the answers to these few questions create what you need to know to mitigate risk, and what you need to know is this:

Is there reliable evidence of sufficient and growing unmet need for a program for which my institution possesses enough brand equity to enroll a desired number of students in my target market?

Not all information contributes to risk mitigation and some kinds of seemingly useful information can actually increase it. Information that can increase the risk in a program launch decision includes data from job boards and the (uncorrected) BLS and enrollment propensity profiles based on demographic attributes and classifications. Even though reports based on these kinds of information appear to contribute to better decisions, the information they contain has low and sometimes negative predictive validity. At most, market research provided by “big box” companies offering boilerplate software-generated reports might suggest areas to consider for development but they do not address the issues central to growing profitably in a highly competitive environment.

Real risk mitigation requires the kind of intelligence that comes from hands-on work in your market to determine net demand and net growth assessed in relation to other factors including, for example, the value proposition and current market share held by competing programs and the ability and desire of these programs to scale to meet additional demand. Early in the process, valid risk mitigation assesses gross demand in the context of finely gradated (qualified) supply metrics and other factors that determine net demand and projections for net demand growth. These “net” metrics are then assessed against other critical success factors. The results of these research, analytic, and interpretative processes is a determination of the probability and conditions of a successful launch and a final launch recommendation, all with considerable detail attached.

The research process by which we get to a final recommendation is determined by methodologies we have refined over 26 years and 3,000 individual research studies, reports, and opportunities to follow through to assess impact. First, we examine our own longitudinal knowledge base. As one example, our detailed research interview notes from more than 40,000 employers, university program managers (competing programs), program accreditors, regulators, etc. are at our fingertips. Given the comparatively small number of academic program studies, we have thousands of personal interview notes in our knowledge base for most programs. In this background preparation phase, we also search external databases for emerging issues. The next phase involves having telephone conversations with university program leaders that will be your competitors. Are they growing? Are they at capacity? Can they expand to meet new demand should it appear? Can they fill their externship placement requirements? We have similar conversations with employers who may want to hire your graduates and may be interested in helping your program succeed. You get the idea by now. I’ll stop here and note that this is real research executed by senior academic researchers with more than two decades of experience.

Questions To Be Answered

The methodology sketched out above provides answers that will eliminate program launch risk to a negligible level. Consider the questions typically answered:

• If you offer this program, which existing programs will be your de facto competitors?

• If blended, is the need located within the area from which prospective students are likely to attend your new program? Is your method of blending optimal for your audience and context; if not, what is?

• If online, what kind of branding and lead generation budget will be required for your program to succeed against your de facto competitors in a defined market attraction area?

• Would you be better situated to launch online, blended, or both?

• If demand increases, can your prospective competitors easily scale to meet need and are they committed to doing so?

• Is uncommitted future growth of the program such that you can secure sufficient enrollments from the expanded market? If not, what proportion of your growth would depend on taking market share away from competitors and what would that entail?

• Does your institution possess sufficient brand equity to offer this type of program in this market? If so, how will your brand stand up against the competition?

• Do employers in your market perceive an unmet need for more graduates of this type of program? If so, what are the details of that need?

• Are employers interested in collaborating with you in ways that might be mutually beneficial? If so, what is their situation and contact information?

Our experience demonstrates that mitigating all of the knowable risk in advance of a program launch decision is possible but only if you ask the right questions with a valid research methodology and if you develop an adaptive strategy based on the answers.

Comments? Questions? We would love to hear from you.

Conventional wisdom suggests that we pay attention to, perhaps even track and manage, what we care about. If this is true at universities, there is little evidence that we care about the unique and potentially transformative goals students have in mind when they decide to assume six figure debt and substantial opportunity cost while committing years of discretionary time to pursue a college degree. Goals are seldom learned during the enrollment process, they do not become a part of institutional memory, they are not tracked for progress, and there are no institutional structures or practices to guide and coach toward their fulfillment. This form of neglect may not be so benign as some believe; the evidence is strong that the incremental achievement and migration of student goals is a powerful process and outcome metric. They are arguably the most important impacts of an education.

The Importance of Goals

Most students, even the shrinking class of traditional students who attend at the urging of parents, have specific goals that are logically and usually also empirically linked to the reason they chose to enroll at a college. Sometimes represented by a vision or meme, these goals are usually complex and multi-faceted, involving their sense of self, their family (and future family), their employer (or future employer) and more. These goals capture or reflect the vision of who these people are and want to become. They are important not only to the students who own them but to those who fulfill the mission to the institution of higher education. Among other things, the strength and clarity of students' goals, along with perceived progress toward meeting them, are instrumental in student persistence and success.

Students' goals are not static. Over time and because of their educational experiences, they evolve in ways that make them richer and more adequate. The enrichment of students' goals reflects an enrichment of their sense of self, their world view, and their place in that view. Indeed, higher education's most important contributions can be viewed as helping its students create and fulfill better refined goals.  

Individual vs. Institutional Caring

Most of us can think of a time when someone said they cared about us but failed to act on that caring when the opportunity to express that caring arose. In human parlance, a person's claim to caring is validated by appraising the relations between symbolic expressions and behaviors that validate those expressions.

When organizations claim that they care, we look for different but analogous forms of evidence. When a retail service company claims that customer service is its first priority, the validity of that claim is assessed by examining not how many times they make the claim but by examining organizational structure, roles, budgets, employee head counts, phone and email directories, data gathering and actions, and so on. Whatever its public statements, if a business does not spend money in manifesting caring behavior, it does not really care.

Colleges and universities are no exception. I cannot recall a meal with a college president who did not, at some time in the conversation, profess the institution's deep care about the success of its students. On every occasion that I can recall, the conversation stalled when I asked for specifics on how that care was manifested and the size of the budget that supported it. Usually, there was a bit of mumbling about how caring professors are. True enough — often — but a change of topic.

How do we validate a claim to institutional caring if we take it to mean that the institution cares whether or not students meet their goals as a consequence of the enormous commitment of time, energy, and money they have invested in the institution?

Follow the Money

For most colleges and universities, a close look at organizational structure, job titles, employee head counts, phone and email directories, and budgets, provides little evidence of caring. To find evidence of caring we might investigate the kinds of data systematically gathered and acted upon.

Very few institutions even ask students about their goals. If they do ask, it is on an admissions essay that is filed away with other admissions papers. Admissions counselors do not spend time helping students identify and clarify their goals, assisting in making them measurable, or breaking them into incrementally achievable steps that can be tracked and managed for progress. The institution does not track, manage, or report on students' progress in achieving their goals. Functionally, the nation's colleges and universities do not measure, manage, or attempt to optimize the achievement of students' goals. In fact, nothing in the institution's structure suggests that students have goals or that the institution has any role or responsibility to ensure that they are met. 

Functionally, the nation's colleges and universities do not measure, manage, or attempt to optimize the achievement of students' goals. In fact, nothing in the institution's structure suggests that students have goals or that the institution has any role or responsibility in ensuring that they are met.

Indifference of Regulators

With a few exceptions, it turns out that the overseers and regulators don't care much either. While they lay claims to caring in much the same way as college presidents, these organizations lack the institutional architecture that would validate a claim to caring. In fact, the regulator class explicitly denies the validity of student goals and the degree to which they are achieved as part of a pre/post framework for defining student outcomes and institutional effectiveness. I believe this is a serious mistake.

• Institutional accreditors dismiss student goal attainment as materially insignificant to the mission and purposes of the institutions they oversee. Soft, unfocused, secondary, and personal are adjectives I hear expressed by leaders at HLC, SACS, and WASC when I raise this topic. There are exceptions. Community college accreditors have added structures to measure and manage goals associated to migration to other institutions of higher education. This is a start.

• The Department of Education doesn't care about students' goals either. They are focused, some would say excessively, on federally insured loans and campus social issues, both of which connect to student goals instrumentally but not directly. One exception is the Department's separate standards constructed to constrain the growth of for-profit colleges and universities. The Department has imputed to these institutions a single goal of having its students secure gainful employment. This might be a good idea except that non-profit institutions are permitted to ignore this goal and the goal itself should accommodate the complexity and multiplicity of goals and timelines for professional development. A case can be made for increased accountability, but a rational case cannot substitute IRS tax status for students.

• Universities, colleges, and academic departments demonstrate a lack of caring when they fail to determine students' goals, refine them if necessary, break them into measurable milestones, and place these data in the student information system. They fail to assess progress towards these goals at points throughout the student lifecycle. Instead of assessing goal achievement upon graduation and placing a detailed report on gains in each student's hand, most colleges conduct occasional surveys that ask a random sample of graduates generic questions related to achieving their unspecified goals. Occasionally the results of these surveys are discussed until someone points out that a 16% response rate does little to inform rational action. At this point, the results of the institution's only attempt to learn about student goals, albeit after the fact, gathers dust on a virtual shelf.

What do higher education's stakeholders care about?

The short answer is that university stakeholders care about the things in which they, and not necessarily students, are interested. An institution's goals may be defined by a faculty senate, program goals are defined by deans, and so on. Yes, these goals are or should be formulated with the interests of students in mind, but the relationship is variable and tenuous, and some may not be commendatory. Anyone who has served on university committees knows all too well that some of the goals we impose on students can be trivial or even self-serving.

Why Should They Care?

Despite the largely tacit belief that students’ goals are peripheral to the business of higher education, a case can be made — a strong one in my view — that students' goals define the core of a relationship between themselves and the institutions that take their money. It is not difficult to argue that this relationship constitutes an implied contract the implications of which extend beyond the obligations accorded or even acknowledged by most colleges and universities. Student goals define central and important outcomes by which courses of study must be evaluated.

For the few college students not yet of voting age, it might be argued that parents "own" the student’s goals. While that view is debatable, I concede it for the purposes of this discussion, noting however:

• Most students today (around three-quarters) are informed consumers who have acquired adult social, civic, and financial responsibilities.

• Many of today’s students (around half) have acquired family responsibilities and have already initiated meaningful and career-centered work.

These students, especially working adult and professional students, enroll in a program for specific reasons. These reasons are tied to concrete personal and professional goals. Whether we would agree with the judgments of a particular student, their legal and social status as adults confers a sense of finality on these judgments and an equal sense in which they are not subject to our purview, certainly not to our rejection as material outcomes.

These students, especially working adult and professional students, enroll in your program for specific reasons . . . tied to concrete personal and professional goals. Whether we would agree with the judgments of a particular student, their legal and social status as adults confers a sense of finality on these judgments and an equal sense in which they are not subject to our purview, certainty not to our rejection as material outcomes.

Like it or not (and we academics tend not to like it), if a student wants a degree in accounting to become a more effective manager in specific ways she has clearly identified and, if that student has made it clear that she has no interest in becoming an accountant, our goal of having her prepare to sit for the CPA examination is largely irrelevant. Understanding the social contract we have with the students requires that we set that goal aside. When we fail to assist students achieve their goals, we forgo an important opportunity to meet their needs by improving the services we provide them. This failure has long-term consequences because these students appraise the quality of their experiences with your institution, including the extent to which your programs and support services were instrumental in attaining their goals.

Benefits of Measuring, Tracking, & Managing to Goals

Institutional

The institution benefits in many concrete and measurable ways by paying attention to students' goals. Many of these benefits are interdependent; most of them translate into financial benefits.

Working with students on their goals establishes a closer relationship which translates into improved retention to break/even, improved retention to graduation, increased forward-looking insight into needs for program development and modification, and overall increased workplace enjoyment. The ability to report progress toward goals at defined milestones (calendar and event driven) and do it in a way that is meaningful to students and other stakeholder's, especially instructors, advisors, and retention counselors, not only increases student engagement and all of its benefits, but it also provides other stakeholders, especially faculty and program developers, insights not available elsewhere.

Shared

Working with students on their goals enables mid-course corrections for the benefit of students and the institution. It becomes possible to accommodate not only progress to date but the refinements that students make in their goals as they progress in their courses of study. These refinements validate one of higher education's most important outcomes, the migration of goals in dimensions of increasing scope, depth, consistency, value integrity, and utility. 

Is Change on the Horizon?

Institutional events that are not attended to, refined, measured, managed, or rewarded are systemically unimportant. It is no more than lip service for an institution to claim otherwise.

It is neither defensible nor beneficial to the institution to ignore students' goals by failing to integrate them into the educational structure and process. Moreover, it is questionable to do as we do today in assigning precedence to instructor goals over student goals.

While these issues may be visible only at the margins today, it is only a matter of time before we see the integration of student goal attainment into academic structure and process.

A related Executive Briefing will examine student and professorial goals and outline methods for integrating goal attainment with institutional structures and practices.

As always, I welcome your views. Please share your thinking below or send me an email.

Robert W Tucker
President, InterEd, Inc.

This series of mini-briefings is based on generalizations arising from the study of elective drops in non-residential (AKA: in adult-centered, non-traditional) higher education programs of study. Among the central generalizations are: (a) elective drops are most accurately modeled as critical events that can progress in a matter of days from initiation to execution; (b) trying to improve retention with predictive modeling, a method growing in popularity largely due to sales initiatives, is unlikely to produce results above chance levels in a well-controlled experiment; importantly, predictive modeling's tagging of potential drops cannot predict specific drops within the actual event horizon; (c) the most effective way to improve retention is to develop a genuine relationship with students such that they feel comfortable to, and will, reach out for help early in a rapidly developing drop event. 

Tips in this series describe something you can do to improve retention that does not require adopting a complete system with which to manage retention with precision. At the same time, each tip is congruent with the kind of precision managed system we know to be most effective.

Tip: Respect Your Counselors

The Retention Counselor role is at the center of the precision managed retention system. The Counselor creates and maintains relationships with students, and is the first line of response for any student problems or concerns. Creating and maintaining an effective retention-focused relationship requires strategic and tactical outbound communications and depends upon the Counselor's judgment regarding the state of the relationship.

Not a Call Center

The goal of InterEd's precision managed retention system is to establish relationships with students such that when they feel pressures to withdraw (academic, family, work, etc.) they reach out to discuss the matter with their Counselor. When this occurs, the Counselor must be able to assist the student and talk honestly and openly about choices and consequences. People who can perform this role effectively are empowered professionals. We are surprised when schools characterize this approach as a "call center" that they think they can staff with student workers or when they assert that their new automated email or texting system can address this need.

Qualifications & Longevity in Role

Retention Counselors grow into their role in six to 12 months and remain effective for several years thereafter. For this reason, a relationship-oriented retention program cannot become effective with the turnover rates generally associated with call center business rules and compensation. Compensation, advancement, and other incentives should be developed around a two to five year longevity projection.

The importance of longevity does not ignore the fact that retention counselors are typically young and/or near the start of their careers in higher education. Recognizing the needs for maturity, some institutions prefer to hire mid-career professionals with advanced degrees but attempt to compensate at the level of entry positions. This decision can increase turnover which, in turn, diminishes results. In addition, institutions mistakenly tend to prefer individuals with previous higher education experience despite evidence that experience in strong customer service organizations is a better predictor of success as a Retention Counselor. 

How to Show Respect

The most common mistake we see is treating Retention Counselors as non-professional staff. Even if you are restricted in your compensation options, other benefits, considerations, and signs of importance and respect can be implemented.

• Make them salaried employees.

• Provide them with offices of equal status compared to staff members at similar levels (Admissions Advisors, for example).

• Empower them through training and information access to address a variety of student problems. 

• Celebrate wins 51%; manage shortfall 49%.

As always, I welcome your views. Please share your thinking below or send me an email.

Keith Blakeman
Vice President of Analytical Services, InterEd, Inc.

This multi-part Executive Briefing focuses on issues of relevance to senior decision-makers. Part I looks at the typical objections to end-of-course assessments and suggests they can be addressed in ways that achieve benefits outweighing risks. Part II will outline key executive considerations in revising an end-of-course assessment system to achieve stated benefits.

As with other Executive Briefings where multiple issues of some complexity are involved, I will focus on assertions of potential value to you. For additional detail, send me an email or call.

The Long View

Since 1985, we have developed, validated, deployed, analyzed, and reported findings on more than 12 million end-of-course assessments administered in more than 600,000 courses. For many of these assessments, we have captured, taxonomized, and analyzed the open-ended responses of students and faculty to open-ended questions about instruction, curriculum, other learners, the learning environment, and support services. These data sources have been integrated to answer the common and the not-so-common questions about the merit of end-of-course assessments. As examples, these assessments have been analyzed for whatever secrets they can reveal about effective classrooms and instruction, grades, rate of learning, achievement, mastery, performance metrics, and even career progression.

This 30 year span of work has brought me face-to-face with more practical problems, more challenges to validity and usefulness, and more inferential dead-ends than I could have imagined in my early days, days I now see as naive in retrospect.

Why Have End-of-Course Course Assessments?

The easy answer is that regional, national, and professional accrediting bodies generally require end-of-course assessments. However, these oversight bodies lack the expertise and the political will to set the kind of standards that would require assessments to produce and use valid information. For this reason, and because faculties have mixed feelings about student assessments, end-of-course assessments often suffer from low institutional priority. In the typical setting, instruments are poorly designed (usually by faculty committee with no collective skill in measurement science or decision-support), comments are not gathered or are inappropriately treated, analyses are sophomoric, reports are not timely nor are they individually customized for stakeholders, and there are no requirements that the findings be used in an intelligent way.

If I were limited to offering one reason for implementing a sound end-of-course assessment system it would be that it provides the first and best line of decision-support for effective process management. Contrary to conventional wisdom, processes are the only points at which you can effect outcomes. Managing academic quality by focusing on outcomes is akin to pushing on a rope.

For most of the 20th century, higher education defined quality based on inputs. A university was said to be of high quality because their faculties had the right degrees and their laboratories and libraries had the right stuff. Driven largely by external forces, higher education gradually and in no small measure begrudgingly accepted the idea that outcomes should be given more attention in its definitions of quality. Unfortunately, this gradual shift to outcomes, including competencies, takes us not to the present but only halfway to the modern metrics of quality Were we in any other service industry, suitability to purpose would be our hallmark construct of quality and we would be managing to that suitability via the close management of processes, not outcomes. In a process managed environment, outcomes are measured but are confirmatory rather than surprise factors. Like a turn-by-turn GPS, end-of-course assessments provide critical information that can be used to ensure that students, instructors, curriculum, and support services remain harmoniously on course, working in unison to achieve the desired outcome.

Talk - All of It Bad

End-of-course assessments have received an unjustified bad rap. Among the claims we hear most are that these assessments:

• Are nothing more than a popularity contest

• Cater to and reward negativism

• Over represent isolated, atypical perspectives, not the mainstream

• Correlate highly with grades awarded (good evaluations are easily “purchased” by awarding good grades)

• Provide little or no valid information to guide instruction, curriculum, or university services

• Are not taken seriously by students or instructors

• Show little or no correlation with learning

• Are invalid because students don’t have the capacity to make informed judgments about their educational experience

The Truth

Are any of these claims true and, if so, to what extent? The answer rests on the specific assessment process in question, including not only its scientific and technical merit but its institutional context, including setting conditions and patterns of use. In practice, many end-of-course assessments are designed and implemented so poorly that they guarantee the truth of some of these criticisms. Poor instrument development can produce skewed findings. Poor administration procedures can reduce student confidence in the process. Poor reporting procedures can stifle the development of skills essential to interpretation and use. A lack of senior administrative support (read: use) can undermine the seriousness of efforts required to implement and manage a good process.

• Popularity contest? False. These assessments correlate highly with independent measures of teaching skill, including classroom management and feedback on student performance.

• Mostly negative? False. More than 75% of student comments are about the learning environment, 50% about faculty and of the faculty comments, 80% are positive.

• Atypical? False. In fact, findings are more likely to be atypical when they suffer from primacy and recency biases such as when they rest on isolated, one-off readings of single course assessments or recollections of a few isolated comments that do not reflect the mainstream judgments (excluded middle bias).

• Correlate with grades? False. The largest run we did was on a stratified random set of 85,000 grades and assessment indices. The strongest correlation was 0.26 which itself was better explained in other ways. This particular issue speaks to one of the great unscientific myths in higher education. (We do see a higher correlation with grades in instances of bad teaching.)

• Not a good guide? False. Well designed assessments provide detailed guidance on curriculum maintenance, areas in which instruction can be improved, and university services.

• Not taken seriously? False. This myth probably arises from instructors' observations that students spend less time on assessments as they progress through their course of study. When we inserted reliability scales in these assessments, we learned that students are simply getting better at the assessments and that this skill is reflected in reduced time-on-task metrics.

• No correlation with learning? Doubly false. The correlation is high but the explanations are complex. Let me know if you are interested in this dimension.

• Students' are not qualified to judge? False bordering on ridiculous. We hear this one a lot from an small segment of those who teach. Let the incoherence if not outright arrogance of these words speak for themselves.

The Bottom Line

Our experience has demonstrated that even a modest application of necessary expertise and conscientious administration will create significant benefit to students, faculty, content experts, university services, and to the planning function of the institution.

As always, I welcome your views. Please share your thinking below or send me an email.

Robert W Tucker
President, InterEd, Inc.

Framing R&D Questions

Legacy Issues

A recent article in Inside higher Education asked the question, How Much Should We Be Spending on Learning R&D? In that article, the author notes the R&D expenditures of leading companies and wonders if higher education should be doing the same thing and, if so, what that kind of R&D would look like. The article didn't get much of a response but it should have. It asks a fundamental question that illuminates one undesirable consequence of higher education's emergence from its legacy as a guild in which product management is controlled by the talent and energies (or the lack thereof) of those who deliver its products rather than by intelligent design of the enterprise.

Who Is in Charge of R&D?

We are all well acquainted with the received view that each member of the faculty directs her own R&D more or less without oversight. In my view, a stronger case can be made that the executive leadership of the institution is in charge of determining how much revenue should be allocated to R&D to support continuous improvement in the institution's basic function. Executive leadership is also in charge of ensuring that budgets, behaviors, and results reflect those allocation decisions.

R&D on What?

How much R&D is appropriate? To what focus? How much are we spending right now? 

Colleges and universities deliver a number of services and products that define their overall structure and place in the community. For most colleges and universities, the case is overwhelming that teaching is the institution's core function. Yet teaching is complex in that it entails learning and learning entails assessment of learning which may go so far as to entail the impact of learning which is linked to credentials (credits, certificates, degrees, licenses, etc.) which can also be viewed as products. Teaching can also be complex in that process and product are sometimes difficult to separate, although seldom difficult to distinguish. These kinds of complexity give rise to disagreements about the merits of various ways of looking at teaching, learning, outcomes, and impact as the institution's central mission.

These disagreements about the complexity of teaching, learning, and impact are sometimes exploited as a basis for doing nothing. Doing nothing in the way of R&D is a path most institutions have been following since 1906.

The magnitude of this mistake is revealed by asking a simple question. How much better would we be at teaching if every college and university had, for the last 110 years, allocated 2% of its teaching revenue to improving teaching?

While these legitimate disagreements may alter the focus of an R&D strategy and agenda; they do not alter the rationale for performing it.

Nature of R&D

When R&D focuses on teaching and learning, the learning and measurement sciences are central to most investigations. Robust experimental and quasi-experimental designs are possible and findings, when aggregated and disseminated, will improve one or more aspects of the instructional process.

Different methodological approaches may be justified when R&D focuses on the dissemination of knowledge, credentials, and other forms of impact. It is unlikely that we can or would always want to secure tight control over variables when we are assessing downstream effects and corollaries.

In all, teaching, learning, credentialing, and assessing outcomes and impact represent a large, interesting, and challenging field. A field, by the way, shared with cognitive and brain scientists, research methodology, and measurement statisticians. 

How Much Do Other Sectors Spend?

While each industry and institution within it must set its own R&D targets, it is possible to derive some guidelines from similar disciplines. Looking at PWC's chart on 2015 North American R&D expenditures by sector we see Software at 16%, Healthcare at 11%, Telecom at 8%, Aerospace at 3% and the lowest, Consumer, at 2%. Aggregate R&D spending is approximately 5% of revenue.

How Much Do We Spend?

If we limit our focus to R&D funded by the institution for the specific purpose of developing and evaluating more effective and more efficient ways to teach and ensure learning, most US colleges and universities spend nothing. 

The picture improves somewhat if we expand the notion of R&D to include externally funded research projects carried out by researchers in departments of education, psychology, cognition, brain sciences, and others. Should these funded research projects count as institutional R&D expenditures? They should if the findings are applied and tested more broadly throughout the institution. They should not count if no one outside of the department is aware of the research and there is no initiative to apply findings to improve teaching and learning. 

How to Think About Education R&D

Having sketched the affirmative case, let's look at the other side of the argument. One could argue that a particular college or university is a structural template analogous to a franchise. In this analogy, the R&D is conducted elsewhere (e.g., McDonald's corporate research facility) and the individual franchisee delivers the products or services structured by that R&D. By this analogy, schools simply deliver the teaching and evaluation practices extant in the profession and among those whom they hire to teach.

This would be a good argument if there were a robust research and development community whose responsibility it was to train college professors how to teach. Unfortunately, there is no such community and most college professors have no idea how to teach in ways that rest on the modern learning, measurement, and brain sciences. Most professors teach the way they were taught and so it goes, recursively, back to 1906, the period in which we developed the "read this chapter, listen to me lecture, ask questions, take this test" model for organizing classrooms.

Why don't we own the fact we choose not to spend a few percent of our teaching revenue on R&D? Why pretend that our classrooms are modern when they are organized around practices that predate the relevant sciences? My recommendation: find a way to fund ongoing research on how to improve quality - especially efficiency - in the classroom. It will pay to do so.

As always, I welcome your views. Please share your thinking below or send me an email.

Robert W Tucker
President, InterEd, Inc.

A recent article in the Chronicle reported on issues identified by college administrators and faculty as they contemplate the future of higher education. Two arguably important issues - whether teaching methods should be derived from modern learning (and, separately, the evaluation sciences) - were not mentioned by any professional participating in the survey. Both of these issues have direct impact on the effectiveness and efficiency with which institutions of higher education fulfill their primary mission – teaching (presumably well). Part I examined current teaching behavior, practices tracing to models stabilized in 1906. Part II examined changes in the current teaching environment that increase the importance of doing a better job of teaching and evaluating learning. This Briefing identifies a few of the most effective and easily implemented ways that the learning sciences can be applied to college classrooms, physical or virtual. Part IV will examine practical implications of applying modern evaluation sciences to how we assess learning.

Applying Learning Sciences

Incorporating the modern learning sciences into one's teaching behavior produces almost immediate benefits and is not particularly disruptive. Faculty and students report higher levels of engagement and enjoyment, more is learned in less time, and what is learned is more behaviorally useful. As I pointed out in previous Briefings, many of those who teach already incorporate the relevant scientific findings and generalizations into their teaching practices; some do so knowingly and systematically, others arrive at science-based practices through experimentation and being sensitive to the needs of students and what methods seem to meet them best.

If you are involved in or track progress in the relevant sciences, you know that the links between findings, generalizations, and classroom behavior are not always direct and that some of the implications for action are contested by experts. This Executive Briefing avoids contested topics in favor of identifying a few well-established science-based practices that students and teachers find rewarding to practice.

Five Generalizations Worth Considering

Right-size (chunk) Learning Objectives

The course, as we know it, came to be a vehicle for teaching a group of related topics over a period of 10 to 16 weeks for a variety of reasons having little to do with learning. Among other things, our then agrarian society played a role. 

Unfortunately, course length does not necessarily correspond to how we master and connect learning objectives, especially those at the higher and performative ends of the learning taxonomies. A better approach deconstructs courses into a several self-sufficient units of learning (typically 5 to 15 determined by content) in which each unit contains a performative learning objective and a corresponding set of activities and evaluation points, along with a variety of classifying properties. (It is these properties that guide the assembly of individual learning objects into an efficient path of study.) Under this approach, a course's learning objectives - which may vary from three to 100 - are re-crafted into from five to 15 learning objects (think: containers), each constructed as self-contained units of learning in which the objective is coupled with activities and assessments to accomplish horizontal and vertical learning.

In addition to increasing the amount and speed of learning, chunking what were formerly courses into learning objects confers a number of systems advantages in time-to-degree, scheduling logistics, financial margins, and prior learning assessment.

A question that comes up early when we help programs transform to this more efficient instructional model is, "What about Title IV?" There is no Title IV exposure under this approach. Individual learning objects are simply rolled up to produce a transcripted "course" under the 1906 model. Thinking about this reveals one of many operational and financial advantages of this approach. A given course can be constructed by way of many different combinations of learning objects, thereby making learning more adaptive and efficient both within and across learners.

Embed Horizontal Learning

Content developers and managers, whether faculty or learning and design specialists, seldom embed horizontal (student-to-student and learning team-to-learning team) learning. They should because horizontal learning increases student engagement and provides multiple sources of rationale for mastering the objective; it refines understanding and application upward by detecting and correcting individual errors and shortsighted views and by socializing superior views held by individuals; it lays the foundation for authentic demonstration and assessment of competence. Indirectly, it increases retention. We will write more on this in a future Executive Briefing.

Balance Across Outcomes

Learning, retention, generalization, and hierarchical development (sometimes called scaffolding) are most efficient when the objectives, activities, and assessments address and integrate the cognitive, affective, and performative dimensions of what is being learned. You can think of this as integrating the what, how, why, and how did I do dimensions of learning. The integration of these dimensions is often what we look for as a mark of expertise and is correlated with personal and professional success.

Ensure Authenticity

A simple working definition of “authentic” is that the objectives, activities, and evaluations mirror the structure of their non-classroom target application. If we are teaching people how to be managers, authentic evaluations would involve reviews of direct reports (by faculty) and peers (by other students) structured in ways that students will perform and have their performances evaluated in the workplace. Authentic evaluations for engineers would be different and would most likely focus on individual and team work to accomplish engineering projects. Along with other benefits, authenticity contributes to efficient generalizations and direct application beyond the classroom. For the most part, this means that learning activities and evaluations will involve other people and will often have a 360-degree component.

The first question we usually hear when talking about 360 degree components concerns the primacy of the instructor's evaluation. ("You can't have students evaluating learning!") Thinking a little about the 30 or so dimensions of form and content related to activities and their assessments shows that the answer lies in the specifics. If the activity and assessments are focused on fidelity to key learning constructs and competence, student evaluations are significantly less relevant and the instructor's evaluation is primary. If the activity and assessments are focused on the quality and efficiency of horizontal interaction, the instructor's evaluation is secondary. If the activity and evaluation dimensions are focused on the clarity and concision of a presentation, both student and instructor assessments are primary, albeit in different sub-dimensions.

Overlearn Selectively

The implications of the modern learning sciences do not reject all common teaching practices. Perhaps ironically, overlearning, an old and largely discarded method of teaching involving repetitive practice well beyond mastery, is supported by the findings of modern brain and learning science research. For certain forms of standardized knowledge (e.g., mathematical functions), overlearning produces benefits superior to other methods developed to date.

What Gets Better with Modern Teaching Methods?

What we know about how to improve learning will continue to evolve, modifying today’s guidance as it should. The ruling reason in this case is not whether we have definitive and uncontroversial guidance from the learning sciences. A better approach is to ask if we have learned anything since 1906 that can be applied to the classroom to produce better learning, including learning that is more efficient. 

Even though the above dimensions are generalized and their application will vary somewhat with the nature of the unit of learning, we have applied them in areas ranging from art history to accounting and from business law to literature.

Among the improvements we can expect when we apply teaching methods that exploit the relevant sciences are: 

• Reduced time to mastery

• More substantive levels of mastery

• Increased ability to generalize horizontally and vertically

• Increased student engagement and motivation

• Increased retention

• Improved ability to teach others

Any one of these potential benefits can make it worthwhile to change one's teaching practices.

As always, I welcome your views. Please share your thinking below or send me an email.

Robert W Tucker
President, InterEd, Inc.

A recent article in the Chronicle reported issues concerning college administrators and faculty as they contemplate the future of higher education. Two arguably important issues - whether teaching methods should be derived from modern learning and evaluation sciences - were not mentioned by any professional participating in the survey. Both of these issues connect with the effectiveness and efficiency in executing higher education’s primary mission – teaching  (presumably well). Part I examined the current situation. Part II examines changes that make it more important to apply what we know about learning to how we teach.

It Matters How We Teach & Assess

It would not be necessary to write most of the following were higher education less reflective of its legacy as a guild. Few of us would visit a physician who took guidance from 1906 science yet we think nothing unusual when a professor of learning science teaches, as most do, out of a 1906 playbook. This needs to change.

Students Have Changed

In 1906, classrooms consisted exclusively of the very smart and the über wealthy Today's classrooms are made up of students possessing abilities spanning 80% of the ability distribution. This diversity holds important implications for teaching effectiveness. Teaching skills were appreciated but not required in 1906. Intellectually gifted students had minimal needs for teachers and often benefited when teachers got out of their way. They still do. Sons (a few were daughters) of wealthy donors might have benefited from good teaching, but they attended and graduated under a different social contract.

Today, the majority of students stand to benefit substantially from good teaching; some of them require it to succeed. Good teachers not only facilitate learning, they teach students how to learn more efficiently, and how to retain and apply what they learn in more useful ways. The modern learning sciences have stimulated the development of better methods for securing engagement, retention, generalization, application, and ongoing learning. Modern evaluation sciences produce more accurate assessments of progress in each of these dimensions, While a few teachers develop skills reflecting modern learning sciences on their own, it is more difficult for them to improve their evaluation methods without mastering detail in the relevant methodologies and tools.

Cheaper, Faster . . . & Better

As the least progressive among our major institutions, our colleges and universities are glacially slow to change. The result is that our colleges and universities have become the least efficient among major social institutions. Inflation-adjusted cost per credit and cost per degree has outstripped the CPI for more than four decades. Some of this inefficiency owes to resources, especially time-on-task required to teach. The skillful application of learning and measurement sciences can reduce time-to-degree by as much as 50% while producing higher quality learning. These gains reduce net costs to an extent that offsets other systemic inefficiencies that have grown over the past 40 years. Degrees and associated outcomes could be obtained cheaper, faster, and better in classrooms powered by the modern learning and measurement sciences. Imagine the increases in access that could result from a 50% reduction in the cost of a baccalaureate degree.

New Demands on Graduates

Whether employed then or now, the 1906 playbook produces graduates that are heavy on facts and light on mastering the generalization, problem solving, and application that used to be deferred to on-the-job training. Outcomes that were heavier on facts and lighter on evidence of critical thinking were aligned with the 1906 workplace of degreed employees. Most new hires were afforded a leisurely ramp-up of expectations. After all, they might remain with the same employer for 30 or more years. Today’s workforce is mobile. New hires are expected to get down to real work shortly after being assigned a desk and meeting with HR. Modern teaching methods, especially those that focus on authentic learning and assessment, produce job ready graduates who think and act at the high end of taxonomies of learning. Instead of struggling to fit what they are seeing into facts they may recall from the classroom, they show up ready to analyze, synthesize, evaluate, and work with others to get things done. They do this because pedagogy driven by modern learning sciences has them doing this very thing throughout their academic experience.

Devaluation of Grades & Degrees

In addition to expecting 2016 graduates to show up ready to work, employers are expressing dissatisfaction with how unevenly prepared they find new employees, even though their transcripts suggest that they are more-or-less equally qualified. This failure goes largely to higher education’s dominant use of tests that assess at or near the bottom of taxonomies of learning and, even then, with low construct validity. Low construct validity produces report cards that bear a weak and unreliable relationship not only with what the student knows but also with how well the student can apply what they know to the real world. A few institutions are attempting to address this problem by providing competency-based transcripts alongside the grade transcripts. This is a positive initiative but only a couple of schools are doing it well at present and we can only hope that it will catch on with schools and employers. 

Is Change Coming?

Unfortunately and against my expectations over the past two decades, a true market has yet to emerge with respect to teaching methods. What are the constraints on the development of such a market?

First, teaching well involves assigning more of your time on task and may result in less time devoted to research, and to community and committee work. It is here that universities continue to embed disincentives to teaching well. While good teaching may earn you a Teacher of the Year award, bringing in research money or being a community personality is more likely to increase your income and job security. In addition to these structural disincentives, incentives are also found in the self-regulating aspects of the faculty community. The bottom line here is that there are no extrinsic rewards and some extrinsic punishments for teaching well.

Second, to be in a position to reward good teaching, you have to know what good teaching looks like. You also need to have a system in place to measure it. Neither condition in in place. Most university administrators, including deans, have no idea how or how well their faculties teach. Given the widespread lack of understanding of modern learning sciences, a reasonable guess is that most deans cannot reliably distinguish effective from ineffective teaching, especially if they rely upon student grades as the dependent variable. In some institutions, the faculty senate prohibits administrators from asking such questions. Other institutions allow some oversight but permit little or no control and what control there might be is vitiated when good teaching is confused with good social skills or when a demonstrably ignorant teacher brings research or development money into the department.

Next

In Part III, I will identify and discuss the contributions of the learning and measurement sciences in more detail, concluding with concrete steps your institution can take to hasten the day when good teaching is not only rewarded but seen as standard practice. 

As always, I welcome your views. Please share your thinking below or send me an email.

Robert W Tucker
President, InterEd, Inc.

Federal policymakers and accrediting bodies are devoting increased attention to one of the primary indicators of effective education: graduation rates. Because graduation is an outcome, options for managing it are limited and generally after the fact. The effective management of graduation rates requires managing the processes that contribute incrementally to retention. To think otherwise is to believe in the possibility of managing effects rather than causes.

A growing number of companies are offering tools to improve retention. Many of these companies offer technology solutions based on predictive analytics. While specific methods differ, the goal is to assign a crude coefficient representing each student's probability of persisting. This is not our approach. 

It's About the Relationship

The human processes that go into persistence and drops form a complicated mosaic. Big data analysis and predictive modeling have done little to illuminate these processes and, on balance, have distracted our attention with a black box that tags winners and losers in the game of higher education. As measurement scientists that have too much fun running regressions and neural networks, we have no objections to statistically driven black boxes. Our objection centers on the fact that such modeling ignores the dynamics of retention in favor of probabilistic labeling. While we think it is important to make statistical assessments of risk associated to student attributes and behaviors, and to use that information to shape interventions, the utility of this approach stops there. Other than by chance, no predictive system will get your staff in front of an at-risk student at the exact time that an intervention is required to retain the student. There is a way, however, and technology can help. Because only the student knows when his or her internal state of mind has begun to think seriously about dropping out, the institution must have established a relationship and a structure by which a student can reach out for assistance and feel comfortable doing so. Email exhortations and warm greetings by the president will not accomplish this. Students reach out and retention is an outcome only when students have an authentic relationship with a retention counselor. This presumes, of course, that you have developed genuinely helpful services to offer the student who reaches out.

The following process measures can help managers insure that retention staff members are doing the things required to establish and maintain authentic relationships with their students. These relationships will improve performance in the incremental retention metrics summarized below as well as in the final retention-to-graduation outcome measure.

Average Outbound Contact Frequency

Retention Counselor activity to support relationships with students should be measured as an average daily outbound contact attempt volume. These contact attempts can occur in any media (phone, email, text) and must be directed at students assigned to them. For each institution, technology, and program, there is a precise number.  

Counselor Name Recognition

This metric is the single most effective proxy measure for effective relationships. On a regular calendar, researchers (ideally independent) call a random sample of students and ask them if they know the name of their retention counselor. If they know the name (or indicate they know exactly how to find it), it is a positive result. If they have their counselor's contact information in their smart phone, that is a doubly positive finding. This metric should be tracked and reported monthly and should reach plateau benchmarks appropriate for your context.

Initial Experience Assessment

Schools that are focused on retaining students are attuned to their students' goals and expectations. Expectation management is an important service provided by good enrollment and retention professionals. The lack of it is characteristic of substandard professionals in these roles. Schools should systematically seek feedback from students during their first term on a variety of topics including how the school handled the transition from prospect to student, how their early experience matched or did not match their expectations, and how their early engagement with their retention counselors has progressed.

Completed Goal-Setting Meetings

Schools that organize their support services around students talk to their students about their goals for their educations. They capture this information near the beginning of the student's tenure and update it annually, checking in to determine if the student thinks they are progressing towards achieving their goals. One metric to measure if counselors are having these deeper discussions is the number of completed goal setting meetings during the term. The metric would be measured against a standard based on workload and assigned student count.

Resolved & Open Cases

Implementing a case management approach to student issues can be a powerful tool for retention counselors. It gathers information about the types of issues students confront as it helps counselors organize their days and prioritize actions. The cases become the home for interaction records related to the issue so that the solution could be reconstructed later. Metrics for managing such a system would focus on the percent of total cases with successful resolution, the number of cases currently in an open status, and the average time to resolve open cases.  It is important to note that a case management system also aggregates the issues confronted by students and staff. Periodic review of this data might suggest needed policy or training changes. 

Dropped Student Interviews

Finally, effective schools gather information from students who leave in order to understand how their retention services failed. This information gathering is best done via a directed telephone interview rather than an online or telephone survey. This information is collected in real time and collated and reported periodically.

Incremental Retention Metrics

Schools effective at managing their retention and graduation rates focus on cohort retention rates. This means that every student is assigned to a start cohort based on the term she started attending the school. Cohort membership never changes and other key factors in the retention calculation such as transfer status, credits at entry, full time attendance, program, etc. become categories of analysis for the start cohort. Students enter the start cohort when they persist past the 100% refund date in their first term. These schools then track and analyze how these students move through their degree programs. Important incremental progress points in this journey become the key measures of retention.

The devil is in the details when it comes to defining and managing the evolution of cohorts. Bad decisions setting up or managing the migration of cohorts through drops, adds, and program re-directions can produce findings that are useless or even counterfactual.

Cohort Persistence to Second Term

This measures the percentage of the start cohort students who persist through the 100% refund rate in their second consecutive term. Note that some will stop-out and return for their second term later, but this metric is focused on the consecutive advancement. Again, important details must be managed to make this metric useful.

Cohort Persistence to Break-even

In order to use this metric, schools need to know their financial break-even point. This is the number of courses or credits that students must take in order to off-set the cost of enrollment with the revenue from their tuition. This is an important measurement point because until the student reaches it, the school would have been better off from a financial perspective (and so would the student, likely) declining to enroll the student and paying him to go elsewhere.

Cohort Persistence to Second Year

Many schools already measure this rate as fall-to-fall retention. While there is nothing magic about persisting for one year, it is a popular benchmark allowing comparison and the fact that the student likely successfully renewed their financial aid is a good indication of continuing persistence. Analyzing the overall persistence pattern for a variety of start cohorts as they advance towards graduation will suggest other meaningful timeframes to track as metrics. These waypoints vary by program type and length, as well as other institutional nuances.

Conclusion

We view federal, regional, and state attention to outcomes, particularly graduation rates, as a positive change. Unfortunately, the federal definition of retention is flawed due to its reliance on first time, full time students, a group that represents an already small and still shrinking proportion of most college populations. Moreover, the feds especially need to understand that retention and graduation are events that take meaning in programs within institutions and not in institutions themselves. Retention can only be managed effectively at the program level. The incremental cohort-based process measures broadly outlined here provide metrics that managers can influence. The process metrics tied to creating relationships with students empower managers to take control of this vital process. 

When we have worked with schools to implement and manage a metrics system focused on student retention, we have found widely varying conditions related to staffing, technological support, and student culture. These variables resist simple solutions and automated approaches. Retaining students through authentic relationships requires focused work from staff and managers.  

As always, I welcome your views. Please share your thinking below or send me an email.

Keith Blakeman
Vice President of Analytical Services, InterEd, Inc.

The nature of student engagement in higher education is evolving at a heady pace. It seems likely that several trends, occurring in and out of the classroom, will continue to accelerate.

Student Engagement in the Learning Process

For decades, the learning and measurement sciences have told us how to optimize learning in physical and virtual classrooms. Higher education has either ignored or been slow to adopt these findings. Ironically, students now expect to consume and interact with information in ways that are consistent with what these sciences have been telling us for decades. Drivers of higher completion rates and stronger employment outcomes are also propelling these changes.

Here are a few of the more important science-guided practices that seem likely to evolve into standard practice.

Active Learning

Active learning engages students in the learning process. It is gaining momentum among progressive instructors and has been adopted in a few progressive institutions. Active learning shifts the faculty role from sage on the stage to guide on the side, focused on facilitating the active engagement of students in the learning process. This transition from teacher-centered to learner-centered instruction requires the active participation of all students and makes non-participation stand out. On balance, students prepare at the content level prior to class. During class, they engage in a variety of hands-on activities facilitated by the instructor. Done correctly, these activities solidify not only remembering the content but also translating it into skills-based outcomes. The common strand among many active learning techniques is that the primary role of the instructor is coordinating students constructing their own learning in contrast to lecturing on content while the student passively listens.

In addition to improving learning outcomes, the active learning classroom is better aligned with the world of work. It requires extensive horizontal engagement with other students and vertical mentoring and feedback from the instructor that helps develop within students the ability to communicate, work in teams, manage projects, and otherwise interact effectively as required by employers.

The effectiveness of classrooms that integrate horizontal and vertical learning activities is substantiated through research and is now widely recommended (although sparsely adopted) in face-to-face, online, and blended learning. At the same time, the horizontal learning components have been ignored by the current generation of competency based education programs that confuse adapting to each leaner's needs with isolating learners from each other. This lack of innovation is unfortunate. Programs that ignore horizontal components sacrifice one-third to one-half of the potential learning outcomes and commit learning to levels that fall toward the lower end of learning taxonomies. Vertical-only adaptive learning courses recreate century old correspondence courses with attendant low completion rates and repeat after me learning.

Relevance and Authenticity of Curriculum & Learning Experiences

The second trend driving change in the classroom is a demand for relevance and authenticity. Employers, taxpayers and students are seeking more direct relevance and authenticity between what and how they learn in the classroom, and what they will be doing in the real-world environment in which they work and live. Connecting relevant information learned in theory and applying it in practice to real world situations—through simulations, case studies, team projects, internships, etc.—increases student engagement and better reflects the needs of the workplace. Outcomes based curriculum with embedded authentic formative and summative assessments - with an emphasis on formative evaluation - ensure that relevant outcomes remain the focus of the instructional process and provide timely and important feedback to students as well as to faculty.

Personalization

The third major trend occurring in the classroom is the personalization of instruction. Notwithstanding the potential lack of focus on horizontal components (a shortcoming that can be addressed), adaptive learning has the potential to personalize the learning experience in ways that increases engagement and deepens learning. Competency based delivery models have the potential to meet students where they are and allow them to progress at a rate more consistent with their individual goals and capabilities. It is important that emerging adaptive and competency-based models remain grounded in the learning sciences to ensure student engagement is in place to support optimal learning outcomes and persistence.

Student Engagement Outside the Classroom

Many of the forces driving change in higher education classrooms are also driving change in student engagement outside the classroom. 

Communicating and “Connecting” with Students Collectively & Individually

Communicating relevant information to students consistently and effectively has always been challenging. Over the past three decades, I have lived this constant challenge of facilitating authentic communication that “connects” with students across my roles as coach, faculty member, athletic director, and for 20 years, senior administrator. The tools for communicating with students evolved from traditional bulletin boards, phone calls, letters and printed newsletters to the enhanced capabilities of the web, email and the more dynamic interface and broader reach these new technologies provide. About the time administrators felt they were catching up, social media exploded and consumer demand for mobile applications that provide relevant and personalized information reached higher education. Mobile applications are enabling students to communicate with higher levels of personalization and relevance. New applications are student centered and offer underlying data analytics that help inform the right messages to reach the right students at the right time. The best of these applications connect students to appropriate support services such as tutoring, research assistance, and career service support.

Similarly, advancements in technology have provided new retention CRM systems that provide a platform for faculty, staff, and administrators to collaborate in coordinating engagement with students by the right person at the right time and place throughout the student life cycle.

Building Quality Relationships with Every Student

The advancement of new student communication and retention technology solutions will not contribute to student persistence unless they facilitate contact between students and the people dedicated to building quality relationships with them.

The addition of student success coaches or advisors who are responsible for building relationships with each student tracked the expansion of online programs and now needs to expand further across all students and instructional delivery modalities.

The goal of supporting students in achieving their educational goals requires systemic processes for personalized and relevant communications that build quality relationships with each student. Trying to predict which students will and will not succeed has proven to limit the objective of supporting each student in attaining their educational goals. Even students with a 3.5 GPA who appear to be progressing nicely toward degree completion can make precipitous and unpredictable decisions to leave the institution without anyone in the institution knowing why before the fact. Thus, the primary objective of the student success coach or advisor role is to build a personal relationship with each student such that the student will reach out if something occurs in their academic or personal life that poses a risk to persisting toward their academic and career goals.

Engaging & Equipping Students with the Thought Patterns for Success

Recent neuroscience studies related to student persistence reaffirm the dynamics of multiple psychological frictions throughout the student life cycle. Everything gets better when we equip students with an understanding of the thought patterns that underpin persistence and success in college and elsewhere. We need to help students understand that what they are feeling is not unique to them, but is common to many who successfully completed college before them, as well as to many of their current classmates. It is also important to equip students with the thought processes necessary to change or “reset” the habits, attitudes, beliefs, and expectations that may be inhibiting them from reaching their full potential. Understanding that the locus of control lies within one’s self and not with something or someone else, is a determinant of persistence and “grit,” and can be life changing for many students.

I am concerned about another mental health issue. I see many students who are unable to maintain attention and focus. The speed at which information is presented today, and the speed at which it changes, has increased the proportion of students who cannot keep pace. Whether through increased diagnostic capabilities or a increased incidence, we are seeing more students suffering from severe anxiety, ADHD, and the common accompanying symptoms of having difficulty maintaining attention and focus on important goals and tasks. Increasingly, we will be called upon as educators to equip students with the skills to filter and prioritize required to be effective in a world of 24/7 stimulation. Providing the “mental technology” that students need to persist and succeed in college and life is an essential part of an effective student engagement model of the future.

Concluding Thoughts

I see merit in looking holistically at the major disruptors, both globally and within higher education, and how these disruptors are changing student engagement. I also think we benefit from self-examination of our organizational cultures, policies, and practices. Are we well served by three-year program review cycles and five year strategic plans that fail to anticipate the external environment? Are we doing business in a way that supports the rapid innovation required to meet the evolving expectations of today’s learners for personalized and relevant engagement? If not, what changes are needed and by what metrics can we judge their success? One generalization seems certain. If our policy and practices were working well a decade ago, they may not be suited for the coming decade. Narrowing the gap between the accelerating pace of change in the world around us and the time it takes us to adapt our organizations to these changes is likely one of the most significant challenges facing institutions of higher education.

As always, I welcome your views. Please share your thinking below or send me an email.

Blake Faulkner
Vice President of Strategic Partnerships, InterEd, Inc.

This series of mini-briefings is based on generalizations arising from the study of elective drops non-residential (AKA: in adult-centered, non-traditional) programs of study in higher education. Among the central generalizations are: (a) elective drops are most accurately modeled as critical events that can progress in a matter of days from initiation to execution; (b) trying to improve retention with predictive modeling, a method growing in popularity largely due to sales initiatives, is unlikely to produce results above chance levels in a well-controlled experiment; importantly, predictive modeling's tagging of potential drops cannot predict specific drops within the actual event horizon; (c) the most effective way to improve retention is to develop a genuine relationship with students such that they feel comfortable to, and will, reach out for help early in a rapidly developing drop event. 

Tips in this series describe something you can do to improve retention that does not require adopting a complete system with which to manage retention with precision. At the same time, each tip is congruent with the kind of precision managed system we know to be most effective.

Tip - Celebrate the Most Personal of Events

Like ours, the lives of students are filled with noteworthy events, large and small. Birthdays, family and job changes, and moving into a new home are among the more common events. You already have the information needed to recognize one of these events in your system: students' birthdays. 

Recognizing a student's birthday is not likely to have a direct impact on retention. Done improperly, it will have no effect or even a negative effect. The causal path to improved retention is via the recognition's contribution to creating relationships with students that increase the probability they will reach out for help when they consider leaving school.

How Not to Recognize a Birthday

• Send a boilerplate email that speaks, if not shouts, "I have no idea who you are and I don't care enough to find out. I have this system thingamajig that sends emails without my having to spend time getting to know you."

• Send any communication that begins "Dear Thomas, III" to a person whose family and friends call, "Ricko" (you have this important information in your SIS, right?). Translation: No one home on this end.

• Send a system-level communication that you can get a 10% discount at the bookstore on your birthday, so long as you can produce a government ID. Translation: Happy birthday. Give us some money.

• Work a back-end deal with a company that makes you money when the student contacts them to get his "birthday discount." This applies to other relationship neutralizing back-end deals that make you money by asking the student to spend money. Translation: See above.

A Better Way

• Contract with a fulfillment house to deliver a small gift at the time of a birthday. Ideally, these gifts build pride in and commitment to the school. Mugs, sweaters, pens, coasters, etc. are examples. The gift might be larger each year as the student approaches graduation.

• A week before the gift is scheduled to arrive, give the student a heads up that the gift will be arriving. Reach out in a personal way and, at the same time, create a little suspense. Convey that the forthcoming gift is your personal invitation to stay connected. 

• Close your greeting with your phone number, email, and an invitation for the student to contact you any time he or she needs a little help sorting out the path to success. Convey that you know how much sacrifice is involved in earning a degree and that you understand it is never easy.

• Exploit your technology to its fullest. All SIS/CRM platforms can queue pre-structured action items based on student birthdays. 

• Consider communicating by text or telephone rather than email. Both are generally more personal modalities than email. If you use email, have your CRM deliver pre-formed templates that require personalization before sending. 

Who should send this somewhat personal email? Ideally, it is the job of the student's full time retention counselor. If that role doesn't exist (yet), consider creating a role for academic advisors and/or faculty to make the contacts.

Can You Afford It?

One of your colleagues may opine that the budget has no room for this kind of generosity.

Really?

You are charging each student $20,000 or more per year, plus fees and the profits you make off of textbooks and supplies. How well accounted and efficient would your institution's delivery system need to be to determine that there was no internal margin with which to offset the cost of a mug that the bookstore sells for $20 at a cost of $2.42 in quantities of 500?

The better question is, Can you afford not to communicate with your students on a personal level? Can you afford to position your institution as the decision of least consequence when it comes to reducing an overly full personal workload? Can you afford to ignore opportunities to establish and maintain the rapport required to make students feel comfortable in reaching out for help at exactly the time they feel that they need it?

You can offset the cost of quite a few mugs or jackets with the revenue derived from retaining a single additional student.

More Tips On the Way

Of necessity, each mini-briefing omits detail, especially the empirical information on which we base our recommendations. Let me know if you would like to explore this or other tips in more detail.

As always, I welcome your views. Please share your thinking below or send me an email.

Robert W Tucker
President, InterEd, Inc.

This Briefing shows that prevailing ideas about academic quality are outmoded and inadequate when applied to colleges and universities. It shows that these shortcomings are both logical and empirical in nature and that they do not exist in comparable sectors of the economy.  Executive understanding of quality matters because it affects how we write policy, set goals, compensate individuals, present ourselves to the public, and how we measure inputs, processes, outcomes, and impact to convey to others, and with which to manage and improve practices.

Uneasy Tension Between Academics & Quality

More than a decade ago, I addressed the nation’s chief academic officers with a simple message. As used in academic circles, the term “quality” has become a nearly meaningless idea, rooted in the past, and used to defend the received view of academic structure and process.

I was not the first person in higher education to observe this. Forty years ago, Ted Manning, the accomplished and visionary Executive Director of HLC, was fond of pointing out that higher education's claims about quality amounted to little more than hot air. Dr. Manning’s observations – astute when he made them – are no less so today.

Notions of quality are at the heart of the most intractable problems confronting higher education. Among other things, they provide the foundation for federal, state, and institutional policy and practice. Yet, leaders are confused about the meaning of quality and those representing different sectors of higher education lack common ground for definitions, standards, and outcomes. In the end, discussions between sectors amount to arguments in which “quality” is used as a shibboleth, employed to defend a point of view in the absence of good evidence or even a decent vision of the goal.

It is something of an understatement to observe that colleges and universities do not understand, define, measure, manage, or communicate product quality to themselves or to the public in a way that would be considered minimally adequate to those who understand the meaning of quality in other sectors of the economy.

From Then to Now

Forgive me for packing a few centuries of progress in philosophy into a paragraph. It will become clear that doing so is useful to this discussion. Several centuries ago, philosophers attempted to identify the meaning of various properties of our value systems, beauty being one example among many. These scholars reasoned that because people find a flower, sunset, face, or painting “beautiful,” there must be a property which, if understood correctly, can be found in all things we deem "beautiful." This property, whatever it turned out to be, must be "beauty" in the abstract. Twentieth century philosophers abandoned that classic thinking for good reasons. Instead of constructing meaning through philosophical debate, they examined how we actually use such terms. In doing so, we saw - against expectations - that many terms do not share a single criterion that applies to all legitimate uses. For example, the only property all uses (or cases or instances) of "beauty" had in common with each other was the word itself. These insights changed the way we looked at many terms and, especially, value terms. Wittgenstein's "Don't ask for the meaning, ask for the use!" became the useful analytic construct. Indeed, many words in our language derive meaning solely from their context and have no single criterion in common across all uses of the term. [1]

Pretty much everything I just said about “beauty” can be said about “quality” as it applies to higher education. Each context of education can be described along a dozen or more dimensions each of which has meaning that we would want to call "quality." However, the mix of each of those dimensions, and their relative importance within the specific mix, is unique to that instance of education. 

Suitability to Purpose – Highest Meaning of Quality

A longer version of this article unpacks a list of progressively more adequate notions of quality. The lower end of this list reflects uses most common in today's colleges and universities. (Click for PDF Table). The list begins with Apodictic Quality, the most ancient among uses of the term (quality is self-evident because of who we are), progresses to a mid-point in which quality appeals to continuous improvement [2], and ends with quality referring to a dimension of suitability to purpose. That is "high quality" means perfectly suitable to the intended purpose (or nearly so).  

Empirically and logically, suitability to purpose subsumes and rationalizes other uses of the term "quality."

Example

Automobile quality provides an uncomplicated illustration of how suitability to purpose informs useful definitions of quality and the metrics that follow those definitions. Most of us can list common criteria for automobile product quality. Your list might be slightly different than mine but our lists would be similar. My short list would include the following: 

• Price

• Safety ratings

• Fuel economy

• Human & luggage capacity

• Driver comfort

• Passenger comfort

• Technical features

• Handling

• Projected maintenance costs

• Style & appearance

• Performance

• Projected resale value

The Logic of Quality

This list of quality dimensions for automobiles should make it clear that there is no single “highest quality” automobile. Defining quality involves identifying the consumer’s purposes (or uses) and applying them to the list of features and specifications. For one consumer, the highest quality automobile is the one that delivers the best mileage at the lowest cost with above-average safety, a red color, a moon roof, and little or no attention paid to other quality metrics. For another consumer, the highest quality automobile is the highest performing, best looking, full featured car that has a superior maintenance record. Fuel economy is near the bottom of this consumer’s list.

Assuming that these 12 dimensions of quality were equally scrutable and were all deemed important and ranked in order of importance by everyone who purchased an automobile, we would end up with  more than 479 million permutations of a high quality automobile for as many consumers. This logic applies to choices in higher education as well. 

Why There is No Such Thing as the Highest Quality

It might seem like my argument is grounded in semantics. After all, one or perhaps only a few automobiles will be found at the top of all criteria on the list? In a way, this approach mirrors the errors of the 17th century philosophers. It is not possible to create such a list for the following reasons. Some quality attributes for automobiles form polar dimensions, either empirically or logically. For example, it is not empirically possible that an automobile can have the best acceleration and top speed and also the best fuel economy. The limitations here are grounded in contemporary physics. It is also impossible for a car to be both compact and Spartan, and large and roomy. or to have the best data on road feel and isolation from road feel. These are logical constraints.

There is a less rigorous but no less compelling way to look at the meaning of quality. If there were such a thing as an automobile that somehow, against logic and science, optimized all of the dimensions of quality that apply to automobiles, the market for this machine would be limited to billionaires. 

This might be a good time to step back and contrast these points with the rhetoric of quality we commonly hear within the community of higher education. Dr. Manning's words are haunting.

Takeaways for Higher Education

Four important considerations emerge from this example of automobile quality:

• Consumers are generally in agreement about what quality can mean when applied to a product or a service that is transparent in the marketplace.

• On balance higher education provides a service but our badges acquire some properties of products . . . and we are far from transparent.

• As applied to their individual purposes, consumers differ as to which criteria are meaningful to them and what weight should be assigned to each criterion. Each consumer’s purpose will define a unique mix of criteria and weights in determining the meaning of quality. This is as true in higher education as it is anywhere else.

• When we think of quality as suitability to purpose, we also introduce questions about whose purpose and, in the case of multiple stakeholders as we see in higher education, how these purposes, all of them potentially conflicting, will be adjudicated for precedence. 

• Unlike older notions of quality that rest on fuzzy, illogical, or zero sum criteria, it is possible for every institution to deliver high quality. It is done by meeting the purposes of a defined group of consumers.

Higher Education Versus Other Notions of Quality

Some colleagues would have us believe that the notion of quality gets much more complicated, inscrutably mystical even, when applied to higher education. Some even believe that “quality” is what Wittgenstein would have called a private event that can be intuited only by them, and not subject to objective scrutiny. I have even been told by some instructors that no one can measure the quality of what they teach. The conversation deteriorates when I ask, given that, how they manage to assign grades.

Is what there is to mean by “quality” in higher education more complicated than we find in the rest of society? Perhaps only slightly because of its opacity. The meaning of quality in higher education ("the use") is arrived at in same way as the meaning of automobile quality.

Perhaps as with automobiles, although with less certainty, most of us can list common criteria for quality in a specific instance of higher education. Your list might be different than mine but, even in this controversial area, we would recognize each other's lists. Here is my first draft:

• Fidelity of content to applicable standards

• Alignment of program to need in the marketplace

• All-in cost (tuition, fees, hidden fees, discounts)

• Mean time to degree (probability of graduating on time)

• Opportunity cost of delays in delivering degree [3]

• Judgment of merit by potential employers

• Convenience (location, blending, pedagogy)

• Focus of the degree (applied/practical, theoretical/academic)

• Faculty experience in relation to focus (academic cf. professional)

• Alignment with learning sciences (goes to benefits and ROI)

The Role of Student Goals in Defining Quality

Questions about goals are questions about purposes and questions about purposes appeal to quality. Because higher education typically involves many stakeholders whose goals are seldom aligned, it becomes important to make decisions about whose goals should be considered and with what precedence in relation to the goals of other stakeholders. Most colleges and universities pay little attention to the specific goals of their students. If they did, students' goals would be assessed and rationalized to a family of metrics at the time of matriculation, and these metrics would be measured and actively managed throughout each student's course of study. They would define success as how well the goals were met.

Which quality metrics will be important to a working adult student who already has a degree in administration and wants to earn a certificate in accounting for the sole purpose of being better informed in her weekly management meetings? Which metrics will be irrelevant? Which criteria will define quality for a student who wants to become a licensed psychologist? How about someone who has no interest in a job and is returning to school purely for personal development? What does quality mean for this person?

Opacity & Optimum Quality

Is it possible for an institution, or department, or program to achieve the highest ratings on all of the above dimensions of quality? The answer is “no” and, as with automobiles, the constraints are both empirical and logical. This fact holds implications for the definitions of quality sometimes invoked by elite institutions.

The automobile consumer will be able to achieve an optimum definition of quality by taking a test drive or two, and by studying the federally mandated window sticker along with a review from Consumer Reports. No such path exists for consumers of higher education. The historically self-serving structures and practices of higher education providers have prevented the kind of transparency that would facilitate rational consumer decisions.

This opacity exists between not only the institution and its consumers; it exists internally at all levels as well. A college president and her cabinet are in a fortunate minority if they possess real-time objective data on two of the above 10 quality dimensions. One would think that the regulatory and oversight bodies would take affirmative action to require not only gathering these metrics but for using them to improve and for communicating quality to consumers. They do not. For some measures, they support the institution in resisting implementation; for other measures, they are silent.

Roles of Other Stakeholders in Defining Quality

Some will object to the fact that I have centered this discussion on the quality criteria of the consumer while ignoring the definitions of other stakeholders. This is intended. The definitions of other stakeholders are important but they have over-determined higher education to the virtual exclusion of the consumer. It is bad reasoning on several levels to continue to push consumers' definitions to the periphery. .

Making the Switch

Managing quality as suitability to purpose is not difficult. Chances are that it is already being done in some areas of your institution, perhaps in materials acquisition. The steps involved in managing quality as suitability to purpose is the topic of another Executive Briefing. Let me know if you are working on these issues. I will be happy to send additional documents that may be helpful. I'll close with an elaboration of a point made above. 

Unlike older notions of quality that posit or allow for a single exemplar of highest quality, it is possible for every institution to deliver high quality. It is done by identifying, managing to, and ultimately by meeting the purposes of a defined group of consumers. This is why it is important that we begin to measure, manage, and assess progress toward students' goals.

As always, I welcome your views. Please share your thinking below or send me an email.

Robert W Tucker
President, InterEd, Inc.

Endnotes

[1] Philosophers call this “family resemblance” after the work of Wittgenstein. Think of members of a family in which a few relatives share a nose but not cheeks, others share cheeks but not a nose or chin, and so on. All of these people look related to us yet there is no single visual resemblance that unites them all.

[2] CQI was the organizing principle behind HLC’s AQIP, introduced in the late 1990’s. Unfortunately, AQIP’s requirements for continuous quality improvement have been honored in the breach. Schools that choose AQIP as their accreditation method are approved year-upon-year based on “progress reports” showing that they are getting ready to get ready or having meetings about meeting structure. AQIP has become a refuge for institutions that wish to escape real accountability – the very outcome AQIP was designed to avoid.

[3] The opportunity cost of delays for a nursing degree might begin at $59,000 per year plus and minus other lesser factors. Opportunity cost is often overlooked in determining the returns of a degree yet they can quickly add up to make a degree earned on time at one institution a better buy than a less expensive institution that has a history of not having needed classes on time.

In the long term, it will be decisively in your institution's academic, financial, and operational interests to create a comprehensive approach for managing retention with precision.

What to Do While Gearing Up for a Comprehensive Approach

In the near term, you may not be ready for a comprehensive approach to managing and improving retention. You might be looking for smaller, one-off changes that will increase retention while your institution is working through the administrative and governance processes needed to clear the way for a more efficient structural approach.

While comprehensive retention management programs can produce very high ROI ratios, adding millions of dollars to your institution's revenue stream, less comprehensive efforts can also produce relatively high returns, even if the magnitude of effect is less. All levels of effort can improve student satisfaction, institutional effectiveness, and quality referrals.

Steps You Can Take to Achieve Near-term Gains in Retention

• Direct marketing and enrollment to focus on students you can serve effectively from matriculation to graduation. Enrolling the wrong students is bad practice on several levels. A growing proportion of inappropriately enrolled and subsequently ignored students will drop before they attain the point of financial break-even in your program and well before they attain their personal goals. This classic “lose/lose” outcome (you would have been ahead financially to have denied admission/they failed to meet their goals and associate that failure with your institution) is unfortunate for all, especially when creating a win/win environment is less costly.

• Identify each student’s operational definition of success in your program. What are their goals, purposes, and symbols of a successful outcome? In most settings it does not require a structural change to add a step in which enrollment counselors work with incoming students to identify and refine goals into incrementally measurable criteria to which appropriate metrics are attached.

• Require a “warm hand-off” between each student’s enrollment counselor and whatever role you have in place for post-matriculation retention management (academic adviser, dedicated retention counselor, etc.). A warm hand-off is achieved by making a three-way emotional connection between the new student, the enrollment counselor, and the post-matriculation person. That connection is typically made by a three-way telephone conversation or, if appropriate, in person.

• Make certain that every new student has not only the name of the post-matriculation counselor but has entered his or her mobile phone number and email into their smartphone. Create incentives for this level of connection. A web link, which may include a Facebook or Twitter connection, may also be appropriate.

• Keep in touch with students proactively. Eliminate business rules that reach out to students only when there is an error or you need something from them. Students should feel that they are connected with someone representing the institution for reasons related to caring. 

In most colleges and universities - including those that include "caring" in their mission - students report that the only time they hear from the institution is when someone has made a mistake or the school needs something from them.

• Interview recently and voluntarily departed students to learn why they left. NB: You need skilled interviewers to do this. Surveys will be misleading because the presenting reason is seldom the one that motivated the drop. This kind of assessment should be a continuous background research activity.

• Interview and track the reasons for thinking about dropping expressed by students you were able to help retain to success.

• Establish a process to evaluate and act on the aggregated goals of students in relation to to program goals. This kind of information is helpful to program leaders and managers. 

The above process changes are relatively simple, inexpensive, and constructive in the larger sense. They will assist your staff in building the kinds of relationships that make it easy and even expected for students to reach out for help at their first sign of being at risk.

As always, I welcome your views. Please share your thinking below or send me an email.

Robert W Tucker
President, InterEd, Inc.

Retention Flows from Relationships

How we conceptualize retention carries implications for how to improve it. This Executive Briefing provides a senior-level summary of an evidence-based way to think about retention. Related briefings focus on interim tactical and long-term strategic actions to retain students to a successful conclusion of their course of study.

I suspect that we all see retention-to-graduation as the sweet spot of convergence among the diverse and sometimes conflicting values held by higher education’s stakeholders. Students, instructors, deans, administrators, financial officers, regulators, and the public are all pleased with a high graduation rate. These same stakeholders see failure when students who have spent a great deal of time and money end up failing to secure desired credentials.

Ignored Class

No one should be surprised that so many students fail to graduate. We do little to acknowledge their existence, much less help them succeed. On the enrollment side, a prospective student may or may not be able to secure the assistance of a competent and dedicated enrollment advisor who will help them work thorough the thicket of rules, requirements, and the many up and downsides associated with various degree paths. Anyone who doubts this claim argues against good evidence. We have been shopping enrollment behavior for more than decade. More than half of the programs into which we drop a phone call or an email inquiry never respond to that inquiry! For those that do respond, only a small proportion do so competently. Nonetheless, the persistent will eventually find a way to enroll. The others will enroll in a school that ignores them less or not at all.  I have been told by many college leaders that needing to fend form themselves screens for students who will succeed. This thinking begs for evidence since the highest drop out rates tend to be associated with schools that provide the fewest pre- and post-matriculation support services. Once admitted, students advance to the next wave of being ignored. In spite of the fact that many admitted students will not show up for the first day of class (admit to matriculate rate), these gap students typically wait it out without benefit of preparatory support from the institution. Finally, those who matriculate are then ignored by everyone but some of their instructors unless they fail to pay a bill or submit a required form at which time they receive d a great deal of attention. For most universities, no one is responsible for developing and advancing a caring and helpful relationship with students. I cannot think of another place to spend from $50,000 to more than $200,000 only to be ignored by people whose livelihood depends on my tuition.

Thinking about Retention - What Does the Evidence Tell Us?

This conversation is about elective drops. We all recognize that people move, change jobs, or experience other disruptive life events that necessitate dropping our of a degree program. There is no shortage of views on how to think about and improve retention. Looking back over 30 years of systems development, I see that my views have evolved from what might be called "an economist's view" in which my focus was on aligning incentives, to my current "psychologist's view" in which I find it useful to understand the dynamics of retention, including the event horizon of the decision to drop out.

We have achieved the highest rates of retention to graduation by implementing a structurally integrated approach to creating and maintaining quality relationships with students. By students, I mean all students, not only those tagged as likely to fail. The primary goal of building these relationships (there are benefits unrelated to retention) is ensuring that students reach out for help at the very moment they perceive themselves to be at risk and are in the early stages of contemplating the benefits of dropping out.

Drop Dynamics

• The most robust way to think about elective drops, whether pending or in process, is as critical events that are essentially unpredictable in relation to their event horizon.

• The period between the beginning and the end of the drop decision is generally very short, often less than one week. A vague concern about "making it" can emerge Friday afternoon (perhaps upon struggling with a project) and progress to an irreversible drop decision by Monday.

Intervening

• Although the objective evidence is less impressive than is generally believed, the predictive models currently in favor exceed chance levels in identifying students at at risk for dropping. However, these models are silent as to when the dropwill occur and when they do offer these predictions, they do not exceed chance levels predicting within the drop event horizon. In addition, adverse consequences are associated to the high number of false positives and negatives produced by these approaches. I would encourage anyone contemplating software-based predictive modeling systems to have a measurement scientist examine the true discriminant validity of these solutions. Chances are these data will not be available.

For the reasons identified above, I do not recommend predictive modeling as a solution to improving retention.

• The best method for reducing elective drops, especially in cases where these potential drops would go on to be successful, is by ensuring that students enjoy a growing relationship with a single retention counselor. The nature of the relationship should be such that it is natural and expected that the student will reach out for help at the exact moment help is needed. Another Briefing will address this in more detail.

High Returns

• The return-on-investment ratios for well designed and correctly implemented retention management systems can easily exceed 50:1 in larger institutions. Even at ROI ratios in the low 20's, if improving retention rates by 20% adds $3M to top line revenue, the sunk and first year incremental costs to achieve these financial gains will be in the area of $100K. The exact ratios and expenses will be determined by a number of factors, including the capabilities of your current CRM system. 

Those who think that developing quality relationships with students is too costly and not warranted, might also think that the high touch concierge services, now standard practice in the most profitable hospitals, reflects bad decisions by hospital administrators. Perhaps they know something we don't.

As always, I welcome your views. Please share your thinking below or send me an email.

Robert W Tucker, Ph.D.
President, InterEd, Inc.

A recent article in the Chronicle reported issues on the minds of college administrators and faculty as they contemplate the future of higher education. One arguably important issue - whether college instruction and evaluation should conform modern learning and evaluation sciences - was not mentioned. This omission is odd in light of growing concerns about costs, outcomes, institutional effectiveness, impact, time-to-degree, and other matters that rest on effectiveness and efficiency. 

Pre-scientific Teaching Methods

Higher education employs a 1906 teaching model that is pre-scientific with respect to the relevant learning sciences. We are familiar with the 1906 model not only because we were taught by professors who employed it, we are familiar because we employ it or a close variant in our own teaching.

The model is, “Read this chapter, answer these chapter questions, listen to me lecture, and take this test.” The model has been modified slightly over the years to include small changes as higher education became less the exclusive province of the rich and the smart and more the public path to careers. The most common modification added, “chat a little about this as a class” somewhere in the middle of the sequence.

Teaching out of a 1906 playbook was defensible in 1906. At that time, early learning science research was underway in a few psychology labs but the findings were inconsistent and had limited practical relevance for the university classroom. Teaching out of a 1906 playbook is not defensible in 2016 following decades of generalizations from cognitive, affective, and brain research pertinent to improving learning, retaining, generalizing, and applying what is taught in the classroom.

The case against 1906 can almost be made by way of analogy. How would we respond to a physician who deliberately ignored the last 100 year's of progress in the medical sciences?  Would anyone pay this person to engage in his trade?

The 1906 playbook has several defining attributes that are worth examining. Most prominent is that teaching and learning describe a vertical relationship. The professor is the source of knowledge that is disseminated downward to the student. Horizontal learning (student-to-student and learning team to learning team) is not provided for in the belief system or the classroom structure and is not evaluated when it occurs. Teaching and learning tend to center on low level cognitive content. By low level, I do not necessarily mean simple or easy to master. I mean that the content centers on knowing something, however difficult or complex, rather than demonstrating a proficiency to do something that subsumes that knowing.

Teaching in 1906 also emphasized the development of abstract cognitive mastery as demonstrated by recall over the kind of demonstrated competence and enhanced learning that is associated with teaching to others what you are attempting to master.

A Responsive Audience

Empirically, 1906 teaching works reasonably well for students who are intelligent in the field of study and whose strengths tend toward abstract thinking. That is, it works reasonably well for almost everyone reading this Executive Briefing. It is therefore not surprising that we teach to people like ourselves.

Invalid Assessments

Multiple choice tests, essays, and term papers – most of them executed in a high stakes environment – are the stock in trade of the 1906 model and today's professoriate. All of these common tools are riddled with multiple kinds of invalidity. While it might be useful to develop a more technical Briefing on this topic (email me if you would like to see one), the common threats to validity of these common methods of assessing learning can be summarized in a non-technical fashion.

Essays and term papers receive different grades if re-evaluated after a lapse of time or when identical content is presented in a neatly organized laser printed document and a less neatly organized handwritten paper. Blind evaluations (students' names removed) receive different grades and comments than the same papers unblinded. Gender identify and surnames can affect grading and comments in many ways. In general, the most common weakness in essay tests is low reliability (e.g., score/re-score). With training and diligence, these structural sources of invalidity can be reduced but not eliminated. Blind scoring and scoring rubrics can increase reliability coefficients but only up to a point and only if they are competently developed and applied. To get decent reliability statistics, it is often necessary to score independently using two independent scorers trained to proficiency using carefully constructed dummy essays.

Multiple-choice tests contain a large number of questions that do not discriminate sharply between knowledgeable and uninformed students. Some of these items, commonly more than 20%, discriminate negatively, meaning that low achieving students tend to answer the question correctly while high achieving students do not. Most multiple-choice tests also contain about 20-30% of items that do not discriminate at all between high and low achieving students.  Like essay tests, training and diligence can improve the validity of multiple-choice tests. Multiple-choice tests can be subjected to various forms of item analysis and refined to meed technical standards of validity.

The lack of validity in what 1906 tools measure is less subject to improvement.

The technical failures of low discrimination, negative discrimination, and low reliability are all low level failures of instruments that fail in a more significant way. Both essay and multiple-choice tests (although the problem is greater for multiple-choice) tend to measure the low end of taxonomies of learning; e.g., being able to recall facts and relations. While assessing at the low end of a learning taxonomy is seldom a good idea, this kind of assessment is understandable in courses for which the rationale for taking the course is to be able to recite facts and relations among facts.

The greater problem in relying on these tools to determine what students learned is that most courses aim for a higher purpose. Many courses seek to create in their students the ability to analyze, synthesize, and evaluate alternative courses of action with respect to a problem nexus. Essays and multiple-choice tools can poke around the edges of these dimensions of learning and, with exceptional effort, can get at them in limited ways, although few actually do; the design and validation efforts required are often beyond the capabilities of the person creating the test.

These old assessment tools fail at an even deeper level.

Increasingly, courses seek to develop proficiency in application, which is defined as specific behavioral competencies. The best way to assess at this deep level is through the use of hands-on (performative) tasks that are evaluated in multiple dimensions, in multiple ways, from multiple perspectives or sources. These forms of assessment work best if they are authentic, meaning that the structure and content of the assessment is faithful to or congruent with the way competence is evaluated in the target or end-state application, whether the workplace, family, church, or community. Drilling down deeply into authentic learning and assessment is out of scope for this Briefing except to note that I believe it will become the emerging form of teaching and evaluating.    

Does It Matter?

• It matters that our professors teach the same way their professors taught them and, in turn, were taught by their professors. Fifty years of learning sciences have taught us more than a little about how to improve the efficiency and impact of teaching activities.

• It matters when a learning scientist teaches her students about research-based ways of teaching and learning that result in reduced time on task and improved retention and application, and yet teaches these scientific findings with pre-scientific methods that are less effective and efficient. This professor has missed an important opportunity to conduct field research.

• It matters when a measurement scientist determines grades based on a single high stakes multiple-choice final examination for which 20% of the test items discriminate negatively and 30% of the test items fail to discriminate above chance levels. This professor has ceded his right to make important distinctions among levels of achievement, arguably a hallmark of good teaching.

• It matters that the standard 1906 assessment tools measure little more than rote memory, and not all that well. Increasingly, the demands of the workplace go to day-of-hire behavioral competence.

• It matters that the correlation between 1906 assessment scores and subsequent workplace competence is not much better than chance after excluding the outliers at both ends of the distribution. Employers in many sectors have lost confidence in the predictive validity of grades and degrees.

• It matters when a teacher cannot beat chance levels in assigning the same grade or making the same comments on the same essay examination he graded six months earlier or graded the week before in a different presentation format. Students are sensitive to these manifestations of inequity. The impact can be demoralizing.

On the other hand, haven't we been managing college classrooms 1906-style since horseless carriages were all the rage. Learning takes place, and the system works even if it is inefficient. In addition, a growing number of one-off professors and a few programs, most notably in the medical and health sciences, have aligned themselves with modern learning and measurement sciences. We can be sure that others will follow.

Reasons to Hurry this Along

The answer to the "Why the rush?" question lies not only in our patience but in our vision of the urgent needs confronting higher education. Let me suggest two good reasons to rush.

• Migrating a degree program from 1906 to a well designed 2016 program would produce a four year degree in less than three years with proportionally less cost and superior outcomes. That extra time is available to be traded off between duration and intensity.

• The retention rates of programs based on learning and evaluation sciences are higher than those of 1906 programs. This is because the learning sciences have taught us something about the impact of engagement, horizontal learning, and authenticity. Remember, people like you tend to arrive with your own engagement in place. You do not represent the typical learner of 2016.

These two reasons among many may be reason enough to push this glacier along.

As always, I welcome your views. Please share your thinking below or send me an email.

Robert W Tucker, Ph.D.
President, InterEd, Inc.

An earlier version of this Executive Briefing was published in 2012 pursuant to guidance we provided clients beginning in 2010. Since that time, external pressures on universities to lower the total cost of earning a degree have increased to what seems like an irreversible tipping point.

Six Percent Problem

In 2010, I recommended that our clients buck the tide of six percent year-on-year tuition increases. Instead of raising tuition and fees, or even holding the line, I suggested that they reduce the cost of attending, and that they seek the net revenue numbers they desired though increased margins brought about by increased retention, better content management and delivery systems, and launching new and needed programs while phasing out legacy programs for which there was no longer any demand. While recognizing that implementing changes to increase margins will take time, we suggested reduction targets ranging from thirty to fifty percent in the total cost of earning a degree. Upon reflection, a few institutions decided to move in this direction.

The only change I am making in today's recommendation is to elevate the importance of this goal.

If our projections are correct, most adult-centered, professional, and many other non-residence programs will be dominated by institutions that initiated a trend to maintain tuition in the lowest quartile of the cost distribution.

This recommendation, and its growing urgency, is based on multiple sources of downward pricing pressure arising as a reaction to current prices that are indefensible and projected rates of increase that are unsustainable. We need to do more than slow the rate at which higher education prices are increasing; we need to roll prices back. Today, we see downward pricing pressures from the White House and the Department of Education, from private foundations, from market-savvy entrepreneurs who sense a rare opportunity created by collapsing anti-market forces, by students facing financially damaging loan amounts, and by an increasingly dissatisfied and vocal public. This problem is only worsened by the "head in the sand" posture adopted by too many colleges as they post yet another 6% year-on-year price increase in a 2-3% GDP growth economy. A few of these institutions believe they have fooled the market by embedding some of the increases in fees.

Evidence of Success

Our evidence is limited to two universities over a short time frame. However, two institutions lowered their tuition dramatically in accordance with our general guidance. Prior to lowering tuition, both of these institutions were experiencing declining enrollments in response to increasing prices to the extent that they were seeing a net decrease in revenue and margin.

Initial post-reduction indications are that these institutions are realizing the increased enrollments and increased revenue that we predicted. These institutions had priced themselves into the zone of negative price elasticity, a dangerous situation that we observe developing in a large and growing number of institutions, especially independents and for-profits. I refer to this situation as dangerous because negative price elasticity can be described as a zone of reversed control. Once in this zone, options for getting out are limited and the institution becomes vulnerable to pricing strategies put in place by a clever competitor.

Elements of a Solution

Whether strategic or tactical, I recommend that any institution concerned about the growing risks of being overpriced in the market develop a fresh and comprehensive pricing strategy. This means dropping all assumptions and developing a vision that will carry forward into the foreseeable highly competitive market conditions.

Informing and developing this vision will take time - three to six months. In the meantime, smaller steps are available that will inform the overall strategy.

To be clear, the following suggestions belong in but do not define a comprehensive strategy to become competitive in tomorrow's markets. Some of the margin increasingchanges that I mention above involve structural changes. The suggestions outlined below can be implemented under current policy and practice.

Account for Revenue & Margin by Program

Establish revenue and margin targets for all programs. Manage each program independently.

Establishing revenue and margin targets for all programs does not mean that all programs need to be profitable. It does mean that, once set, each program needs to meet its targets, even if they are modest, neutral, or even negative.

For example, a strongly mission-driven program might operate under a negative 30% margin, thereby establishing how much the institution is willing to lose on the program. Once set, however, the program's leaders must work to achieve that level of loss if the program is to be retained.

As obvious as this recommendation might appear, many institutions cannot implement it without addressing limitations in their accounting models, decision-metrics, and software choices. Nonetheless, if I were to make a single recommendation, this would be it. I have thoughts on this topic that are out of scope in this Briefing.

Manage Content

Stanford may have a distinctive particle physics course that justifies five years of development and seven figures in development cost. Does it follow that an English or Math course in the typical independent college is or should be distinctive in the same ways? In most cases, the smart move is to develop, outsource, or buy standard course content on a lean model. If you develop in-house, the benchmarks begin at 45 days and $3,500 per course, all-in, and do not go too much higher than that for courses that are not equipment-intensive. If you are ready in terms of infrastructure, this may be a good time to migrate toward a scalable content management system that lowers maintenance costs and, in time, produces a substantial program development dividend. While scalable content represents the future and its economies of scale are high, I recommend a measured approach. In the early phases, migrating to durable learning objects can consume available resources, leaving little for substantive output.

Manage Class Size

Class size accounts for the greatest variance in most margin models. This single variable's high coefficient of leverage is responsible for the "accounting rules only" models that drive classroom size as high as possible. This is unwise in the opposite direction.

We have considerable data on outcomes, operations costs, retention, and student and faculty satisfaction suggesting that an optimum class sizes exists for specific programs and types of students.

Get it wrong and metrics for learning, operational hassles, and margins suffer. Ironically, and counter-intuitively to the accounting mindset, moving off target in either direction reduces program margin. Even though optimum class size is based on variables such as program, faculty and students capabilities, and learning platform and pedagogy, I feel obligated to provide some numbers.  All I can say is that the convergence of our data sets rarely suggests an optimum class size below 12 students and never suggests one above 24-28 students.

Blend Everything

Not all blending results in financial benefits. My comments about blending are limited to a form that contributes to revenue, margin, and academic quality. Generally, this form is referred to as fully blended programming. This form blends every required course in a degree or certificate program. Programs that offer some courses online and others in residence do not capture most of the financial, operational, or academic benefits of blending.

The evidence that blended programs can produce learning outcomes that are superior to either fully online or fully in residence is plentiful and growing.

Given this evidence [of impact on quality], and given the economies of scale and scope and the increased geographical reach possible with blended programs, there are few if any good reasons to offer any courses fully on-ground. This is true even for institutions that do not wish to develop an online presence in the market. This guidance extends to fully online programs as well. Examine the zip code distributions of your fully online students; if the majority of them fall within twice the radius of your expected on-ground service reach, you will be better off - financially and academically - blending all courses in those programs.  

Even within fully blended programming, different forms of blending produce different margins and have different effects on a program's geographical reach. In most cases, the best choice is to blend in a 15/30/10/30/15 (numbers represent percent of total weeks, classes, or workshops) or similar model in which the first, middle, and last classes are in residence (or distributed to remote sites) punctuated by typically longer periods of online collaboration. To be successful, specific instructional elements and curricular goals must be embedded into each of the five segments. For example, the first in-residence segment may focus on meeting and greeting, establishing course objectives and expectations, creating learning teams, and building relationship-oriented task commitments. The middle in-residence period may be devoted to assessing mid-course progress and making course corrections. The last in-residence period is typically devoted to a round robin of reports from horizontal learning teams.

Evaluate the Current Potential for New & Old Programs

You evaluate compensation and operations agreements annually. Do you apply the same standard to evaluating emerging needs for new programs? Are all existing programs still needed? Given the number of new programs each year that did not exist in the previous year, you may be falling behind if you are not adding and phasing out programs each year. I mention this under a discussion of ways to reduce prices because new programs create additional revenue and opportunities for new pricing.

Consider Market-based Pricing

If you are pricing by categories sch as graduate and undergraduate, modulated by a nominal competitive pricing analysis, you are not pricing based on value to the student, and you are probably not pricing based on amortized production and maintenance costs as well as delivery costs. This topic requires a separate and deeper discussion but revenue and margin can sometimes be increased significantly by pricing to the market.

Focus on de facto Rather than Nominal Competition

Like market-based pricing, this topic really requires a separate and deeper discussion. What I offer here is the observation that many institutions make the mistake of anchoring themselves to the prices of programs that are not true competitors.

To be a true competitor, the other school's program must be positioned such that it enrolls students who might otherwise have enrolled in your program, and vice versa.

To get an actionable sense of the pricing market, identify all institutions that offer the same program (e.g., MBA) , then eliminate programs that do not occupy a similar position in the market. To be a true competitor, a program must have similar admissions requirements, delivery methods, time-to-degree, features, location, and branding in the market. If you are not branded with Stanford or MIT, their tuition and fees are irrelevant to you, however much one or two of your instructors might like to believe otherwise.

As always, I welcome your views. Please share your thinking below or send me an email.

Robert W Tucker
President, InterEd, Inc.

In another Executive Briefing I discussed 1906 teaching and assessment practices that dominate today's college classrooms, and why they need to change to accommodate the last five decades of generalizations from learning and measurement sciences.

The prevalence of a pre-scientific mentality is not limited to traditional college classrooms; it extends to MOOCs, an education model that is pedagogically similar to (albeit technologically advanced over) the correspondence schools that used to be advertised on the insides of matchbook covers. To be clear, there is an audience for MOOCS, just as there was an audience for the matchbook correspondence schools. That audience consists of individuals who are self-motivated to learn the subject area and who possess enough in situ competence to progress without the help of a real teacher. These requirements describe five to fifteen percent of those who might want to learn the topic. These percentages are closely reflected in the success rate of MOOCs. For most topics, however, most of us benefit from the services of competent teachers and structured interactions with other students (horizontal learning).

In their present most common form, MOOCs cannot teach or assess what I see as “the more important half” of learning outcomes. Specifically, the horizontal and affective dimensions of learning that are highly correlated with personal and professional success.

I can understand the motives of administrators of online programs to embrace MOOCs. They fit the prevailing online platforms well. I wonder, would the Department of Education or the institutional accreditors approve of a robust and exciting learning environment based on what we now know about how people learn, retain, and apply their learning efficiently? It seems unlikely at present. Anyone who doubts this should reflect on the fact that the Department's IG wants to close online programs that do not conform to his 1906 notions of Carnegie seat time. How does Carnegie align with MOOCs?

Considering a MOOC?

Like placebos, MOOCs are worth considering while they have the ability to do good. Few will argue against the potential for high financial margins.

Should you develop a family of MOOC programs? It depends. Both matchbook and MOOC programs are effective ways to educate individuals who have well-developed self-education skills and who are motivated to succeed in the specific content area. MOOCs succeed - that is, a decent percent of those enrolling complete the course of study - when these two criteria are applied as admissions filters. MOOCs even do a little better than their matchbook counterpart because they embed additional pedagogical tools.

If you have content in mind for an engaged and competent audience MOOCs may be a good idea. If you do not have a good handle on the audience or if you are thinking about self-selected open admissions, keep in mind that your success ratio may be in the range of 5-15%. The 85-95% who fail MOOCs are made up of students possessing a combination of being less motivated, lacking self-study skills, or requiring the services of a teacher.

If you choose an open admissions model, you will want to make an affirmative decision about how much to charge for programs that demonstrate very high failure rates. Perhaps you would choose to charge for transcribing credit rather than for attending the course. However you charge, keep in mind that MOOCs are not appropriate for students who need the services of a professional teacher.

As always, I welcome your views. Please share your thinking below or send me an email.

Robert W Tucker, Ph.D.
President, InterEd, Inc.