Evaluation Baron

Evalu8 - 3/24/2009

2009-03-25T04:27:11Z

Never begin an evaluation without a thorough needs assessment.
Identify and involve all stakeholders in the evaluation to the extent possible.
Explore different evaluation options according to the needs for the evaluation.
Do your homework before the evaluation begins.
Conduct the evaluation to the highest technical standards possible
Don't be afraid to consult a third party for guidance.
Seek an external evaluator with the education and experience corresponding to your organizational needs.
Evaluations are not guaranteed to have a certain result.

What measurable metrics are worth monitoring in online learning environments?

2008-09-09T20:57:10Z

So what type of areas or topics can web analytics assist you with? Here are some examples from the online learning field:

Ensuring class members complete online assignments
Discovering access or informational problems with a website
Finding out if others are benefiting from the information on your website (with reference to open learning as applicable)
Obtaining feedback from users regarding content, usability, and applicability - and being able to make adjustments accordingly
Assessing the need for certain courses or topic areas
Learning who the most frequent users are, why they participate, and how they can propel your courses to success

How do evaluators and their clients benefit from web analytics?

2008-09-09T20:54:52Z

Web analytics, an up and coming technology that evaluates website activity, can be a huge benefit to all organizations, whether you have a small website or a large e-business. Here's how web analytics can improve your organization:

Find out (among other vital data) who is on your website, for how long, and what they look at most
Based on the above information, you can cater your website you specific groups of users and you can provide them the vital information that may mean the difference between having them do business with you and having them go elsewhere
External evaluators (i.e., those who help you make sense out of the web information), can learn more about how customers and potential customers feel about your organization so they can make accurate and appropriate recommendations for improving your organization.

Conducting Evaluations in an Academic Support Environment

2008-06-11T04:53:41Z

During the past few years, I've had an amazing experience conducting evaluations with an academic support group on the Brigham Young University (BYU) campus. The Center for Teaching and Learning (CTL) strives to “improve student learning in the context of the Aims of a BYU Education. The center accomplishes this goal by assisting BYU faculty, instructors, and administrators to -

refine effective program- and course-level learning outcomes,
design learning experiences to achieve desired outcomes,
enhance the quality of learning experiences through technology integration,
evaluate the effectiveness of learning experiences, and
advance knowledge of effective learning and teaching.”

]]> As a member of the evaluation team at the CTL, I worked with instructional designers, quality assurance experts, teaching & learning consultants, as well as other evaluators to help faculty members ensure that whatever product or service they wanted to create (e.g., a neuron simulation for a biology class, an instructional DVD for a marching band) was aligned with their learning outcomes for the students and helped students to learn more and to have fun while they learn.

The team environment was exceptional. As an academic support unit, the CTL is the right arm of the university, and constantly strives to be in tune with the academic needs of the faculty. As a member of an evaluation team, we all worked together to not only resolve evaluation issues, but to enhance our own evaluation skills via professional development sessions and references. Evaluators stick together and help each other to continuously improve.

I would recommend to any evaluator to seize the opportunity to collaborate with other evaluators, whether in an academic or business setting. It's a win-win situation when you do.

Evaluation vs. Research: What's the difference?

2008-06-11T04:45:42Z

The evaluation field in general loves to debate and discuss the important issues surrounding evaluation. One longstanding topic (if there is anything more longstanding than the quantitative/qualitative debate :) is the difference between evaluation and research.

There are many opinions, and there probably is no right or wrong answer, but since this is my blog, I get to state my opinions and experiences to my heart's content. :)

As I understand it, both research and evaluation ask what is. Then research asks why, which indicates a theoretical basis, whereas evaluation asks what should be, which makes reference to the standards and criteria of evaluations.

]]> That is not to say that evaluation never asks why, or even that research never asks what should be. However, in the evaluation field, we are making a value judgment about the merit or worth of a program, product, or process. We find out what the situation is (i.e., the current state, if you will, of the evaluand), what the criteria (i.e., what should the evaluand be doing) and standards (i.e., what is our level of acceptability for the criteria) are for the evaluand, and then we judge the worth of the evaluand based on those criteria and standards.

Research, on the other hand, seeks to understand what the situation is, but then probes into why the situation is that way and how we can better understand the situation with regard to developing, expanding, and testing various theories.

In the evaluation field, both evaluation and research are necessary to further the work and to train evaluators. The area of evaluation research in particular deals with conducting research about evaluation (as opposed to conducting evaluations). We seek to understand why evaluations happen (or do not happen) in particular ways. We develop evaluation theories (although some choose to refer to them as models) to try to explain these occurrences, and continuously test these theories in different environments.

Evaluation and research form a symbiotic relationship. Evaluators, in turn, seek to understand and learn from both.