Evaluation Baron

Make a Difference: Put Evaluators on Your Education Grant Teams

2010-04-22T06:01:59Z

After Governor Charlie Crist of Florida vetoed SB 6, which was designed to overhaul merit pay for teachers, he convened a group to prepare for the second round of the Race to the Top grant. This group included teachers, superintendents, school board members, members of the PTA, and other leaders. However, a vital member of the team was noticeably absent: a professional evaluator.

Professional evaluators help organizations plan and conduct evaluations of programs, products, processes, policies, etc. In the education realm, evaluators help to determine (among other aspects) whether a program is increasing student achievement, decreasing achievement gaps, increasing high school graduation rates, and increasing college enrollment - all of which are components in the Race to the Top grant program.

When you have a professional evaluator on your team, that person (1) helps you plan and implement techniques to gain visibility as to whether your program is meeting the set objectives, (2) advises you on how to allocate resources where they will bring the most benefit, and (3) provides results, recommendations, and information for you to make informed decisions on how to proceed in making that program a success.

Based on a listing by the American Evaluation Association (AEA), "an international professional association of evaluators devoted to the application and exploration of [evaluation]", there are at least ten different evaluation organizations in the state of Florida that specialize in educational evaluation. I'm sure that one or more of them would be happy to assist Florida in the grant application process.

This should be a wake-up call for all states applying for grants of any kind: if there is an evaluation component in the grant (and there definitely IS in the Race to the Top grants), then you need a professional evaluator to assist you in planning and conducting that evaluation.

How you plan and conduct that evaluation can mean the difference between winning and losing the grant - which equates to vital federal funding that states cannot afford to pass up.

Evalu8 for the Week Ending April 17, 2010

2010-04-15T18:53:22Z

With every problem comes a solution.

Evaluation can help your organization improve existing processes.

Select an evaluator who keeps up with the trends and issues in your industry, and with whom you can discuss viable solutions to what ails your organization.

Think outside the box with regard to evaluating a program, product, etc. There are a number of evaluation techniques that may get at the heart of what you're looking for.

The evaluator should discuss a range of options that provide value to you and to the organization. Be explicit about your information and service needs so the evaluator can readily accommodate you.

Before every project, and throughout the process, ensure that both you and your evaluator are clear on what's being evaluated, your needs and goals, and that you understand the data collection methods and other techniques your evaluator will use to help you.

A proficient evaluator will document not only the costs of the project, but the value you as the client gain for the endeavor.

Before engaging in any project, consider how it will impact your business in the short, intermediate, and long terms.

To be Effective, any Teacher Effectiveness or Merit Pay Evaluation Must be Holistic in Nature

2010-04-15T18:37:45Z

Topics of teacher effectiveness and merit pay have developed from such programs as the NCLB reform and the Race to the Top grant programs. Individual states such as Florida, Colorado, and California, to name a few, have begun to step up to the plate in designing their own evaluation plans for these programs.

If nothing else, the Race to the Top grant program has become a catalyst for change to a more efficient and effective paradigm: that being a more holistic evaluation plan that takes a host of factors into account and puts power into the hands of school leadership to work together with teachers and parents for improved student achievement, a school culture geared toward learning, and a cadre of young people prepared for the post-high school plans of their choice.

An effective evaluation of teacher effectiveness comes not from one test, nor from testing (or any other one factor) alone, but from a comprehensive array of factors. While the only true measure of teacher effectiveness may come from the results of a student's life ten years hence (my 8th grade math teacher comes to mind, who helped students feel valuable and confident in their abilities), schools can and should design a holistic evaluation for teacher effectiveness and merit pay scenarios. These factors may include, but are not limited to, the following (schools may want to focus on different aspects depending on their school's purpose):

Student test scores
Teacher observations and interviews
Teacher self-reports
Teacher portfolios of their work
Parent involvement and interaction

Some argue that there are many factors beyond a teachers' control (such as socio-economic status, broken homes, etc.), and they fear that teachers will be held accountable for those factors. While I am not advocating that teachers be called on the carpet for these situations, there is more to a teacher's effectiveness than simply test scores.

As states and schools wrestle with defining and evaluating teacher effectiveness, they may want to ask themselves the following questions:

What difference are teachers making in their students' lives?
How are teachers preparing students to succeed in and out of the classroom?
Do students develop a love for learning because of their teachers?
Is the school culture geared toward learning, or is it run like a business or an assembly line?
Are teachers rewarded for their efforts in their students' learning, thus giving teachers the motivation to think beyond the textbook?
Are teachers given access to professional development opportunities and other resources to build their skills and increase their ability to help make a difference?
Do school administrators make the time to meet with teachers often to encourage efforts and provide guidance?

If a teacher is being paid to simply exist in the classroom, then we as a society are doing our children a disservice.

Designing and executing a comprehensive evaluation of teacher effectiveness and merit pay will not only provide visibility and evidence of effectiveness, but will pinpoint areas for improvement so that schools may allocate resources where they are most needed. An effective evaluation spans the whole realm of a given program, and directly stems from the missions and goals of an organization. Schools and states would do well to step back and formulate a theory of change (also known as a logic model, impact map, etc.), and then use that information to create long-term evaluation plans to help capitalize on strengths, identify areas of improvement, and develop their capacity to evaluate their programs.

Evalu8 for the Week Ending 3 April 2010

2010-04-04T04:45:44Z

Utilize the problem-solving abilities of the evaluator you work with, not just expertise, as you work together to make your organization more efficient and effective.
Evaluators take their own medicine - they are constantly upgrading their own efficiency, too.
Your evaluator will work with you on a project schedule that works for everyone.
A professional evaluator strives for both quality and value in assisting you in making your business successful.
A professional evaluator should care about your organization enough to at least offer preliminary guidance on your project regardless of whether you are officially contracted with that person for work.
A professional evaluator evaluates the quality of his or her own work (called metaevaluation). Be prepared at the end of a project to discuss the outcome and quality with the evaluator.
Evaluators often serve as advisors to boards of directors. In that capacity, they are able to use their expertise, learn from others, and give back to the community.
Be open to new ideas and possibilities. What may not have worked well in the past may be just the solution to today's needs.

Evalu8 for the Week Ending March 27, 2010

2010-03-23T05:18:30Z

Evaluators who are not only practitioners, but who also conduct research and/or teach, bring a comprehensive wealth of knowledge to your organization, and can use the resulting information for a 360-degree perspective of your project.

Professional evaluators maintain their independence and objectivity in all situations in order to provide you the most comprehensive and focused evaluation about your project.

During your evaluation project, a thorough data analysis ensures synchronization among methods, findings, and recommendations. Make the time to refocus as necessary.

Professional evaluators often team up with other consultants to provide diversity and expertise on an evaluation project.

When starting a project, work with the evaluator to determine the most effective communication mechanisms that will help you access and use information to your advantage.

Focus both on content and information use to help your organization improve.

Combine business process knowledge as well as knowledge of evaluation techniques so the project flows smoother.

When conducting an evaluation, your primary thought should be how whatever you're evaluating fits into the big picture - how it aligns with the mission and goals of your organization.

Evaluation Encompasses the National Education Technology Plan for 2010

2010-03-23T04:38:42Z

The Department of Education's draft of the National Education Technology Plan for 2010 (http://www.ed.gov/technology/netp-2010) has evaluation at its core, with overall goals describing outcomes-based education, the effectiveness and efficiency of educational infrastructure, the need for monitoring and evaluation of progress, and accountability for performance.

What can the Department of Education do to ensure that these evaluation goals become a reality?

Facilitate the process of recruiting and using evaluation experts at the school, district, state, and national levels to design, conduct, and report on the continuous progress of these goals and trends.
Provide generous funding through direct contributions and grant opportunities for the development of core standards and specific learning goals, the creation of realistic activities and assessments directly aligned with those goals, and the monitoring and evaluation of progress and trends associated with these goals.
Ensure that information needs are well defined at each level in the educational hierarchy to ensure data-driven decision-making can take place.
Allow for the collection and analysis of relevant performance data via refined educational evaluation policies.
Facilitate the diffusion of their educational philosophy so that all stakeholders are willing and able to use evaluation results on a continuous basis.

Educators cannot afford to try to handle these processes on their own. Evaluators internal and external to organizations must step in to provide evaluation expertise and to build evaluation capacity within schools and administrations in order to increase program visibility and self-reliance of educational developments.

In this economy, who needs evaluation? You Do!

2010-01-16T05:46:19Z

A common response as I discuss evaluation with people and organizations is that they don't see a real need, or that they cite economic woes as a reason to ignore evaluation. However, here's some of the translation of what they're really saying:

"I don't care how my business is doing. I just want to keep making widgets."

"I don't know a thing about evaluation, so maybe my problems will go away by themselves."

"Evaluation is too much work. I'm busy enough as it is."

The fact is that our current economic state, organizations cannot afford to ignore the need for evaluation. Federal and state governments are focusing on evaluation to root out inefficiency. Non-profit organizations must continually be accountable to funders. For-profit organizations must increase the efficiency of their business processes in order to be ahead of the competition.

All of the above is accomplished by evaluating programs and services to align what should be happening (via criteria and standards) with what is happening. The results may then be used for decision making to make those programs and services more refined, more productive, and more serving of beneficiaries.

Take the time, and make the time to evaluate. The status and growth of your organization depends on it.

The Evaluation Baron, LLC, can introduce you to evaluation concepts and how they will benefit your organization. Contact us for a free initial consultation!

Evalu8 for the Week Ending January 16, 2010

2010-01-16T05:39:18Z

Look for an evaluator who is a clear communicator, and can discuss evaluation in terms you can understand.
Professional evaluators collaborate together at conferences and other engagements, and can provide you with fresh ideas to evaluate your programs.
In accordance with The Guiding Principles for Evaluators, evaluators continuously develop their evaluation competencies through professional development courses or other training opportunities.
Prior to beginning any evaluation activity, consider what the goals are of the program or other area you are evaluating, and what your information needs are for that program.
Specify the best means of communication and reporting with your evaluator to ensure seamless information exchange.
Be clear about your intent to all stakeholders.
Be open and willing to use the results of an evaluation.
Be open to change.

Evaluation is the Handmaiden of Our Global Community

2010-01-09T07:20:42Z

There have been many reports of the growing need for educational reform and its effect on the global economy. Many leaders in education and government are also voicing their concern and stressing the need for accountability, visibility, and evaluation of federal programs (see Obama Administration Launches Program Evaluation Effort at http://www.govexec.com/dailyfed/1009/100709e1.htm). Science, Technology, Engineering, and Math (STEM) education is no exception.

Recent news reports of the reform of STEM education (see The State of Stem at http://www.techlearning.com/article/26400) recommend integrated teaching of the STEM components in order to enhance learning effectiveness. "An intriguing finding was that engaging young people in hands-on projects in engineering and design provides effective ways for them to learn core math and science concepts."

These are some of the benefits of evaluation: to assess the impact and effectiveness of teaching STEM components separate or together; to understand the impact of STEM pedagogy on the global economy, and to maximize the role of teacher effectiveness in this process. Evaluation provides visibility of these vital decision-making tools!

These issues are not without political unrest, though. Nevertheless, teacher unions, administrators, parents, and the students themselves must come together on this and other educational issues. Being able to evaluate and use the resulting information helps individuals, schools, and communities. A first step is to facilitate evaluation of STEM programs in the classroom. Together, we can foster a competitive nation and world (see Universities Pledge to Train Thousands More Math and Science Teachers by 2015 at
http://chronicle.com/article/Universities-Pledge-to-Train/63439/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+chronicle%2Fnews+%28The+Chronicle%3A+Top+Stories%29&utm_content=Google+Reader)
- one teacher, one student, and one classroom at a time. The key is to evaluate!

The Evaluation Baron, LLC can help you in this process. Contact us for a free initial consultation!

Evalu8 for the Week Ending January 9, 2010

2010-01-09T07:15:30Z

Look for an evaluator who is not only competent in what he or she does, but who has confidence in your organization and respective programs - someone who wants to help your organization succeed.
Evaluation is known by many different terms, such as performance improvement, review, and tracking. An evaluator can help explain the concept in terms you can readily understand.
Discuss with an evaluator any ethical concerns you have about a project before proceeding.
Take the time to brainstorm ideas with an evaluator on the optimal approach to an evaluation, and ask a lot of questions to increase your understanding.
Although much evaluation work can be accomplished electronically, be sure to make time to meet in person with your evaluator from time to time to exchange information or ideas, and to discuss the progress of your evaluation project.
Before brushing off a new idea, investigate why that idea didn't work in the past, and how your circumstances have changed.
Learn about what you are evaluating from the start.
Invest in planning.

10 Ways Evaluation Can Help You in 2010

2010-01-03T01:28:58Z

Accrediting schools
Judging the quality of education or corporate training curriculum
Judging the effectiveness of a training program
Improving the recruitment process of volunteers or employees
Improving a commercial product
Improving employee retention
Determining public perception of a new policy or program
Understanding the cost-benefit, cost-effectiveness, or cost-utility of an employee benefit package
Deciding whether to implement a community development program
Developing greater accountability to project funders

The Evaluation Baron, LLC, can help you understand the benefits of these types of evaluations, and what it means to your organization specifically. Contact us for a free initial consultation!

Evalu8 for the Week Ending December 26, 2009

2009-12-26T22:20:01Z

Building trust with an evaluator helps that person understand your needs as an organization and can bring greater objectivity, precision, and usefulness to an evaluation.
Look for an evaluator who brings enthusiasm and life to an evaluation or other project. This helps infuse new ideas for improvement into the organization.
Select an evaluator who is serious about professional development - individually and in collaboration with colleagues.
When contracting for evaluation services, keep in mind the quality and value you receive.
With proper training from an evaluation consultant, your organization can become self-sufficient with regard to your evaluation needs.
Look for an evaluator who listens to your organizational needs, and then makes recommendations as necessary.
Look for an evaluator who keeps his or her professional skills current, to include evaluation, consulting, management, and communication.
Make meetings meaningful.

Evalu8 for the Week Ending December 19, 2009

2009-12-20T06:20:50Z

Work with your evaluator to decide who the stakeholders are and their degree of involvement in the evaluation or project.
Anecdotes or stories are an excellent way to get your ideas across with an evaluator and with your staff.
Consider subscribing to various consultants' websites for tips on evaluation, organizational development, or other topics.
An evaluator makes recommendations; you as the stakeholder and client choose to accept or reject those recommendations to any degree.
The evaluation reporting process is flexible to accommodate your organizational needs, so do not hesitate to discuss with the evaluator a particular reporting style or protocol that you prefer.
Discuss with the evaluator various consulting experiences as they pertain to your organization to get an idea of philosophy, work style, and usefulness.
Look for an evaluator who brings value to your organization - someone who helps translate information into knowledge you can use to be more effective in your business.
Look for an evaluator who will provide useful recommendations you can implement immediately to make a difference in your business.

Evalu8 for the Week Ending December 12, 2009

2009-12-13T04:59:44Z

Ask "What if" questions about your program to know how to evaluate relevant aspects of it.
When evaluating productivity and development, consider professional development as an investment in your organization's future.
Make sure that people can identify with your program (evaluate that aspect).
Don't be afraid to act on the results of an evaluation regardless of the outcome of that evaluation.
Learn to ask "Why" as a problem-solving tool for your program. Asking why is the first step in evaluation.
Set realistic expectations when designing an evaluation.
Coordinate with the evaluator regarding your reporting protocol or preferences.
Keep up with reading in your profession, as well as on evaluation of aspects of your work. An evaluator can recommend books or articles to peak your interest and inform your business.

Evaluation is the Key to Education Reform

2009-12-05T03:18:48Z

A recent news article from the Christian Science Monitor, "Eyeing Stimulus Money for Education, States Adopt Reforms (posted Thursday, December 3, 2009 at http://www.eschoolnews.com/funding/funding-news/?i=62081), highlights the need for evaluation in making progress with education reforms.

In particular, the article points out four areas where states could receive grant money: "Establishing data systems that track students' progress from preschool through college, developing and using rigorous standards and assessments, improving teacher effectiveness and the distribution of high-quality teachers, and turning around the lowest-performing schools." The article states that the last two issues have not been addressed much because of the obscure nature of defining teacher effectiveness and developing a system to improve schools.

That's what evaluation is all about. Organizations work with professional evaluators to define topics such as teacher effectiveness, establish criteria and standards for what should be happening, and then take steps to evaluate whether or not those things are happening. Organizations may then choose to implement recommendations generated from the study to enhance and improve the program.

While creating standardized definitions, criteria, and standards for schools across the country will certainly be a more involved process with all levels of education, schools can and should start somewhere by evaluating what is working or not working in school reform (to at least establish a baseline), and then proceeding with strategies for improvement.

National, state, and local organizations must work hand in hand in order for true reform to succeed. Schools would do well to look at this grant offering as an opportunity to evaluate their current situation and then take steps toward improvement.

The Evaluation Baron, LLC can help you evaluate how your school, district, or state is doing in any of these topics. Contact us for a free initial consultation!