Forensics for Recruiting

The result of this TV series has been an explosion in enrollment in criminal justice and law enforcement related programs at colleges and technical schools across the country.  The outcome will be a flood of graduates imbued with knowledge and skills, hopeful about being hired by leading edge crime fighting police departments who want more science in their prosecution.  In recruiting, hiring decisions should be based upon evidence supported with sound data collection and analysis too.

A similar science-based approach for establishing decision support evidence is available for recruiters: Industrial/Organizational Psychology (I/O Psych).  These professionals are the CSI: Recruiting specialists – Candidate Science Investigators

This professional discipline was established in the early 20^th century, to apply psychological principles and techniques to business and industrial problems, as in the selection of personnel.  Before crime investigator forensics hit prime time TV, forensic specialists worked quietly behind the scenes improving the quality of data collected and used to build a case.  Industrial/Organizational Psychologists (I/O Psych) are quietly at work around the world, building data collection methods that improve the quality of  hire  with selection science.  These professionals design the data collection methods called assessment.

Every law department has someone trained in forensics because they know better evidence improves the decision in our justice process.  Will it take a fast paced TV series before companies sit up and take notice of the work being done by CSI Recruiting?

Doctorate degrees have been offered in I/O Psych for over 80 years.  These graduates have been snapped up by leading edge companies who understand the competitive advantage of more science in their recruiting process.  The evidence of I/O Psych’s contribution is compelling.

Here are a few simple examples from our work with selection science and HR Analytics that shed a bit of light on to the potential of better candidate data.

A retail operation with thousands of stores had been using a candidate screening criteria of “years of experience in a similar industry.”  Intuitively, every one thought it had made sense.  However, during a recent study by a team of I/O Psychologists, evidence determined that the longer a candidate had been in the similar industry, the less likely they were to be an above average contributor at this firm.

Similarly, in a capital equipment field sales representative position, the company had established screening criteria of hiring people who had worked for the major competitors.  After an investigation by a team of I/O Psychologists, the evidence demonstrated the longer a candidate had worked for the competition, the less likely they were to be an above average performer.

Ironically, “related experience” is often a candidate screening criteria.  In both cases, using that factor places positive weight on a evaluation criteria with negative value.  The CSI Recuiting teams had been chasing bad leads, intuitive assumptions, proved wrong by evidence.

In both cases, the evidence collection process was dramatically improved across a broad range of factors that contributed to a better quality of hire through CSI.

More examples can be found on our case study page.

Crime scene investigation is typically more like making one high stakes hiring decision.  The consequence of the decision can be significant.  The approach used to leverage I/O psychology varies depending on the scale of the staffing process.  A once every few years hiring decision requires a different solution than staffing processes which make hundreds or even thousands of hiring decisions every year.

If your company has a job or job family with over 100 employees in it, I/O psychology can begin to add measurable impact on performance with each hiring decision.  If you have a job with thousands of employees engaged in fundamentally the same work, it could be the basis for a charge of Recruiter Negligence for not engaging an I/O psychologist.  In high volume hiring processes, the size of the data set, the frequency of decision making, and the potential for significant performance variation almost mandated Candidate Science Investigation.  Click here to review some criteria to consider to determine if CSI is appropriate for your recruiting situation.

The decision to hire will always be an act of personal judgment.  However, every executive knows a decision is only as good at the data behind it.  There is a great opportunity for staffing process owners to do real CSI: Candidate Science Investigation.  Without the use of selection science, data collection and analysis, recruiting could be activity without insight.

Watch our movie to learn more.

Three Shaker Consulting Group psychologists discuss the power of combining simulations with other assessment data for more predictive power.

Throughout the history of psychometric assessment, researchers have focused on discrete measurement modalities such as multiple choice biographical data, scale-based personality, and various types of mental ability measures. Researchers and practitioners have generally studied each modality individually, and attempted to show that individual constructs have a degree of reliability and correlate significantly with some outcome measure. As sample sizes and, correspondingly, statistical computing power have grown, researchers have also been able to examine complex effects such as interactions between constructs, though even these analyses are most often done within a particular assessment modality (e.g., examining the interaction between two personality scales).

Society for Industrial Organizational Psychology

Now, as computing and networking technologies continue to evolve at an exponential rate, nearly any type of assessment modality imaginable is possible to study and implement (Greene, 2011; Trull, 2007). As a result, novel assessment techniques are being developed faster than ever before. Today, simulations are at the forefront of modern assessment design and, as work samples, are known to be quite predictive of job performance (Schmidt & Hunter, 1998). Yet, as a category they are quite heterogeneous.

This presentation will contribute to the growing literature in this area by examining two aspects of simulation design:

1. Whether measuring respondent behaviors during the simulation rather than just responses to embedded questions might confer a psychometric advantage.
2. How simulation measures may interact with other types of assessment measures, to better predict various outcomes.

Ultimately, as computerized simulation of the real world increases in complexity and fidelity, our science can move from creating measures of theoretical constructs to measuring actual behavior. In this manner, in time we may begin to transcend the need to measure proxy variables and theoretical constructs, and instead, use more direct measures of behavioral outcomes.

Simulations offer us a venue to ask many new questions that were not feasible and/or informative before. What can we learn, for example, by observing how a person interacts with an environment, be it virtual or real? For example:

• What do errant mouse clicks tell us?
• What do we learn when a respondent changes her answer?
• Can we learn something about a candidate’s personality by whether she skips over questions she doesn’t know, or spends extra time trying to get each question correct?
• Does repeating an example item during simulation instructions tell us anything about an individual’s personality?

In short, by simulating environments, psychologists can expand the measurement space from the traditional and simple question and answer, to a broad range of non-question-based behavioral measures. As simulations grow in complexity and fidelity, we may be able to move away from actual questions, and more towards measuring how an individual actually interacts with a particular environment; in other words, measure that individual’s choices and behaviors (Hornke & Kersting, 2006). At that point, our measurement focus will be on anything and everything that can be measured, from how a person moves through a space to what they actually do after making a decision.

While one simulation-based measurement advance is thus a gradual moving away from construct measurement to direct behavior measurement, another advance occurs as practitioners combine simulations with other assessment types. This allows researchers to measure interaction and other complex effects across measurement modalities. For example, we may find that a person with a high score on a multitasking simulation is better at a certain job, but only if that person also has a high score on attention to detail. Or, perhaps we will find that multitasking scores are only predictive for people who have a background doing work that involves multitasking. This type of cross-modality research is in its’ infancy, but it should increase as researchers integrate more measures in to online assessment experiences.

Psychological science has a hundred-year history of theory and research, but we feel that it has barely begun to tap its’ potential to explain and predict human behavior. Traditionally, researchers in our field focus on finding that elusive, “holy grail” measurement construct that can predict uncharted amounts of variance in job performance. We believe that this pursuit is ill-fated. Instead, researchers should explore ways to measure human characteristics with increasing fidelity, and in particular, move beyond paper-and-pencil measures of abstract constructs to actual direct behavioral measurements, especially in virtual environments. Furthermore, and especially with the vast amounts of data increasingly being captured by modern organizations, our science needs to focus less on the direct effects of measured variables, and more on complex and cross-modality effects. The latter topic represents a tremendous area of untapped opportunity for exploration.

References
Greene, R. L. (2011). Some considerations for enhancing psychological assessment. Journal of Personality Assessment, 93(3), 198-203. doi:10.1080/00223891.2011.558879
Hornke, L. F., & Kersting, M. (2006). Optimizing Quality in the Use of Web-Based and Computer-Based Testing for Personnel Selection. In D. Bartram, & R. K. Hambleton (Eds.), Computer-based testing and the Internet: Issues and advances (pp. 149-162). New York, NY: John Wiley & Sons Ltd.
Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124(2), 262-274. doi:10.1037/0033-2909.124.2.262
Trull, T. J. (2007). Expanding the aperture of psychological assessment: Introduction to the special section on innovative clinical assessment technologies and methods. Psychological Assessment, 19(1), 1-3. doi:10.1037/1040-3590.19.1.1

Now Serving East Coast

Dr. Christie Cox, a Maryland resident, recently joined the firm and opened the new office in February. Shaker Consulting Group’s other offices are located in Atlanta, and Pittsburgh, with headquarters in Cleveland.

“The growth of Shaker Consulting Group’s international client work was very interesting to me. I will be able to immediately leverage my global pre-employment testing experience,” said Dr. Cox, an industrial organizational psychologist.

First, I am honored by how my thinking and writing inspired a great article that triggered some thoughtful dialogue. The comment thread was interesting.

Thank you Balazs and Jillyan.

Candidate Experience Award 2012

Second, metrics are for those who want to understand the impact of an experience, and, make adaptations based upon learning from experience.  Do not think of it as a  SHOULD we measaure, but more of a  WHAT should we measure? And WHY?

Third, with candidate as customer, does a “net promoter score” (NPS) type evaluation tell you what you need to know?  Do you ask all your candidates for feedback?  What to they say?  What percent take you up on your invitation to provide feedback.  Here are some examples of what candidates say after a virtual job tryout.

Lessons I learned from the early days of the USA quality movement include: the customer defines quality and if you don’t measure it, you can’t manage it.  Even if YOUR set the standard for quality, you are compelled to measure to see if you are delivering to your standard.

Fourth, there is a growing community seeking ways to advance and improve the candidate experience.  Read more about that initiative here.  Or click the picture to take you there.

I interviewed Gerry Crispin on the candidate experience several years ago. Listen to his clear message here.

Jim Gilmore, one of the authors of The Experience Economy and I recently met to continue our dialogue on the candidate experience. Watch for my video interview blog with Jim, to be posted here in the near future. He offers some interesting things for our collective consideration.

When trying to scale up to fill hundreds, or thousands, is that format reasonable?  Well maybe not, but it’s close cousin approach certainly does.

Request a Work Sample

The Virtual Job Tryout invites candidates to complete a range of work samples in an on-line experience.  Fully branded and chock-full of realistic job preview, the Virtual Job Tryout delivers a very similar experience to every candidate, not just a few finalists.  The win here is that finalists are determined by the results of their creative capabilities and efforts to address job relevant challenges and situations.  After all, this is the experience economy.  So, why not deliver an amazing candidate experience?

Over 90% of candidates state this type of experience better prepared them to decide if the job was right for them.  Over 90% of candidate state they will refer others to apply due to the unique and engage format of the application.  This is a form of candidate experience that delivers a brand-positive impression, adds more objective and better candidate data into the evaluation process and can create a differentiated workforce that delivers superior results.

Jennifer found a number of companies seeking better ways to evaluate capabilities and determine job-fit.  Getting a work sample makes a lot of sense.  If you have significant hiring challenges, a process that is scalable may make sense for you.  Check out a few criteria that can help you determine if this approach might be a right-fit for you.

What is Average?

Let’s begin with the concept of average.  Webster offers this definition:   The result obtained by adding several quantities together and then dividing this total by the number of quantities .  In the context of recruiting, I prefer to describe it as the result of mistakes that occur while trying to only hire top talent.  Your worst hires pull your average down.

William Scherkenbach, noted  author and former Director of Statistical  Methods for Ford Motor Company in his book The Deming Route to Quality reminds us that, “if  you believe in the law of averages you will always have above average and below average, and there is  not a darn thing you can do about it.”  What that means in the simplest of terms is that half of your candidates are below average.  All the time, every time.  Think about that.

So what is an above average candidate?

The law of averages surfaces as variation in your process.  You hired your best, and you hired your worst, using the same candidate evaluation methods.  Average is a place on the continuum between the two hires that bookmark the ends.

When we apply this to a candidate population two question emerge:

1. What is the average?” and,

2. How does the average in the candidate population compare to the average in the current employee population?

At Lake Wobegon, the children are all above average.  But, above average on what?  And Scherkenbach would balance that with an assertion that the children in the sister city, Lake Whoa-is-me are all below average.

Quality of hire carries with it a notion that there is one or more metrics against which a candidate can be evaluated.  Enter productivity metrics, competency models, KSAOs, success criteria, performance frameworks, etc.  These methodologies for describing and evaluating on-the-job behaviors of current employees become the metrics for evaluating candidates as well.

Scherkenbach goes on to assert the real opportunity is to know what your average is and continuously raise it.  This is where the notion of above average really comes into play.  It is possible for the best candidate to be below average, when compared to the average of current employees.  And hiring the best candidate from a below average pool, lowers the average of the current employees.  Dwell on that for a moment.

Defining Average

It is possible to conduct an analysis that determines average or actually a variety of averages within existing employees.  Data about their performance exists or can be created.  Objective measures of performance can be used to collect data on a group and calculate an average level of productivity.  Supervisor ratings of observed performance can be collected using behaviorally anchored rating scales. This data can be used to calculate the average of each competency as well.  These two data sets comprise the metrics for quality of hire.

Quality of hire metrics become the standard for differentiating among candidates.  How then, does one gain insight into a candidate’s ability to achieve certain objective measures of performance?  And, how does one ascertain a candidate’s capacity to deliver behaviors similar to those defined in a competency model?

Over time, new hire performance can be evaluated using the same criteria, objective results and competency ratings.  When that data is collected, a quality of hire evaluation can be documented and reported.  Muse over the value of that.

Proxy Measures and Evaluating Average

Most forms of candidate evaluation are proxy measures. A proxy measure is a substitute or surrogate measure.  The most obvious and common proxy measure is level of education.  A diploma or degree is often set as a threshold measure of some functional level of literacy.  Assumptions are made about basic reading writing and reasoning levels associated with each level of academic attainment. I wonder if Garrison Keillor is referring to above average high school achievement in Lake Wobegon?

No doubt you have seen two individuals with similar academic credentials but with very different levels of literacy.  This is an example of where a proxy measures fail to do its job.  At issue here is the variation that exists in the proxy measure, or how abstract the proxy measure is in relation to the quality of hire metric.

Behavioral interviewing is a form of proxy measure that assumes story telling about past behaviors relate to what we might expect in terms of behaviors in the future.  And there is a large degree of truth to that.

There are two inherent challenges in achieving an effective evaluation with interviewing.  One is interviewer skillfulness at probing and documenting responses in a useful manner. The second is a candidate’s ability to articulate how they accomplish results.

Beginning with thoughtfully constructed questions that elicit job-relevant examples of past efforts is considered a best practice.  Recruiters who stay on-script with competency-based, behavioral interviews do get a pretty thorough evaluation of candidate-job fit.  Candidate responses can be rated against evaluation criteria and ratings can be used to determine if the candidate is below, at, or above average.  Unfortunately only about 35% of recruiters stated they apply this level of rigor to interviewing practices. (Ask for a copy of the survey white paper)

There are more reliable and direct ways to evaluate and indentify above average.

Direct Evaluation of Average

The closer the evaluation exercise is to actual job demands, the more accurate it can be in assessing capabilities in relation to an average.  That is precisely the role of simulations for pre-employment assessment.  Job specific simulations present the candidate with work scenarios and job-relevant work exercises to capture data on how an individual actually handles a range of job demands.

Having a large group of current employees complete a simulation captures the data to document in-house average.  With these two data sets it is easy to differentiate among candidates.  Candidate results on the simulation produce a score that can be compared to other candidates and to existing employees.  The score on a simulation can help identify candidates with above average capabilities.

Having all applicants for a job complete a simulation provides the data to document candidate pool average, as well as the variation from low to high.

Handsome, Good Looking and Above Average

Garrison Keilor sends the listener on their way with a closing comment about characteristics of the entire population at Lake Wobegon.  I’d like to send you off with a few closing comments about characteristics of above average recruiting practices.

You can invest your interview time and effort with above average candidates.  But you must first invest in the rigor of defining the metrics, creating an objective evaluation method and calibrating it with your existing population.  Using off-the-shelf evaluation resources may be a good step in the right direction.  However by default that implements a ‘me too’ approach to candidate evaluation when in fact you may be working hard to create a workforce that is distinctive.  And may not contribute to a competitive advantage.

On page 10 of  The Differentiated Workforce, authors Beatty, Becker and Huselid provide a visual model for considering the need for and strategic impact of job specific, company specific workforce practices.  One might draw a sound conclusion that certain jobs in your organization demand an extremely rigorous candidate evaluation experience.

Calibration is the business term for validation analysis.  Validation analysis documents the relationship among competency ratings, objective performance metrics and simulation results.  Validation is the measurement rigor that links candidate evaluation to your business drivers.

Recruiting departments that us HR Analytics and conduct in-house validation analysis go beyond above average, they ensure their efforts create aworkforce that delivers superior results.  That by itself makes them pretty good looking to the CEO.  If you want to do that, we can help.  Let’s think about that together.

Want Hire Power? - Get and App for That!

The current dynamic business environment is spurring many companies to look to at how their employee selection system contributes to their strategic plan. The event will connect HR practitioners from prestigious business, with local and national thought leaders offering insight into effective talent management practices

Attendees will learn how Shaker Consulting Group deploys their Virtual Job Tryout across a wide variety of industries and job families to achieve staffing process improvement.

The event is expected to draw more than 500 human resource professionals. Attendees will learn strategies to align their HR practice such as the staffing process improvement with their strategic objectives.

Shaker Consulting Group will be located at booth #407 in the Exhibit Hall. Ask for a copy of our presentation Three Paths to Return On Investment (ROI).

Take a few minutes to explore a demonstration and learn how the Virtual Job Tryout combines realistic job preview (RJP) and pre-employment testing in a company branded message.

To arrange a private meeting or obtain additional information please email Joe Muprhy, Vice President, at joe.murphy@shakercg.com.

Got Data?

Lance Haun of TLNT wrote a great article about Moneyball.

What Moneyball brings to light is the same discipline I/O psychology brings to staffing process improvement.  In Moneyball the various qualifications of the candidate pool are supported by rigorous data collection and HR analytics.

ATS profile questions often gather proxy data.  Interesting but not valuable data.  Careerbuilder recently posted the average ‘Look” at a resume was less than 2 minutes.  How much hard data was collected?  How much data was entered into a data base for analysis?  What scoring algorithm was used to drive the Yes, Maybe No sorting process that took place.

Proxy data are the easy to capture but often substitutes for evaluating the underlying trait or characteristic.  In Moneyball, a scout asserted one player lacked confidence. The proxy measure used was the arbitrary rating the scout attributed to the player’s girl friend.  Well intended recruiters deploy proxy evaluations with little or no substance behind them. I’d be happy to describe a few we have helped organizations debunk – just ask.

It is common for a robust validation analysis to capture and analyze 250,000 data points to examine the relationship among candidate evaluation data and on-the-job performance.  Companies that engage in this level of analysis create a workforce that delivers superior results.  Better candidate data supports better decision making.

One of the best lines in the movie was the job offer scene in Boston.  It went something like this, You got just as many wins as NY on 25% of the salary.  Anybody who does not take note of that will be watching the Series from their sofa.

Footnoted – Boston went on to win the Series two years later.

One of the hardest lessons to learn from Moneyball was that it is not about hiring the superstar.  Quality of hire is all about consistently and objectively raising the average.

Stop in and talk about this at our booth at ERE in San Diego. I will have a gift for you if you mention this post.

“Her experience in global assessment implementation and HR analytics offer great value for supporting our multi-national clients. Her expertise in pre-employment assessment design and HR analytics will prove invaluable as we expand our client base in Asia and Europe,” said Joseph P. Murphy, vice president of Shaker Consulting Group.

Cox, a native of Virginia, with a Ph.D. in Industrial/Organizational Psychology, brings a unique mix of HR Analytics and global implementation capabilities to the firm.  She is a true selection scientist.

For more information, read the full release: Shaker Consulting Group Hires Dr. Christie Cox to Support Virtual Job Tryout with Global Clients.

Bruce begins by asking us to define the issue.  You can ask “How do I reject the 99%?” and get some clear methods to send a message stating you did not get the job.   However, with the question “How can I build a valuable community?” you get a very different set of answers.  Here are the four themes Bruce invites us to explore.

1. Rethink your talent pipeline – get out of the transaction mindset
2. Deploy community building resources – share information
3. Think long-term relationship – consider fit for the future
4. Use incentives to convert candidates to sourcers

Rethink your talent pipeline – get out of the transaction mindset

Just-in-time was an inventory management strategy to reduce assets in the distribution channel.  Just-in-time sourcing puts candidate assets into the talent distribution channel much like the proverbial pig in the python.  In hard-goods distribution channel, the feeding is transitioned to small bites.  In recruiting, we not only still serve pig-sized servings, the pig is never digested.  Applicant tracking systems (ATS) are chock-full of every candidate that ever applied.

For some reason, fresh candidates must seem better.  Bruce suggests we think about, and actively engage the inventory in our ATS.  Look back, look inside and label the best candidates that did not get hired.  Keep in contact with them.

Deploy community building resources – share information

The web has proliferated interactive community building resources. Some are extremely simple.  Create an invitation only group on LinkedIn.  Use a communication strategy to foster dialogue, share information about the firm and the future.  Create a candidate experience that goes beyond the resume’ capture.

Think long-term relationship – consider fit for the future

I do not recruit, but I do refer.

A friend called with a ‘think about this’ sourcing request.  About a week later, a young man came to mind that was a client some 20 year previous. It was pre-personal digital assistant/smart phone.  I had an old paper-based phone number in my drawer.  I called and to my surprise, he answered, we spoke, and eventually he took the job in play at that time.

The funny part was that he left the firm where I had his phone number, came back years later and got the same number.  I reconnected by chance.  These days, it would have been easy to keep him in my community.  In filling some jobs you get to meet bonus candidates, those you don’t hire at that time, but just might later.  Create some ties that bind.  Let them know you want to keep the relationship active.  Then live the promise.  That will change the candidate experience for those individuals and enhance your down-stream recruiting success.

Use incentives to convert candidates to source

I have to admit, this one scares me a bit.  Incentives can create unique behaviors.  Getting the right behavior is essential.

We did some quality of referral and quality of hire analysis.  Getting referrals from high performing individuals proved to be a winning formula.  Referrals from candidates with low scores on their pre-employment test tended to also perform marginally on their assessment.  The moral is more than be careful what you ask for.  It expands to Be careful whom you ask for what.  We also perform HR Analytics that examine yield and quality of hire by source.  You may find that of interest as you examine social media links and relationships  of candidates.

The nature of your candidate experience has a lot to do with how actively a candidate will refer others.  The Net Promoter Score (NPS) is often looked to as an indicator of a highly regarded experience.

Every candidate that completes a Virtual Job Tryout is asked for feedback on the experience.  Over 90% state they will refer others to apply, based upon the nature of their candidate experience.

So getting candidates to source for you may not take incentives, just a great candidate experience.  A candidate experience that extends and enhances your brand.

For more information on this topic, read the multi-author Candidate Experience Monograph

Take note of the 2012 Candidate Experience Award.  Application information can be obtained here.

Thanks to Bruce for giving us some food for thought.