Jim Manzi
Chief Executive Officer
Applied Predictive Technologies
(APT)

The retail channel is full of ideas to improve profits, and many of them are about spending money to make money. From building new stores to remodeling existing ones, from buying TV advertising to running in-store promotions, from introducing new products to training new staff, any new program that works across a large network of stores can generate a lot of cash. Unfortunately, the converse is also true – a new program that does not live up to expectations can have a devastating impact on profits if it is rolled out across the network and fails.

The central issue in applying the principles of evidence-based management in these situations is the correct identification of causality. The causal impact of most programs is small compared to natural variation in performance, and the total number of data points is very small. Most correlations determined through historical analysis are spurious, and it is impossible to know when this is not the case.

In 1999, I founded APT to help address this problem. APT helps large consumer-focused organizations apply an evidence-based management process to identify, quantify and exploit causal relationships between management actions and financial outcomes, known as Test & Learn™. It is now being used by a growing number of retailers, consumer goods manufactures and financial services institutions to increase the effectiveness, profitability and speed to market of new programs and capital investments. When I say “Test and Learn,” I am referring to the process of testing an idea in a small number of stores to understand its impact before rollout.

The basic steps of the Test & Learn™ process are not complicated – in fact they’re intuitively obvious:

1. Establish a Hypothesis
2. Design a Test
3. Execute the Test
4. Analyze the Test
5. Plan the Rollout
6. Execute the Rollout

However, when a test is executed in an ad hoc manner, a serious challenge manifests itself at a critical stage of the process. That stage is when it comes time to spend money to rollout the idea. If the executive in charge of that decision is not sufficiently confident that the idea is going to work, he or she simply won’t approve the rollout.

Even more frustrating to management is that there is no way to accurately predict what customers will do in response to a new program before it’s launched. One can execute deep customer research, draw on analogous experience from other retailers, create virtual reality simulations, or consult one’s gut, but no research-based approach can reliably answer the question “what will my returns be if I make this investment?”

The only way to know for sure is to treat the initial implementation of the rollout as a test. A set of stores should be selected that is large and diverse enough to present an accurate cross-section of the network. The performance of these stores should be carefully tracked and compared to the performance of a well-matched control group, to capture the attributable impact of the program’s components, and truly measure the investment returns.

Both the size of the stakes and the complexity of gleaning actionable insights from in-store experimentation grow geometrically as the network grows. Often times these organizations are running multiple tests of important retail ideas at any given time. The ideas tested cross the enterprise, from marketing and merchandising, through operations and HR, to investments in capital improvements.

Up until the last ten years, in-store experimentation was practiced by special occasion only (if at all) by nearly all large network retailers, due to the nearly insurmountable challenges of mobilizing, tracking and analyzing the massive amounts of data required. Over the last ten years, however, the explosion in computing power (and the steep decline in its cost) has profoundly changed the face of retail analytics, allowing large retailers to make the experimentation process both more rigorous and systematic. Today, a best-practice Test & Learn process reliably, consistently and economically answers three questions about any tested store program before rollout:

1. What impact will this idea have on my results – revenues, profits, investment returns, market share, etc.?
2. Will the idea have a larger impact on some stores than others? What is the predicted impact by store and by market, and can I design a rollout program that maximizes returns?
3. What value is created by the individual components of the idea? Can the idea be engineered before rollout to maximize its value?

By institutionalizing a strong organizational process, Test & Learn combines the evidence based management principles of being committed to “fact based” decision making, encouraging experimentation, and utilizing proper testing techniques while decreasing the potential for weak recommendations.

I’ve watched as best-practice Test & Learn practitioners have showcased the value of this style of decision making, generating profit improvements in the tens of millions of dollars per year, rapidly fine-tuning their responses to market developments, and achieving competitive advantage that is difficult for their retail adversaries to see and even more challenging for them to copy. And they’re just getting started…

Andrew V. Abela, Ph.D.
Assistant Professor of Marketing
Dept. of Business & Economics
The Catholic University of America

It’s ironic: practicing Evidence-Based Management often involves presenting evidence, and yet the way we present that evidence frequently itself violates other evidence, evidence about effective presentation design.

Beliefs that only 7% of your message is in what you say and the rest is non-verbal, for example, or that each slide should contain seven bullets of seven words each, are based either on a faulty misreading of the empirical research or are directly contradicted by the research.

Fortunately, there is ample evidence—from research in communications, psychology, marketing, education, multimedia computing, and law—that can be used to establish design guidelines for effective presentations. The overall conclusion of this research is appealing from an EBM perspective: spend less time on decorating your slides and more time on perfecting your content—your evidence. Here are ten principles for evidence-based presentation design: the first three are about eliminating embellishment, and the following seven are about perfecting content.

Less time embellishing your slides

One of the most harmful consequences of the use of presentation software has been the ease with which such products enable users to add adornment to their presentations. Here the research indicates that, almost without exception, this not only wastes time but also hinders effective communication.

1. Eliminate all irrelevant information or images

Several studies have shown that irrelevant images and details reduce audience comprehension and recall—in advertising, for example, irrelevant information weakens consumers’ beliefs about product effectiveness. Even simple (but still irrelevant) elements as 3D effects in graphs can harm communication effectiveness. In general, anything that is not squarely focused on delivering or supporting your message seems to detract from it.

2. Avoid slide transitions

Research indicates that using slide transitions reduces audience attention and agreement. Therefore avoid using fades, wipes, or “cha-ching” sounds when moving from one slide to the next.

3. Use animation only if absolutely necessary

The only time that animation within a slide is useful is when the animation itself provides information that static images could not, such as when you are trying to explain how various complex parts of a process come together. Animation for its own sake is harmful to communication.

More time perfecting your content

To strengthen the effectiveness of your presentation, use the time you saved from not embellishing your slides to refine your argument, details, story, and graphics.

4. Support your recommendations from multiple angles

The evidence here indicates that the more arguments and support you offer in favor of your recommendations, the more likely your audience will be persuaded. Perhaps surprisingly, you should also include arguments against your recommendations: this increases your credibility and protects you from counter-arguments. Lawyers call it “stealing thunder”: bringing up—and responding to—negative arguments before anyone else does significantly weakens those arguments.

5. Provide details

Concrete details make your communication more interesting, memorable, and persuasive. The evidence here is that even the more conceptually oriented members of your audience, who may choose to ignore your details, will nevertheless perceive you to be more credible because you provided details.

6. Explain why your recommendations will work

Experiments show that causal arguments (e.g. “adding this new feature increases customer satisfaction because it provides a sense of control”) are more persuasive than statistical evidence (“adding this feature increases customer satisfaction by 8%”).

7. Weave your content into the form of a story

Stories are easier to remember than lists of evidence, in part because they allow the mind to make more connections, which improves memory. They are also more memorable to the extent that they engage emotions. Under some circumstances, people appear to take information that comes to them in a non-story form and turn it into a story. For example, legal research suggests that juries in criminal trials use a story approach to make sense of the trial data, and there is reason to believe that other audiences do this also. If your audience is going to turn your data into a story, why not help them by doing it for them? This way you will have more control over the story and make your presentation more memorable at the same time.

8. Use graphics extensively

When graphics are included in presentation visuals (such as charts, diagrams, or photographs), then audience recall, persuasion, and positive attitudes to the material all improve.

9. Keep material together

The evidence here is that—contrary to popular belief—breaking complex information down and presenting it in multiple simple pieces is actually less effective than presenting everything at once. This is likely because remembering each piece as it goes by is burdensome. Better to show the entire thing in all its complexity, and explain it. Other research concludes that presenting graphics and accompanying text close together in both space and time improves audience learning. So show a graphical representation of your complex information, place text labels right next to the more important parts of the graphic (this increases memorability), and explain the graphic verbally. For a sufficiently complex diagram, provide printed copies rather than projected slides because this allows you to use much finer detail and smaller type fonts, and therefore keep more of your information together on the same page.

10. Use color for emphasis—and only for emphasis

Other than in color photographs—where the color used is natural—it would seem that the only role for color in serious communication is for emphasis. Research on print advertising found that where color is used to highlight or reinforce particular graphical elements, this tends to improve persuasion. But where color is used just for decoration, then the more likely outcome is that the audience will waste mental effort trying to decode the meaning of each color (“Why is that bar green and the other one blue…? Are they related to the green and blue frame around that text box…?) Black and white slides do not have this problem.

***

When you present evidence, make sure that you use presentation techniques that are themselves supported by evidence.

Selective Bibliography

Introduction

7% claim based on a faulty reading of Mehrabian, A. (1981), Silent messages: Implicit communication of emotions and attitudes. Belmont, CA: Wadsworth; this faulty reading repudiated by Mehrabian here [book description]

Seven bullets of seven words claim based on a flawed reading of Miller, George A. (1956), “The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information,” Psychological Review, 63, 8109; repudiated by Miller in a 1998 email to Mark Halpern.

1. Eliminate all irrelevant information or images

Bartsch, Robert A. and Kristi M. Cobern (2003), “Effectiveness of PowerPoint presentations in lectures,” Computers & Education, 41 (1), 77-86.

Edell, Julie A. and Richard Staelin (1983), “The Information Processing of Pictures in Print Advertisements,” Journal of Consumer Research, 10 (1), 45.

Fischer, Martin H. (2000), “Do irrelevant depth cues affect the comprehension of bar graphs?” Applied Cognitive Psychology, 14 (2), 151-63.

Mayer, Richard E. (2001), Multi-Media Learning. Cambridge, UK: Cambridge University Press.

Meyvis, Tom and Chris Janiszewski (2002), “Consumers’ beliefs about product benefits: The effect of obviously irrelevant product information,” Journal of Consumer Research, 28 (4), 618.

Moreno, Roxana (2006), “Learning in High-Tech and Multimedia Environments,” Current Directions in Psychological Science, 15 (2), 63-67.

Slykhuis, David (2005), “Eye-Tracking Students’ Attention to PowerPoint Photographs in a Science Education Setting,” Journal of Science Education & Technology, 14 (5/6), 509-20.

2. Avoid slide transitions

Vogel, Doug and Joline Morrison (1998), “The Impacts of Presentation Visuals on Persuasion,” Information & Management.

3. Use animation only if absolutely necessary

Tversky, Barbara, Julie Bauer Morrison, and Mireille Betrancourt (2002), “Animation: Can It Facilitate?” International Journal of Human-Computer Studies, 57, 247-62.

4. Support your recommendations from multiple angles

Allen, Mike (1998), “Comparing the Persuasive Effectiveness One- and Two-Sided Message.” In Persuasion: Advances Through Meta-Analysis, eds. Mike Allen and Raymond W. Preiss. Cresskill, NJ: Hampton Press.

Pechmann, Cornelia (1992), “Predicting When Two-Sided Ads Will Be More Effective than One-Sided Ads: The Role of Correlational and Correspondent Inferences,” Journal of Marketing Research, 29 (4), 441.

Petty, Richard E. and John T. Cacioppo (1984), “The effects of involvement on response to argument quantity and quality: central and peripheral routes to persuasion,” Journal of Personality and Social Psychology, 46 (1), 69-81.

Williams, Kipling D., Martin J. Bourgeois, and Robert T. Croyle (1993), “The effects of stealing thunder in criminal and civil trials,” Law and Human Behavior, 17 (6), 597-609.

5. Provide details

Artz, Nancy and Alice M. Tybout (1999), “The Moderating Impact of Quantitative Information on the Relationship Between Source Credibility and Persuasion: A Persuasion Knowledge Model Interpretation,” Marketing Letters, 10 (1), 51-63.

Fernandez, Karen V. and Dennis L. Rosen (2000), “The effectiveness of information and color in yellow pages advertising,” Journal of Advertising, 29 (2), 61.

Morgan, Susan E. and Tom Reichert (1999), “The message is in the metaphor: Assessing the comprehension of metaphors in advertisements,” Journal of Advertising, 28 (4), 1.

Rossiter, John R. and Larry Percy (1980), “Attitude Change Through Visual Imagery in Advertising,” Journal of Advertising, 9 (2), 10.

Walker, Ian and Charles Hulme (1999), “Concrete Words Are Easier to Recall Than Abstract Words: Evidence for a Semantic Contribution to Short-Term Serial Recall,” Journal of Experimental Psychology, 25 (5), 1256-71.

6. Explain why your recommendations will work

Slusher, Morgan P. and Craig A. Anderson (1996), “Using Causal Persuasive Arguments to Change Beliefs and Teach New Information: The Mediating Role of Explanation Availability and Evaluation Bias in the Acceptance of Knowledge,” Journal of Educational Psychology, 88 (1), 110.

7. Weave your content into the form of a story

Cahill, Larry, Ralf Babinsky, Hans J. Markowitsch, and James L. McGaugh (1995), “The amygdala and emotional memory,” Nature, 377 (6547), 295.

Kazui, Hiroaki, Etsuro Mori, Mamoru Hashimoto, and Nobutsugu Hirono (2003), “Enhancement of Declarative Memory by Emotional Arousal and Visual Memory Function in Alzheimer’s Disease,” Journal of Neuropsychiatry and Clinical Neurosciences, 15 (2), 221-26.

Mallon, Bride and Brian Webb (2000), “Structure, causality, visibility and interaction: propositions for evaluating engagement in narrative multimedia,” International Journal of Human-Computer Studies, 53, 269-87.

Pennington, Nancy and Reid Hastie (1991), “A cognitive review of juror decision making: The Story Model,” Cardozo Law Review, 13, 5001-39.

—- (1992), “Explaining the Evidence: Tests of the Story Model for Juror Decision Making,” Journal of Personality and Social Psychology, 62 (2), 189-206.

8. Use graphics extensively

Benbasat, Izak and Albert S. Dexter (1985), “An Experimental Evaluation of Graphical and Color-Enhanced Information Presentation,” Management Science, 31 (11), 1348-64.

King, Wesley, Marie Dent, and Edward Miles (1991), “The Persuasive Effect of Graphics in Computer-Mediated Communication,” Computers in Human Behavior, 7.

Rossiter, John R. and Larry Percy (1980), “Attitude Change Through Visual Imagery in Advertising,” Journal of Advertising, 9 (2), 10.

9. Keep material together

Mayer, Richard E. (2001), Multi-Media Learning. Cambridge, UK: Cambridge University Press.

Moreno, Roxana (2006), “Learning in High-Tech and Multimedia Environments,” Current Directions in Psychological Science, 15 (2), 63-67.

Myers-Levy, Joan and Sandra A. Peracchio (1995), “Understanding the Effects of Color: How the Correspondence between Available and Required Resources Affects Attitudes,” Journal of Consumer Research, 22 (2), 121.

Nadolski, Rob J., Paul A. Kirschner, and Jeroen J.G. van Merrienboer (2005), “Optimizing the number of steps in learning tasks for complex skills,” British Journal of Educational Psychology, 75, 223-37.

10. Use color for emphasis—and only for emphasis

Christ, R.E. (1975), “Review and analysis of color coding research for visual display,” Human Factors, 17, 542-70.

Fernandez, Karen V. and Dennis L. Rosen (2000), “The effectiveness of information and color in yellow pages advertising,” Journal of Advertising, 29 (2), 61.

Kelly, K. J. and R. F. Hoel (1991), “The Impact of Size, Color, and Copy Quantity on Yellow Pages Advertising Effectiveness,” Journal of Small Business Management, 29 (4).

Lewandowsky, Stephan and Ian Spence (1989), “Discriminating Strata in Scatterplots,” Journal of the American Statistical Association, 84 (407), 682-88.

Myers-Levy, Joan and Sandra A. Peracchio (1995), “Understanding the Effects of Color: How the Correspondence between Available and Required Resources Affects Attitudes,” Journal of Consumer Research, 22 (2), 121.

© A. Abela, 2008

Tony Kovner

Author of the book “The Practice of Evidence-Based Management,” with Richard D’Aquila and David Fine, published in 2008 by Health Administration Press

We commissioned 14 case studies on management interventions in health care organizations using an evidence-based approach. The managers did not follow a rigid template but made significant attempts to properly frame research questions, obtain evidence as to why an intervention might work in various contexts, evaluate evidence with a balance of viewpoints represented, decide whether the intervention could be adapted to their organization, analyze what it took to make the intervention actionable, and consider whether further evidence was needed.

The interventions involved: emergency preparedness, leadership development, the chief learning officer, forming a corporate university, criteria for hospital evacuation, chronic care management, improving pain management, improving health of underserved children, the business case for a hospital palliative unit, state Medicaid management, quality management in home health, inpatient planning, and improved operating room scheduling. The authors and editors of these cases studies believe that managers in these context had better evidence than is customary in considering these management interventions.

Evidence-based management is not widely used by health care managers for the following reasons: first, the business case for return on investment has not yet been reliably made. Second, widespread use would shift power away from senior toward junior managers. Third, hospital boards do not regularly review the quality of the managerial decision-making process. David Fine suggests that in the field of management, unlike clinical medicine, students are not taught to value and depend on studies as physicians are, and in part because of the lower priority, there are fewer studies done.

The new book can be used as a text to teach a capstone course in which teams of 4-5 students work two semesters in teams carrying out a project (often involving a planned management intervention) for a client organization. The materials have already been classroom-tested during Fall 2007 at NYU/Wagner in New York City.

…………………………

Book Chapters:

1. On the Practice of Evidence-Based Management
2. The Transformation Toward Evidence-Based Management
3. Theories and Definitions
4. Case Studies of Management Interventions Using an EBMgmt Approach
5. Implications for Management Education and Research
6. Lessons Learned and Summing Up
7. Supplementary Materials

This book is written primarily for master’s students as a capstone course (course outline ), and for health care managers with an interest in learning more about evidence-based management.

In today’s knowledge economy human capital is no longer a means of competitiveness – it’s now the means of competitiveness. And now, ironically, after years of struggling for a strategic role, Human Resources executives are finding themselves increasingly responsible for – and held accountable for – driving business results. This new role serves as a mandate for Human Resources executives to make their case – to clearly demonstrate how human capital strategies impact the business.

While this approach may appear obvious, it isn’t. According to research by The Conference Board, only 75% of companies measure anything in their HR departments while fewer than 25% indicate that they have anyone on their HR staff who can define, measure, and track HR metrics that are aligned to business strategies. A mere 12% indicated they use HR metrics to drive their company’s overall strategies.

In the fall of 2006 The Conference Board committed to improving this situation by launching a member-based Research Working Group, centered on bringing an evidence-based approach to Human Capital Analytics. The response from our members was overwhelming. In total 42 companies committed to joining in the year-long research exploration of evidence-based approaches to HR. At our invitation, many companies chose to pair a HR leader with a leader from another business function (including Finance, Operations, and R&D). For our Research Working Group participants, Evidence-Based HR is an effective catalyst for forming strategic alliances with other business functions, and positions them to contribute to the formulation of the business strategies.

While the interest is there, it would be misleading to construe this as proof that all these companies are doing Evidence-Based HR. This is not yet the case; but there are a few companies making inroads. These companies are beginning to make their own links between their HR practices and performance; a handful have begun testing hypotheses regarding their own HR practices; a few have even turned to the academic literature for guidance in designing their programs.

Through the course of the past 18 months we have, along with our Research Working Group members, identified the essential elements of applying an evidence-based approach to the practice of Human Resources:

• Evidence-Based HR focuses on the business strategy: Evidence-based practitioners start with the business’s key financial and operational measures, and use a rigorous approach to identify their human capital strategies that drive these key outcomes.
• Evidence-Based HR applies a high standard for “evidence”: The evidence-based approach is a rigorous approach, with the goal of identifying causal relationships.

In addition, in order to sustain an evidence-based approach our members believe it is equally important to recognize that this approach changes the way practitioners think, the way they look at information, the way they solve problems, and the way they evaluate their solutions. As a result, an evidence based approach will change how HR professionals use their existing tools.

For most HR professionals, the evidence-based approach represents a fundamental shift in their business roles. It provides them with an opportunity to become strategic partners in the business, but it also presents some daunting tasks. Fortunately, our experiences indicate that many traditional HR practitioners are willing to learn the new skills necessary to be evidence-based.

In December we published the first publication from the Evidence Based Human Resources
project: Evidence-Based Human Resources: A Primer and Summary of Current Literature.
Tracing back to the work of Jac Fitz-enz we show how HR measurement has evolved from identifying the basic measures of HR to today’s more sophisticated techniques which place these HR measures alongside a company’s financial or operational measures. Technology has certainly played a significant role in the measurement and use of HR metrics, as did the profession’s drive to “get a seat at the table.”

We also highlight some of the significant work in the academic literature, which lends a body of evidence for the practitioner to use when making decisions. Summarizing research from management, organizational psychology, and economics, we capture the quality of the available evidence. As practitioners move away from casual observations and begin demanding empirical evidence to support an idea, there will be mutual benefits to cross-discipline cooperation between economists, sociologists, industrial psychologists, and others.

We conclude by making the case that the value of the evidence-based approach lies in its ability to drive business performance. That is the ultimate responsibility of the business’s strategic partners. The time has come for senior HR leaders to be a strategic partner in the business. The evidence-based approach gives them the knowledge and discipline to be a successful strategic partner.

If you would like a copy of our research, and your organization is a member of The Conference Board, please feel welcome to go to the link below and download a PDF copy of the report at no charge. If your company isn’t a member of The Conference Board, but you’re interested in receiving a copy of the report, please contact us at our e-mail addresses below.

Evidence-Based Human Resources:
A Primer and Summary of Current Literature
Download this report

Christopher Woock
Research Associate
Management Excellence
The Conference Board
christopher.woock@conference-board.org

John Gibbons
Sr. Research Advisor
Management Excellence
The Conference Board
john.gibbons@conference-board.org

Summary of an article under review for publication

How can organizations improve their operating processes and hence performance? Experts advise organizations to leverage front-line staff knowledge about existing organizational processes via collaboration among front-line staff and between front-line staff and managers. Collaboration refers to individuals working together to achieve a common goal via information-sharing, joint decision-making, and coordination of activities. We tested the effectiveness of this strategy in 23 hospital neonatal intensive care units (NICUs) that participated in a quality improvement collaborative sponsored by the Vermont Oxford Network (VON) during 2002 and 2003. NICUs provide care to premature infants and infants with complications. As part of the collaborative, they worked to implement potentially better practices to improve patient outcomes.

We hypothesized that improved patient outcomes would be associated with higher levels of collaboration in three types of activities: (1) unit management (shared governance), (2) routine production (team meetings and collaborative communication), and (3) process improvement (selection of improvement projects and use of “learn-how” activities such as dry-runs). We tested our hypotheses at the patient-level (N = 1061) with population-averaged, logistic regression models to account for the correlation between infants treated in the same NICU and our binary outcome variable, 2004 patient mortality. To assess risk-adjusted improvement in mortality over the course of the collaborative, we controlled for NICU’s 2001 risk-adjusted mortality rate.

Contrary to our hypothesis, shared governance was significantly associated with an increased chance of mortality (Odds Ratio (OR) = 1.70; 95% confidence interval (CI) = 1.08 to 2.68; p = .02). Our other two hypotheses received partial support. Collaboration in routine production as reflected in collaborative communication was associated with reduced chance of mortality (OR = .74; 95% CI = .53-1.02; p=.06), but team meetings had no effect on mortality. Similarly, collaboration in process improvement as reflected by the use of learn-how was associated with a decreased chance of mortality (OR = .59; 95% CI = .36-.97, p =.04), but shared project identification was not. In sum, we found no evidence to support a positive relationship between collaboration in unit management and better performance; instead, we found the opposite. We did find some support for a positive relationship between reduced mortality and the more “active” forms of collaboration in routine production and process improvement.

Our results support the recommendation that organizations leverage front-line staff’s knowledge via multiple forms of collaboration (Institute of Medicine 2004). However, they show that not all forms of collaboration are helpful. Shared governance may interfere with performance improvement by adding time-consuming steps to change processes or diverting staff’s attention from core functions. Also, obtaining consensus across staff and managers may result in less-effective compromise solutions. On the positive side, collaboration in routine production and process improvement may solve two common problems: fragmented service delivery (Institute of Medicine 2004) and new process implementation failure (Cabana et al. 1999). Collaborative communication may facilitate better outcomes by enabling effective coordinated responses to changing conditions. Learn-how may help overcome resistance to change because staff adapt new practices to their setting. We hope this concrete knowledge about the effectiveness of different types of collaboration aids other organizations striving to improve performance.

References

Cabana, M. D., C. S. Rand, N. R. Powe, A. W. Wu, M. H. Wilson, P. A. Abboud, and et al. 1999. “Why don’t physicians follow clinical practice guidelines? A framework for improvement.” JAMA 282(15):1458-65

Institute of Medicine. 2004. Keeping Patients Safe: Transforming the Work Environment of Nurses. Washington, D.C.: National Academy Press

Most organizations today are awash in evidence about talent, yet lack the logic to use those numbers for vital strategic decisions. Lots of data with little logic is why Pete Ramstad and I proposed that we need a decision science for talent, in our book, “Beyond HR.”

We show that you don’t always have to invent these logical systems from scratch. They are often as close as the models you already use for other resources. As an added bonus, when you use these existing models, you create collaboration between leaders inside and outside of the HR function.

Here’s an example from the area of talent sourcing. Your supply-chain model applies to talent management decisions just as it does to raw materials, unfinished goods or technology (Boudreau & Ramstad, 2001; “Beyond HR” Chapter 7).

Supply-chain analysis optimizes supply chain elements, to achieve desired outcomes with the minimum resources. If the quality of raw materials drops, supply chain logic compares the value of things like switching suppliers, more careful screening of deliveries, or adjusting manufacturing processes to handle lower quality materials. Supply-chain analysis isn’t about one-size-fits-all, but about making the right decision.

In “Beyond HR” (Figure 7-2) we describe talent supply-chain as a series of labor pools: (1) A “potential labor pool” of individuals who could amass the right qualifications; (2) A “labor pool” of individuals who have actually achieved the right qualifications; (3) An “applicant pool” of individuals from the labor pool, who apply to your organization; (4) A “candidate pool” of those applicants you have screened and selected to receive offers; (5) A “new-hire pool” of those that accept your offers; and (6) A “productive employee pool” of those that accept your offer and develop into fully productive employees.

When a line leader complains that they are getting inferior talent, or not enough talent for a vital position, HR too often devises a solution without full insight into the supply chain. HR often responds by enhancing interviews or tests, and presenting evidence about the improved validity, even when a more effective solution would be to keep the same validity, but recruit from sources where average talent quality is higher. GE does this when it recruits military officers, allowing the military selection and development system to build a candidate pool that matches GE’s needs.

Or, consider what happens when business leaders end up with too few candidates, and instruct HR to increase recruitment. HR is often too eager to answer with more recruiting activities, when in fact the number of candidates presented to business leaders is already sufficient. The problem is that some leaders are better at getting candidates to accept offers. The better answer may be to improve the performance of the leaders who cause candidates to reject offers! This may be a difficult message for leaders to hear, and for HR to deliver. Yet, there is credibility in the supply-chain logic. It’s no different from the fact that you don’t fix a problem in your manufacturing operation producing too much waste by blaming the supplier!

Leaders are accustomed to a logical approach that optimizes all stages of the supply chain, when it comes to raw materials, unfinished goods and technology. Why not adopt the same approach to talent?

Now consider your own talent analysis and measurement systems. Can those systems tell you which stages in your talent supply chain hold the best opportunities to enhance end-state talent quality or quantity? Can you map the talent supply-chain stages described above? Can you tell whether better recruiting would have a bigger effect than improved testing? Can you tell which of your leaders do a great job landing top talent and which could improve?

If you can, you’re ahead of most organizations, competing for and with talent. If you cannot, you may be guilty of diligently analyzing the efficiency of your talent supply-chain, but missing the opportunity to optimize it.

References

John W. Boudreau and Peter M. Ramstad (2001). “Beyond Cost-per-Hire and Time-to-Fill: Supply-Chain Measurement for Staffing.” Center for Advanced Human Resource Studies, Working Paper 01-16. Ithaca, New York: Cornell University. Also available as Center for Effective Organizations working paper #G04-16.

Beyond HR: The New Science of Human Capital
John W. Boudreau & Peter M. Ramstad
Boston: Harvard Business School Publishing, 2007
Find it at Amazon

Every day we use our computers to perform a myriad of valuable functions from word processing to spread sheets to the gathering of information on the worldwide web. What we see on our screens are various applications, such as Excel, Word or Internet Explorer. From time to time we are asked to do various updates to these applications. Even better, we have enabled the automatic update function so that it occurs without our paying any attention. For us end users, all of the operating systems that are constantly running in the background are like a “ghost in the machine.” It is a modern day version of magic, a series of zeroes and ones that produces miraculous results. But if those updates do not occur and something goes wrong, the phantom that we ignored suddenly comes back to haunt us. Everything that we take for granted with the computer comes to a halt, leaving us frustrated, desperate and angry.

So it is with Evidence-Based Practices (EBP). We learn the concepts, we go to training, we develop action plans, and then we move to implementation…only to discover that 85% of the time, our efforts to realize what we have learned result in failure. We sit there thinking that there must either be something wrong with us or that EBP is just another new fad, not worth pursuing. In reality we have just encountered another “ghost in the machine,” only this one has to do not with technology but with our organizations. We are taking complex human behavioral interventions–in my world of criminal justice dealing with adult offending and juvenile delinquency–and trying to make them work on an antiquated operating system. It would be akin to surfing the internet on a computer still running the original version of DOS. It just can’t happen.

Yet every day we try to use advanced EBP applications and run them on an outmoded command-and-control, hierarchical bureaucratic system. Even more puzzling is that fact that we have come to assume that, in terms of EBP and our organizations, there is some type of automatic update process functioning in the background. We want to believe that implementing the knowledge of EBP will simultaneously transform the operating system of our organizations. If only the world were so simple.

Somehow we need to recognize that we can only apply research and science to the content of the work that we do, after we have applied the same principles to the organizations that we lead. Unless we understand, accept and take seriously this simple fact, which is of course where evidence-based management enters this picture, we will constantly be haunted by a “ghost in the machine” whose magic can only produce one nightmare after another.

Five Principles of EBM -

#1  Face the hard facts, and build a culture in which people are encouraged to tell the truth, even if it is unpleasant

Those of us committed to evidence-based management, the truth seekers of the business world, would be horrified at the prospect of inviting the legal profession in to adjudicate for us on what constitutes the truth about management effectiveness. Nevertheless, we would do well to take heed of one of their guiding principles; that no sound case can be built on circumstantial evidence. A company’s success can often be down to having a great product or market dominance rather than any special management capability. Coca Cola’s sales figures do not tell us that they employ great salespeople or even that they have any particular competence in marketing. Whatever expertise they possess, it did not stop their launch of Dasani water failing in the UK. In fact, it is precisely because simplistic, financial measures do not provide evidence of effective, organization-wide management that EBM has become such an important issue.

So what does effective management look like? One definition could be getting the best possible value out of all the resources available whilst ensuring a sustainable level of performance. This is inherently a long term prospect. However, some private equity partners might beg to differ. They might define their own management effectiveness as maximising short term profit, at the expense of increased debt and reduced long term value. If the EBM movement cannot resolve these potentially conflicting perspectives though it is unlikely to have a future. In short, EBM needs to be seen as the best way to create value if it is to be accepted as the best way to manage. But this will require a radical re-think about what organizations, themselves, value.

Certainly those who still say profit is king will have to admit that their perspective is a very one-sided view of the world. Microsoft still makes huge profits but that just tells us it is managing to hold onto its monopoly (for now). We also know that Microsoft has been heavily fined for some of its tactics, which makes it difficult to argue that profit automatically equates to acceptable management practice. There is also the issue of market context. If EBM is about unearthing some of the fundamental truths about effective management perhaps the fast-changing world of IT is not the best testing ground: management really has its work cut out in mature industries, like fast moving consumer goods, automotive and pharma. These are the crucibles in which EBM needs to be tested and its prime focus will have to be on evidence about the people dimension because all other levers for competitive advantage have already been pulled.

Conventional indicators will be of little use though. Market capitalisation, shareholder returns and return on net assets all suggest that something must be going right but they are all historical and are not great predictors of future performance. If EBM is to be a better predictor it needs to offer better alternatives to the welter of KPI’s, CSF’s, scorecards and dashboards that abound. Performance measurement, heralded as a cure, has become worse than the disease. If EBM is to achieve a pre-eminent position in organizational leadership thinking it needs simplicity and insight, not more sophisticated numbers. The emphasis should not be on performance measurement but the extent to which every individual understands the value objectives of the organization, is willing to give their best to achieve those objectives and knows exactly what is required.

The key piece of evidence we need is value. Reducing cost is not value if it means cutting corners. Improving customer satisfaction is not value if loyalty does not translate into revenue. Improving productivity is not valuable if quality falls. Value does not lend itself to deconstruction into a scorecard, it is what the whole organization is about. Every single employee has to understand what value means to them and any evidence of conflicting objectives is evidence of an organization that is not effectively managed.

Using these types of criteria, not only can we make our own minds up about how Microsoft has been managed in the past but also, more importantly, we can predict the way it is likely to go in the future with some confidence. Here is a view from one of their own employees, a Windows developer who had been with the company for five years – (quoted from the UK’s Sunday Times, 18th June 2006)

‘Deep in the bowels of Windows (the business unit) there remains the whiff of a bygone culture of belittlement and aggression. Windows can be a scary place to tell the truth.’

What is the long term, value potential of an organization that is a scary place to tell the truth? What more evidence do we need?

© 2007 Paul Kearns

Joseph C. Mello
joe.mello@davita.com
Chief Operating Officer, DaVita Inc.

Over the past number of months I have been in a quandary about something that I would love your opinions about. That is, “the myth of the mean”. I’ve been calling this out more and more lately with my direct reports. In essence: I believe that the mean/median/mode has only minimal power in true evidence based management. Increasingly the use of “average” data never gets to the essence of what is happening with any discrete population that drives performance. Let me give you but one simple example.

In measuring clinical performance for DaVita we pay attention to lots of indicators. One of them is “average hematocrit level”. We’ve been patting ourselves on the back for moving this indicator higher over the past 3 years. I am certain that is a good thing. However, in any given month when that number goes up, I have no window on whether we really improved the aggregate population, we got the patients that were already high higher, or we got the low patients up –which of course would be the best thing.

I could provide you with many other examples that are relevant to managing a portfolio of anything: reducing turnover is a good thing, but how many of our centers have zero turnover? How much of the reduced “average” turnover comes from the worst performing centers going from 110% to 80%? We’ve reduced our nursing vacancies from 548 to 392 in the past 12 months. If you’re at a center that has 2 nurse openings you don’t really care about that statistic. I’m not saying that averages don’t matter. I just don’t think they are as useful for managers to make decisions/take actions as other indicators. As you know, we at DaVita are relentless at only measuring stuff that we believe will require a management action at the other end.

I thought at first that “the myth of the mean” idea was only relevant for looking at the big picture. Stuff at my level of the organization. So I decided to test it a bit. I visited a few of the better Facility Administrators (our dialysis center managers) and asked them how they drove improvements in clinical indicators. They, of course, said that they focused on the bottom performing patients and worked on improving them first. So, what value can I add as a senior management schmuck by looking at the mean? Not much. I can say “improve those averages”. Or… I can say “look at these 22 centers that are the outliers”.

So…while we have intuitively always done a bunch of managing at the tails of the distribution I am focusing much more on outliers. Systematically measuring the “less than” and “greater than” populations. Looking at data on the best performers and the worst performers. Gaining insights from the best and sending SWAT teams to the worst. This seems to be working and I can almost generalize it to any population. About four years ago, we coined a moniker here that we refer to as “B-52’s”. At the time we had about 500 dialysis centers so really we were looking at the bottom 10% performers. Now that we are 1,300 centers, we still look at the bottom 10% (and we still like the B-52 label). Driving performance in this population drives aggregate performance up and if all else stays constant, the mean will improve as well. I no longer trust the rest of the population, so we now use averages or means very infrequently in our metrics, particularly our in process metrics.

Food for thought…

About Joe Mello

Joe Mello has been Chief Operating Officer of DaVita since June, 2000. He joined the company in the very early stages of a significant financial and operational turnaround. DaVita is a leading provider of dialysis services for patients suffering from chronic kidney failure. The Company provides services at kidney dialysis centers and home peritoneal dialysis programs domestically in 42 states, as well as Washington, D.C. As of December 31, 2006, DaVita operated or managed over 1,300 outpatient facilities serving approximately 103,000 patients.

As COO, Joe has responsibility for all operating units, IT, Nursing Services & Recruitment, DaVita University and Academy, Clinical Operations, Compliance Operations, BioMed Services, Reimbursement Operations, and People Services (our moniker for Human Resources).

Prior to DaVita, Joe was President and CEO of Vivra Asthma & Allergy, the nation’s largest single specialty PPM focused on chronic respiratory disease. In addition, Joe held various management positions with MedPartners (through its acquisition of Caremark) including senior vice president/chief operating officer – southeastern region. Prior thereto, he was a partner in the healthcare consulting practice of KPMG for ten years.

Joe holds an MBA in Finance from Golden Gate University and a Bachelor of Science in Industrial Engineering from Georgia Tech. He resides in La Quinta, California.