Volume VIII, Number 2, Fall 2002

Issue Topic: Evaluation for Continuous Improvement

Promising Practices

Risk

John Bare of the Knight Foundation shares his foundation's definition of the term “risk” when it comes to investing in initiatives, borrowing from the language of money managers.

I’m a little nutty when it comes to risk. I go looking for fried chicken, which increases my chances of heart disease, but alter my path to avoid walking over city sidewalk grates. I’m scared of being the million-to-one rube pulled from the sewer.

At least I’ve thought it through. In philanthropy, we frequently use the term “risk” without explaining what we mean.

At the John S. and James L. Knight Foundation in Miami, where I work as director of program development and evaluation, the strategic plan that trustees adopted in 2000 says the 26 U.S. communities where we make grants “will be considered laboratories, where high-risk and experimental ventures are tested and approaches that have been demonstrated to work are implemented and nurtured.”

Our staff loved the language, but it put us on the spot. We needed a common understanding of “risk.” We got help from evaluator Michael Quinn Patton who encouraged us to borrow concepts from money managers. Some grants are sort of like long-term bonds, some are blue chip stocks, and so on. From that, we developed ways to describe risk and manage it across a portfolio of grants.

Up front, we work with nonprofit partners to agree on a grant’s anticipated outcome. In financial terms, this is the ROI—return on investment. In recent years we added this explanation to the grant summaries we write for our board. This sharpens the grant development arguments. Are we assuming risk to produce effects that benefit individuals directly? Or to help communities learn something? Or both?

Caroming through trial-and-error innovation may not be the surest way to provide immediate benefits to individuals, but the promise of knowledge that benefits communities is a counterweight.

“We’ve learned that invention requires failing, starting over, learning from the mistakes, trying again,” our annual report says, pointing to the example of Thomas Edison discovering thousands of ways not to invent the incandescent electric lightbulb.

In explaining how we assess risk, I start with a simple example. Let’s say the desired outcome is for me to lose 20 pounds. Three types of risk come into play.

There’s idea risk. This represents the risk associated with the idea’s track record and the logic connecting the activities to the desired result. I found my favorite example of idea risk in The New York Times in summer 2001. Thieves stole art from a Jewish museum in New York and pledged to hold the objects until there was peace in the Middle East. Talk about your poorly formed theory of change. It’s just the opposite with my weight-loss example. For me, a low-fat diet with regular aerobic exercise carries almost no idea risk.

That brings us to implementation risk. Bad ideas can be well-implemented and vice versa. My diet-and-exercise solution fails because of botched implementation—I haven’t been able to stick with the regimen. In philanthropic terms, I wouldn’t give me a grant.

Untangling idea risk and implementation risk helps our staff integrate evaluation into grant development. When do we need more conceptual work? When do we need technical assistance to ensure effective execution?

Then there’s evidence risk. In my weight-loss example, there’s almost no evidence risk. Not only will the evidence be unambiguous evidence, it’s quite depressing.

But with most grants, evidence is dicey. It’s hard to agree on evidence standards. Then detecting effects is inexact. We risk not knowing one way or the other. Deciding when it’s important to diminish evidence risk helps with front-end evaluation planning. These discussions also help us reach consensus on investments that merit scarce evaluation dollars.

We’ve got a growing list of factors that can affect all three types of risk, and we have added a section on risk in the grant summaries we write for our board (see box).

With this prospective analysis, we don’t pretend to predict the future. We need a clear account of our original expectations to honestly compare purported risks against hazards actually encountered.

We are currently working with trustees to set tolerances for idea risk. We want agreement on how much we’re willing to invest in approaches that tilt toward the tried-and-true versus experimental approaches that ensure higher rates of failure, but promise greater learning opportunities.

Download the graph accompanying this article, entitled Hypothetical Risk Tolerances (15KB Acrobat file).

John Bare, Ph.D.
Director of Program Development and Evaluation
Wachovia Financial Center, Ste. 3300
200 South Biscayne Blvd.
Miami, FL 33131-2349
Email: bare@knightfdn.org